Is it possible to crawl sites similar to the URL? What I mean is suppose lets say I crawl Youtube is it possible when I crawl youtube, YACY can also crawl sites similar to youtube lets say like Vimeo and Twitch and all? w
Of course you can just put such similar domains in your crawler but there is no knowledge inside of YaCy which tells which pages are similar.
But thats a kind of good idea to get such a “is similar” list. How would we find such a kind of list?
We could use typical "list of … " sources like
- wikipedia “list of” like List of content management systems - Wikipedia
- “alternative to” comparisment pages like content management system - Search on AlternativeTo.net
It would be required to build up a huge list and put that into a release of YaCy and use it as a kind of recommendation system during crawl start.
1 Like
It would be required to build up a huge list and put that into a release of YaCy and use it as a kind of recommendation system during crawl start.
That’s exactly what my idea was. Interesting will look into it. One more thing @Orbiter is it possible to search index’s or sites from other peers? Like suppose my friend has crawled like 100 sites and I crawled lets say 400 or something. And he wants to use the sites that I have already index and all. Is it possible for him to use my index?
you could i.e. export the index as host list and use that host list for a new crawl
2 Likes
Interesting, I thought using Senior and Junior Mode we could like search the index with importing or exporting. Can you please explain to me how to change and use senior mode. Also how to search in Global Index 
Thank you.
1 Like