Very Large YaCy Folder

After a new installation of YaCy, we ran a few crawls on some publications. This rapidly resulted in a very large folder (over 30GB).

YaCy is engineered so that large storage isn’t usually needed. Why would such a large folder result from about 20 crawls?

How big is your index? So how many sites are indexed? Do you cache the sites?

The index was about 32GB. I was using the default settings and tried running some crawls on quite large sites. I think I tried about 10 or more sites.

I wouldn’t mind caching the results, if they could be made of use to the rest of YaCy users. I could see some improvement in the search results I had after those few crawls, which were still in progress when allocated space became full.

By the way, if I wish to reinstall YaCy to a larger drive, is it an easy process to copy paste a folder into the new installation to avoid having to go through the crawling again?

1 Like