TW1920 - thank you for asking.
I don’t know if we are caching the websites, but if that is the default setting, then we probably are. What are the benefits of caching the crawled websites? If we turned off caching, would the cache be automatically cleared? Could the cache benefit others, if we are going to delete it?
This is crawling about 12 sites to a depth of 4.
Looking at the YaCy folder there are a few gigantic files:
And there is a large folder with lots of files:
It is all over 300GB now, after just a couple of days of crawling.
This sort of storage usage might discourage new users. Perhaps there ought to be a different default setting to not cache, if the cache is the problem.