@LA_FORGE how did your export go?
I had similar troubles migrating around 500GB. (For anyone brave who thinks to give a try: do not use IndexUpgraderTool | Apache Solr Reference Guide 8.8.2 unless you are ok to miss a few fields mostly dates, and have a few java exceptions in your logs for these incompatible fields.)
Next think I did, I setup a separate solr 7.7.1 instance and used that for export, thinking that export speed will be increased if solr is in a different process space. I also changed YaCy code to export xml with lz4 compression instead of gzip in order to speed up the export. (I didnt use json yet, let’s hope that produced xml will work I will revert in a few hours)
However @Orbiter I see the following thing during export (with gzip or lz4 or json or embedded solr, it does the same)
It runs fast for a 1-2 hours and then the performance degrades:
The communication between yacy and solr over localhost is as follows:
I had a look at your export code in Fulltext.java and especially in your BlockingQueue that you get from solr, but it doesnt seem obvious to me why it behaves like that. I dont have much solr experience so it’s not obvious why solr speed degrades over time during export. (the machine is a high performance yacy dedicated machine with 64GB ram and there is nothing else consuming resources)
Any ideas? If we manage to somehow improve the export speed, I can write a small migration tool to export from one solr and import to the other if anyone else finds this useful, since you froze your version at 1.924 and you can no longer publish changes.