connection limit of pregenerate_training_data.py
## ❓ Questions & Help

Inside pregenerate_training_data.py, one can use multiprocessing to process each epoch in parallel.

However, since the communication between process is limited by pickle limit size. We can only transfer arguments less than 1GB and argument: docs are very likely to exceed this limit.

I save the docs as pickle file and ask every process to read on their own.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
