Solved by ptrblck in post #4 
              
              
                Yeah, I understand the issue and stumbled myself a few times over it. 
I think one possible approach would be to use shared memory in Python e.g. with multiprocessing.Array. 
You could initialize an array of your known size for the complete Dataset, fill it in the first iteration using all workers, …