Hey and thanks for your answer.
I already have datasamples split into several compressed files and code I attached is used to do exactly what you specified.
However I wanted to speed it up as I’m able to decompress a few files at once,
so I wanted to simply specify to workers which file they should be working with.
fe at every epoch n lists of randomly chosen separate subsets of files are passed to n workers,
and each worker iterates randomly over samples from his given files, file by file.
I obviously know there is mupltiprocessing built in to DataLoader but it simply cannot work for code given above, without some additional knowledge, which I’m unable to pass.
And as I stated before I can’t afford using random sampling, cause it would force me to decompress random file upon drawing EVERY sample.