Problem Statement
Given is a corpus of large files with multiple non-overlapping chunks (i.e. samples) per file.
The corpus is too large to be pre-loaded completely.
Since DataLoader relies on lazy loading, each time a sample is requested, the entire file is loaded, the corresponding chunk extracted, then the file discarded. To remedy this inefficiency, cache file’s signal.
But, how to fit caching (sharing state) seamlessly with DataLoader or multiprocessing in general?
Attempts

Chunking offline: not desired
Pre-Loading corpus partially, split epoch into sub-epochs


not nice
reduces randomness


Sharing memory


need to write to shared memory for loading to cache (slow)


Splitting corpus, using DistributedSampler and DistributedDataParallel with 0 workers


avoiding shared memory
sampling to cure loss of randomness

Is there another more convenient way of handling this?