Hi Bencherian,
I think Pytorch’s ChunkDataset meets exactly what you need here. It works with a ChunkDataReader to randomly pick chunks, shuffle examples in the chunk and caches them into a internal buffer. Dataloader will retrieve batch data from this buffer. When the buffer is exhausted, ChunkDataset will automatically loads more chunks.
To make this tailored for your data format, all you need to do is to implement your own ChunkDataReader, which knows how to read a single chunk of data with a given index. Pass this class to ChunkDataset and the rest logic is handled by ChunkDataset.
If this is something you are interested in, you can take a look at some example code in the test: DataLoaderTest::ChunkDataSetGetBatch
Hope this helps