Hi everyone. I’m trying to implement chunked data randomization in order to effectively randomize a dataset where getting random access to individual samples is impractical. Is there a good/canonical way to do this with the Dataset/DataLoaders classes?
I’m basically trying to do the following:
My dataset of N samples is split into M chunks (where M << N). I’d like to load K randomly selected chunks into memory and then randomly sample data from those chunks. As approximately one chunk worth of data is exhausted, you’d randomly load an additional chunk and continue to randomly draw samples from the chunks that are in memory.
(As some background, this is essentially how Deserializers in CNTK work, in which dataloading code returns chunks which are then randomized in the manner described above.)
Is there a simple way to do this? Based on my initial read of the documentation the only way to do this is to implement all of this buffering logic in the Sampler.