Thanks a bunch!
I will give the method you suggested a go and get back to you.
NB. I think the batches of the data I am using can be treated in a similar way to the Cifar10 batches of torch vision datasets (using pickle command to read each instance, as shown in Cifar10_pytorch_dataset 11). But that’s even a bit complicated, so I thought of a dataset class that we could pass the batches and the metafile to get things done on the fly.