Thank you for your reply ptrblck. I tried something similar to this at first but I didn’t get it to work. I have the data set split in several ‘.npy’ files and in init I am trying to read an arbitrary file of these. Each ‘.npy’ file that I have has like 5000 images so it’s numpy array is of shape (5000,3,32,32) because RGB images and 32x32 pixels.
When I tried to do it in a similar way to this PyTorch generated: (mini_batches, 5000, 3, 32, 32). So it generated mini_batches of the files which were all already 5000 images.
So sorry for being unclear but the init loads a file that is a subset of the dataset and contains about 5000 images.