Hi everyone,
I have a tensor stored in a file (initially it was a dataset stored in a matlab .mat file that I read using scipy and then stored as a tensor using torch.save). the data represents RGB images stored in a tensor of shape (N * C * H * W) where N = number of training examples/images, C = number of channels, H * W =  size of the images.
I load the tensor using torch.load, but the problem is that I want to load these images in batches, so that I don’t have to load the whole file to memory for training (actually I don’t even have enough memory to load it all).
I read the Pytorch tutorial about loading custom datasets but in the tutorial they load the whole tensor file containing image names/paths (stored as .jpg files) then load the images one at a time (I can’t do this because my images are all stored in a single tensor).
Thanks a lot !
TL;DR: how to load a tensor from disk to memory in batches? (the tensor file is saved with torch.save)