Hey!
I’m a first time responder and pretty new to pytorch and deep learning. I am also trying to train a CNN (UNet to be specific) with image data and think I’m running into a problem related to this topic.
I have a h5py data base file that is too big to load (~27GB). It has 8,000 sampls and each sample shape is (14,257,256). I think It’s worth to mention that I am creating that data base and I can control the number of files and each file size.
Acording to the solution here I can create 8000 different h5py files and use the getitem method to get a different sample from a different file every time (In that case does the len method still returns 8000?).
I was thinking of a way to combain these 2 ways and creating 8 files 1000 samples each. In that case the loader would look like(?):
class h5_loader(data.Dataset):
    def __init__(self):
	    self.file_path = '/content/drive/My Drive/Project Dataset/trainFile'
    def __getitem__(self, index):
        i = index/1000
        mod = index%1000
	    h5_file = h5py.File(file_path+str(i)+'.h5','r')
        input = h5_file['Input'][mod]
        target = h5_file['Target'][mod]
        return input, target


Would that work?
What would you say is the most “clean” and fastest way to load my data into the model?
I’m working on google colab from my 8GB RAM laptop (I use 0 workers).
Thanks!