Hello, I’m trying to load a large image dataset that won’t fit into RAM. I’ve looked up a similar question here on the forums, but can’t seem to get the answer working. the variable data_loc has the directory to images and targets.
class MyDataset(Data.Dataset):
    def __init__(self):
        self.data_files = os.listdir(data_loc)
        #sort(self.data_files)

    def __getindex__(self, idx):
        return load_file(self.data_files[idx])

    def __len__(self):
        return len(self.data_files)

set_test = MyDataset()
loader = Data.DataLoader(set_test,batch_size = BATCH_SIZE, num_workers=8)


for step, (x,y) in enumerate(set_test):
    *do stuff*
    set_test = MyDataset()
    loader = Data.DataLoader(set_test,batch_size = BATCH_SIZE, num_workers=8)




But I get a not implemented error for the set_test. Any thoughts on how to fix this?