Sorry for the necrobump, but I am also having a similar issue. I have a huge list of numpy arrays (>100, 000), and I am trying to create a custom Dataset that imitates ImageFolder with two classes (fake and real).
However, I still run out of memory even before training starts. How can I load the data lazily? Below is my code that implements my custom dataset. I call it with CustomDataset(args.data_root), where args.data_root is the root directory of my dataset, then I pass it to the vanilla DataLoader. Any help is appreciated. Thanks!
class CustomDataset(Dataset):
    def __init__(self, data_root, transform=None):
        fake = join(data_root, 'fake')
        real = join(data_root, 'real')
        data=[np.load(join(fake, array)) for array in os.listdir(fake)]
        data.extend([np.load(join(real, array)) for array in os.listdir(real)])

        target=[np.zeros(1, dtype=np.long) for i in range(len([array for array in os.listdir(fake) if os.path.isfile(array)]))]
        target.extend([np.ones(1, dtype=np.long) for i in range(len([array for array in os.listdir(real) if os.path.isfile(array)]))])
        target=[target[i][0] for i in range(len(target))]
        self.data = torch.from_numpy(data)
        self.target = torch.from_numpy(target)
        self.transform = transform