Based on your __getitem__ it seems like you have pre-batched the data and now loading the files individually. The pickle.load() is a little suspicious because it might use a lot of memory. Have you tried reading from one source and setting the batch_size parameter on torch.utils.DataLoader instead? Also it would be helpful to see how you are using the data loader during the training loop.