I assume you are loading the data in the Dataset.__init__ method?
If that’s the case, you could preload the data before creating the Dataset and pass it to its __init__ method:
data = torch.load(...)
dataset = MyDataset(data)
loader = DataLoader(dataset)

This should avoid reloading the data. However, the Dataset.__init__ method would still be called in each epoch, if I’m not mistaken, so you should make sure that no heavy loading is there.