I think you’ve identified the problem with X = X.cuda(), it’s sending all your data to the GPU.
I’m not sure if you’re unaware, but there are already torch.utils.data.Dataset and torch.utils.data.DataLoader 11 classes in pytorch which you have basically reimplemented.
The idea is you load a batch per time in RAM, and then send each batch to your GPU device with batch.cuda() or batch.to("cuda").