It looks like you’ve used some caching mechanism in your Dataset.
Is it working properly after the first epoch?
You could try to set pin_memory=True in your DataLoader and pass non_blocking=True to the tensor.to() call to speed up the transfer.