Loading the data in the Dataset's __getitem__ is a good idea.
Are you using multiple workers in your DataLoader (num_workers > 0)?
Is your data stored on a local SSD or are you pulling it from a network device?
PS: I would recommend to use tensor.view instead of tensor.reshape in your forward method.