If your data is too big to fit in memory. You can use a Dataset/Dataloader similar to what is used for Imagenet: Every time a sample is needed, it is read on the disk, preprocessed then returned. The dataloader num_workers argument allow you to have multiple processes doing this loading/preprocessing on the side to make sure it is not a bottleneck.
In particular, you can implement your own torch.utils.data.Dataset 4. Which requires to implement a __getitem__ to retrieve one sample from your dataset and __len__ that tells the length of your dataset.
You should do loading and any required preprocessing in the __getitem__ method and return one sample ready to be forwarded in your network.
Then you can use the basic dataloader with few workers to make that dataset loading parallel !  Note that you should experiment with the number of workers and find the best one by experiment.