Hi,
I would advice getting all your samples into a single Tensor. Wrap that into a TensorDataset. And use the builtin DataLoader to get samples for training. That will make your dataloader as efficient as it can be. Use a couple workers to make sure you get as much data as possible.
Then you might want to check how multithreading is behaving. In particular, if you have a small network, it might be beneficial to set torch.set_num_threads(x) to a value x that is smaller than the number of cpu threads. You will need to experiment with that value as it will depend a lot on your cpu and workload.