jagraves21:

If the problem really is data movement, how would setting num_workers help since the DataLoader does not move the data to the GPU?


Based on the description and provided code, I doubt the transfer is the bottleneck, but the data loading, but correct me, if I’m wrong.



 jagraves21:

I’ve considered writing a custom torch.utils.data.Dataset , that moves the data to the GPU before returning it and then using a DataLoader with multiple workers, but this seems like it would cause a lot of memory fragmentation on the GPU.


Why would this cause a lot of fragmentation?
I wouldn’t recommend this approach in general, as multiprocessing and CUDA calls might be hard to handle.