ptrblck:


If the problem really is data movement, how would setting num_workers help since the DataLoader does not move the data to the GPU?

Based on the description and provided code, I doubt the transfer is the bottleneck, but the data loading, but correct me, if I’m wrong.


The data is loaded completely into RAM and stored in X and y initially. The problem is that the data does not fit entirely on the GPU.  The only difference between the first block of code and the second is when the data is moved to the GPU.  I’ve tested with subsets of my large dataset that will fit into memory and found that the second block of code is about 1.5 times slower than the first.



 ptrblck:

 jagraves21:

If the problem really is data movement, how would setting num_workers help since the DataLoader does not move the data to the GPU?
I’ve considered writing a custom torch.utils.data.Dataset , that moves the data to the GPU before returning it and then using a DataLoader with multiple workers, but this seems like it would cause a lot of memory fragmentation on the GPU.

Why would this cause a lot of fragmentation?
I wouldn’t recommend this approach in general, as multiprocessing and CUDA calls might be hard to handle.


I just assumed that it would not be the best use of GPU memory to have the torch.utils.data.Dataset move each sample to the GPU when it was requested.  Currently, 128 samples are moved to the GPU at a time.