If the problem really is data movement, how would setting num_workers help since the DataLoader does not move the data to the GPU?
I’ve considered writing a custom torch.utils.data.Dataset, that moves the data to the GPU before returning it and then using a DataLoader with multiple workers, but this seems like it would cause a lot of memory fragmentation on the GPU.