It shouldn’t be a bottleneck and I would even recommend using it!
The __getitem__ method is usually called by the CPU and I’m not sure if it’s possible to call it from the GPU or just to transfer the calculations to it.
Using num_workers > 1 will use multi-processing on the CPU.
AFAIK even num_workers=1 creates another process, while num_workers=0 processes the data in the main process.
It’s not slower necessarily, since the data can be loaded and processed by your CPUs while the GPU is busy performing the forward and backward call. After the GPU finishes it can just get the new batch, which is probably already waiting for it. You should also use pin_memory=True in your DataLoader to speed up the data transfer between CPU and GPU.
GPU operations are called in an asynchronously way, so that they are not blocking your CPU.