Could you post the code you’ve used to benchmark the data loading?
How large is each data tensor you push to the device?
Based on your description it seems the to() call is quite expensive and I would like to make sure you are indeed timing the right operations.
Note that CUDA operations are asynchronous, so you would need to synchronize via torch.cuda.synchronize() before starting and stopping the timer.
Otherwise some operations might create a synchronization point and thus collect the execution time of the currently executed async calls.