Thanks for the reply. As my experiments confirm, transfer to GPU is significantly faster for data in pinned memory, so it is worth doing it. Issue is that transfer to pinned memory itself costs time, and only saves time overall if it can be parallelized. The data loader seems to be doing this - it spins up a separate thread for transfer to pinned memory when pin_memory flag is set to True. When you call enumerate or next(iter), dataloader waits until a batch is available in pinned_memory so if the processing time on GPU is sufficiently long, then the latency of pinned memory transfer should be hidden, at least partly.
The question is why is transfer to the GPU still slow even though the batch is in pinned memory? One difference between the manual approach and the regular approach (not calling input.pin_memory() manually) is that in the manual approach, the transfer is done over the main thread, while in the regular approach, it is being done on another thread. Does this make a difference?