You could try to use pinned memory and apply to('cuda', non_blocking=True).
If the GPU is busy, the transfer will be added to the queue.
If you are concerned about the best peak performance, you could have a look at this prefetcher 6, which I would not recommend for a general use case.