To add to James’ comment:
http://pytorch.org/docs/notes/cuda.html#use-pinned-memory-buffers 278
Rather than placing everything on the GPU in one go, you can ensure that each CPU-GPU memory transfer is as fast as possible.