There are two things which I found really useful.


Using the pinned memory can help improve the training speed a lot if you are training the network with multiple GPUs.


Enabling torch.backends.cudnn.benchmark would allow cuDNN to use faster algorithms for both forward and backward propagation but it may use more memory.