Oh that’s super nice, have to give it a try later. So, basically I just need to wrap a torch.nn.DataParallel around my model and it’s good to go!? Neat.
PS: maybe it’s worth mentioning the multi-GPU in the Readme or so (e.g,. as a https://github.com/pytorch/pytorch#a-gpu-ready-tensor-library 149 subsection). For instance, as Adam Paszke wrote on https://github.com/apaszke/pytorch-dist 115

Multi-GPU ready
PyTorch is fully powered to efficiently use Multiple GPUs for accelerated deep learning.
We integrate efficient multi-gpu collectives such as NVIDIA NCCL to make sure that you get the maximal Multi-GPU performance.