Solved by ptrblck in post #2 
              
              
                nn.DataParallel will scatter and gather the parameters, gradients, etc. between the devices as explained in this blog post and is thus adding some overhead. However, it should not slow down the execution assuming you’ve scaled up the batch size. 
We generally recommend to use DistributedDataParallel …