Solved by ptrblck in post #2 
              
              
                The training might differ generally, as the gradients will be reduced from all devices to the default device, which would be equivalent to training with a bigger batch size. 
Also, if you are using batchnorm layers, their running estimates won’t be synchronized in nn.DataParallel, which might also m…