Solved by ptrblck in post #2 
              
              
                I think the difference might come from the gradient reduction of all device gradients after the backward pass. 
Here is an overview of the DataParallel algorithm. 
Unfortunately, I can’t test it right now, but from my understanding after the parallel_apply of model’s backward pass, the gradients wil…