Solved by ptrblck in post #2 
              
              
                DDP should synchronize the gradients in the backward using AllReduce operations, such that each optimizer should use the same model parameters as well as gradients, thus also creating the same internal states. 
I don’t think there is a need to synchronize the optimizer’s state, but @mrshenli might c…