Solved by pritamdamania87 in post #2 
              
              
                Typically you want to run the forward and backward pass on each process separately and then average the gradients across all processes and then run the optimizer independently on each process. 
I’m wondering what is the reason you’re trying to build this yourself. PyTorch has a Distributed 
Data Par…