Solved by mrshenli in post #2 
              
              
                Hey @Kunlin_Yang 
zero_grad
The reason DistributedOptimizer does not provide a zero_grad API is because the gradients of each backward pass is stored in its own dedicated context (instead of in param.grad), and the context will be cleared when exiting the with dist_autograd.context() as context_id: …