Solved by mrshenli in post #2 
              
              
                Hey @ankahira, usually, there are 4 steps in distributed data parallel training: 

local forward to compute loss
local backward to compute local gradients
allreduce (communication) to compute global gradients. This would be allreduce with SUM + divide by world size to calculate average
optimizer ste…