Solved by mrshenli in post #2 
              
              
                Hey @111344 
If each DDP (DistributedDataParallel) process is using the same batch size as you passed to DataParallel, then I think you need to divide the reduced loss by world_size. Otherwise, you are summing together losses from world_size batches. 
Another thing is that batch size and learning ra…