Solved by mrshenli in post #6 
              
              
                correct. 

The input data goes through the network, and loss calculate based on output and ground truth. 
During this loss calculation, DP or DDP work differently. 

correct. 

Each loss in the GPU has the different loss result. 
DP used mean value because DP send every output result to main GPU an…