Yes, I checked this thread, I knew DDP is better than DP as it dose fully All reduce  and multi processing, instead DP need to calculate the loss in main GPU thus in balanced GPU utilization.
But here my question is if # of GPU is equivalent to batch_size * # in DP ,  is it same in DDP? since in DDP “At the end of the backwards pass, every node has the averaged gradients” , so to me seems both are doing some “average” here