johncookds:

In my understanding if the gradients are both averaged across GPUs and there is no batch-norm the two methods should be consistent except DDP is just much faster


I’m assuming you mean “DDP is just much slower” here? Do you have some sample code illustrating this performance gap? If so, it is much easier for us to troubleshoot why DDP might be much slower than DP in certain use cases.