Do I understand correctly that DP will not average gradients of “replica” batches, while DDP will?
Is it always the case that gradient averaging is correct (assuming the loss averages over the “replica” batchsize)? If yes, is it the case because of the product operation  in the chain rule?
Thanks!