Thanks for this explanation, I wonder about the following sentence in the context of DDP:

We would have to multiply loss value with the number of GPUs only if the loss function had “reduction=sum” and not “reduction=mean” (to cancel out the DDP gradient averaging)

If reduction=sum is used for computing the loss, it shouldn’t matter whether:

the loss is multiply with the number of GPUs prior calling backward(),
the gradient is multiply with the number of GPUs after calling backward().

Do you agree?