Yes, I think I agree with that. As long as your “main loss” is a sum of some terms (either via reduction=sum or it’s just composed out of a few different losses), you would have to multiply it somewhere to cancel out the 1/num_GPUs averaging that comes with DDP. From what I have seen so far, the most common thing is to just multiply loss prior to calling the backward() (your first suggestion). However, multiplying after should be fine too but you have to be careful and multiply before the optimizer.step() is called. I guess this second approach can then be achieved in two different ways: either by multiplying gradients (which could be slow) or by multiplying your learning_rate so that the final weight update has a proper scale.