I guess because the vast majority of loss functions in PyTorch have the default behavior to average losses across all samples in the batch, i.e. they have reduction=mean. To get the mathematically equivalent gradients in a DDP experiment (like the ones you’d get by running the 1-GPU experiment), you have to average them. If your loss function has reduction=sum, then you have to multiply the loss value at each GPU process with the world_size to cancel out this averaging.