maxbit:

Because in the distributed case, you sum M gradients. In the non-distributed case, you just have one gradient which results from the average loss.


This is not accurate. In the distributed case, the processes first use AllReduce to sum the gradient and then divide it by world size. So, the gradient is averaged. If you need a sum instead, you can multiple the param.grad field by world_size.

I do not understand what the loss has to do with that as we deal with gradients on that level.

This is mainly for mathematical equivalence, which can have impact on how to tune other params like LR. E.g., if you use MSE as the loss_fn in local training, then the grad you get is per-sample average. When switching to DDP in this case, you might still want grads being per-sample average, and hence it should be AllReduce sum divided by world_size. However, if you use sum() as the loss_fn in local trianing, the grad you get is per-batch sum. And when switching to DDP (per process batch size probably should be N/M in this case), you might also want to keep the grad as per-batch sum. In this case, the grad should be AllReduce sum without division.

The second thing I do not understand is why we want to average instead of summing the gradients in the first place. According to the book in my initial post, it should be summing and for me, that’s more intuitive as well. Is there are reason why we average by default?

That’s not sth that PyTorch Distributed package can decide for you. It is application’s choice.