Hi,
this question is somewhat in between PyTorch’s implementation of DistributedDataParallel and Paramters Server in general.  My “theory” source is the Dive into Deep Learning book [1].
Figure 12.7.1 on [1] suggests the following approach:
Assume batch size 32. If we have one GPU and 128 training datapoints, each epoch has 4 mini batches. After every mini batch, we update our model (i.e. there are 4 updates). Hence, we have to calculate four gradients (one per minibatch).
For the multiple GPUs case, assume two GPUs and 128 training datapoints. We feed each GPU a mini-batch, let them calculate a gradient, sum them and update our model with the sum. Hence, there are two steps instead of four involved (from a BSP point of view).
My questions are the following:


Is my described understanding on how parameter servers work correct? Especially I am not sure if we keep the same batch size 32 per GPU or if we have to divide the batch size by the number of GPUs.


Why do we sum the gradients instead of averaging them? This is even more confusing to me, as in the DistributedDataParallel documentation of PyTorch (cannot post the link due to the link limit for new users), there is the following statement:



When a model is trained on M nodes with batch=N, the gradient will be M times smaller when >compared to the same model trained on a single node with batch=M*N (because the gradients >between different nodes are averaged). You should take this into consideration when you want to >obtain a mathematically equivalent training process compared to the local training counterpart.

There are two things confusing here:
2.1. It states that the gradients are averaged, which does not comply to the D2L book which states that we sum the gradients.
2.2. Until now, I always thought, in mini-batch gradient descent, the loss function (optimization goal) averages the error of the mini-batch data points. Hence, if M nodes run mini-batch gradient descent with batch size N, and we take the average of their gradients, we should receive a number that is in the same order of magnitude as if 1 node runs mini-batch gradient-descent with batch size NM, as the single node averages the error functions of nm data points for the loss function, while with M nodes we just take the average of averages.
I am not sure whether the DistributedDataParallel class of PyTorch can be seen as a parameter server (especially because there even is a guide on how to build a parameter server in PyTorch [3]), but it maps to what is described in the book as a parameter server.
Any help on resolving my confusion is much appreciated. Thank you very much!
Kind regards,
Maximilian
[1] https://d2l.ai/chapter_computational-performance/parameterserver.html 3
[2] https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html 8