zazzyy:

of dataset. During the model update stage, parameter server gather gradients from each workers say g_1, …, g_n with respect to each worker node, average them, and using the averaged gradients to update the “global model” (i.e. w := w - lr * 1/n \sum_{i=1}^n g_i, in which w is the global model, n represents n worker nodes, lr is the learning rate).


Hi, you may be interested in Synchronized Multi-GPU Batch Normalization http://hangzh.com/PyTorch-Encoding/syncbn.html 64