mrshenli:

Weight update is done by the optimizer, so if you are using the same optimizer the weight update strategy should be the same. The



Well, many people talk about “reduce” but in the context “reduce” does not seem like the literary “reduce”. That is why I ask the question. Because it keeps coming but no one defines this term first when they use it.
what is different between reducing gradients and weight update. Do you mean, DP and DDP exactly update the same weight and same updated each layer right? It is also confusing to me.
Do you mean Batch size or LR size?  You link the batch size about it.
I face that there is no improvement when I use the DDP with synchronized BN. That is why I am asking third question.