Thank you, Shen Li.

DDP computes the average of all gradients from all processes, so the gradient should be the same value as local training for the  sum() / batch_size  case.

I interpret it as that the difference is taken care of when computing the global gradient from the local gradients, and we will see no difference from the serial cases.

What might affect the learning rate is the batch size you configured for each DDP process.

I think that whether or not we expand the global batch size is a choice between computation speed per iteration and algorithmic efficiency of total convergence, with a larger learning rate that you mentioned. Besides, we can make use of the GPU memories if we choose a large batch size. I feel that a larger batch brings about faster convergence even in the wall clock time bases, if we can efficiently utilize the multiple GPUs. That’s what I’m trying to do.
Thank you very much. I appreciate your time for this long discussion.