hhxx:

Another question is if we do not divide batch-size by 8, the total images processed in one epoch will be the same as usual or eight times?


The total number of images processed will be 8 times, because each DDP instance/process will process batch_size images.

Let us back to DDP scenario (2 node, 4gpus per node), what learning rate shall we use？ 1×lr or 4×lr or 8×lr？

It should be 1x lr. Because DDP calculates the average instead of sum of all local gradients. Let’s use some number to explain this. Assuming every image leads to a torch.ones_like(param) gradient for each parameter.

For local training without DDP, if you set batch_size = 64, the gradient for each parameter will then be torch.ones_like(param) * 64.
For 8-process DDP training, if you set batch_size = 64, the local gradient for each parameter will also be torch.ones_like(param) * 64. Then DDP use collective communication to calculate the sum of gradients across all DDP instances which will be torch.ones_like(param) * 64 * 8, and then DDP divides that value by 8. So the final gradient in param.grad field will still be  torch.ones_like(param) * 64 (the code actually first divide and then do globally sum). So, when set lr, you only need to consider local batch_size.