Thank you for your reply.

The linked post focuses on DistributedDataParallel (DDP), which is different from DataParallel (DP). For DDP, every process/rank/device compute its own local loss, while for DP, the loss is global across all devices. I am not sure if this contributes to the discrepancy you saw.

It is true that the loss is global across all the GPUs, but I think that should still hold for doing gradient accumulation. I do not divide the loss if I use multiple GPUs, but I should divide the loss if I have gradient accumulation step > 1. Right?
Because if I run the model for a batch and get some loss and some gradients and again run the model for another batch and and get another loss and other gradients and accumulate the gradients for both batches (because I do not run optimizer.zero_grad() after the first backward()), it is like the case that those loss values are summed up, not averaged. Therefore it makes sense to divide the loss by the accumulation steps. Is that not true?

What do you mean by effective batch size? I assume the input batch size used in every iteration should be the same?

Yes. Each time before the optimizer.step() the same number of training samples are fed to the network.
For example: in case of one GPU with gradient accumulation, I set the batch size to 3 and the accumulation step to 2 (so overall I have 6). In the case of 1 GPU setting without gradient accumulation I set the batch size to 6 and for the case of 2 GPUs, I set the overall batch size to 6 (so 3 on each GPU).