For my better understanding, @mrshenli - can you please answer?
Suppose we have 32*8 images in the dataset, and the batch size is 32. We want to train the model for 1 epoch only. Now consider following 3 scenarios. Note that we use same LR and optimizer function in all three cases below.
(1) Single Node - Single GPU: In this case, one epoch will require 8 steps to execute i.e. in each step, 32 images will be processed. The gradient will be calculated and applied 8 times. The model parameters will get updated 8 times.
(2) Single Node - Multiple GPU using DataParallel: Suppose we use 8 GPUs. In this case, one epoch will still require 8 steps to execute i.e. in each step, 32 images will be processed (albeit each GPU will process only 4 images and then the gradients will be summed over). The gradient will be calculated and applied 8 times. The model parameters will get updated 8 times.
(3) Single Node - Multiple GPU using DistributedDataParallel: Suppose we use 8 GPUs. In this case, one epoch will require just 1 step to execute as in each step 32*8 images will be processed. However, this also means that the gradient will be calculated and applied only 1 time. And consequently, the model parameters will get updated just 1 time. So, in order to get similar results as previous two points (#1 and #2 above), we will have to execute 8 epochs instead of 1, as gradients are averaged in the backward function and hence effective weight updates are almost similar in scenario 3 and those in scenarios 1 and 2 above.
Is this understanding correct?