TT_YY:

Using 8 GPUs (K80) with a batch size of 4096, the Distributed Data Parallel program spends 47 seconds to train a Resnet 34 model for one epoch, while the Data Parallel program took only 32 seconds.


Does this mean each DDP process is consuming 4096 samples per iteration and the DP process is consuming 4096 * 8 = 32768 samples?

I suppose that this post 25 discusses the same issue, and it seems that the issue has been addressed.

For the post you mentioned, it is only true for BERT models and it has been addressed in PyTorch v1.6.
BTW, how did you initialize DDP? Could you please share a repro?