Hi, there.
I have implemented a Cifar10 classifier using the Data Parallel of Pytorch, and then I changed the program to use the Distributed Data Parallel. I was surprised at that the program has become very slow. Using 8 GPUs (K80) with a batch size of 4096, the Distributed Data Parallel program spends 47 seconds to train a Resnet 34 model for one epoch, while the Data Parallel program took only 32 seconds.
I run the program on a cloud environment with 8 vCPU with 52GBytes of memory, and it does not seem to be a data transfer problem. So, I roughly measured the time spent for each task within the DP and DDP processes. The results are shown below.
DP

DDP

In the above screen shot, the left most number is value of loss and other numbers are execution time of each task in seconds. The “out” represents the forward path and “back” represents the backward. As you can see, DDP takes more than twice of the computation time compared to DP for both the forward and backward path. I do not understand why this happens.
I suppose that this post 18 discusses the same issue, and it seems that the issue has been addressed.
However, it still happens in my program. The torch version of the program is 1.4.0. Should I update the version to solve the problem? Or, should I use Apex Distributed Data Parallel?