@mrshenli,
I have been trying to reproduce your results, but, for some reason,  my experiment of DDP with two GPUs end up with about  4000 msec, unlike your results of about 500 msec.
I also tried DP with the same settings and the time was 1100 msec. The experiment indicates that DP is faster than DDP for a batch size of 2048 / GPU.
I used VGG11 model for the experiment, because K80 cannot accept 2048 data of Cifar10 with a large model like Resnet34 in its memory. I tried even a smaller model, but still DP is faster than DDP.
Would you please specify the model and data that you used to produce the results in your former post?
Are the GPUs that you used utilize NVLink?
Thank you for your cooperation.