@mrshenli,
Thank you for sharing your GPU spec. That’s what I should have asked in the first place. GP100 far exceeds K80 in almost all the aspects of the performance. In addition, in my platform, K80 does not use NVLink. (Maybe it originally does not support it.) I suppose that the source of difference is the spec difference in the performance of DDP.
On the other hand, I still don’t understand why DP is faster than DDP with the same GPUs in my environment. My guess is that the DP directly performing all-reduce by a high spec CPU can be faster than the DDP performing all-reduce by gpu0, which communicates through PCI bus and memory on board. But, I 'm not sure.
Thank you very much.