TT_YY:

My guess is that the DP directly performing all-reduce by a high spec CPU can be faster than the DDP performing all-reduce by gpu0, which communicates through PCI bus and memory on board.


DP uses replicate, scatter, and gather operations, which are basically cudaMemcpy under the hood. I suspect cudaMemcpy can be faster than allreduce operations used by DDP on some hardware.