Thanks for your reply!
I didn’t use elasped_time to measure how long it takes for one iteration but I did use time.time() to calculate the iteration latency. As long as there is one GPU that remains unused, the process runs slower (sometimes 50% slower and sometimes 100% slower).
The weird thing is that it seems my DDP program is influenced by other programs (they are not necessarily DDPs) running on other GPUs, in a counterintuitive way: normally we expect that programs are competing for the computational resources, but here more programs bring better performance…
(This thing didn’t happen to my colleague’s non-DDP program so I guess it must have something to do with the DDP).