In the documentation of  DataParallel 38, it says:
During the backwards pass, gradients from each replica are **summed** into the original module.
In the documentation of  DistributedDataParallel 26, it says:
During the backwards pass, gradients from each node are **averaged**.
I think this difference caused the phenomenon described here 63:
Important: The default learning rate is for 8 GPUs. If you use less or more than 8 GPUs, you need to set the learning rate proportional to the GPU num. E.g., modify lr to 0.01 for 4 GPUs or 0.04 for 16 GPUs.