for RTX-3090
and A100.
DP vs DDP
DistributedDataParallel (DDP) is typically faster than DataParallel (DP), but it is not always the case:
* while DP is python threads-based, DDP is multiprocess-based - and as such it has no python threads limitations, such as GIL
* on the other hand a slow inter-connectivity between the GPU cards could lead to an actual slower outcome with DDP
Here are the main differences in the inter-GPU communication overhead between the two modes:
DDP:
At the start time the main process replicates the model once from gpu 0 to the rest of gpus
Then for each batch:
each gpu consumes each own mini-batch of data directly
during backward, once the local gradients are ready, they are then averaged across all processes
DP:
For each batch:
   1. gpu 0 reads the batch of data and then sends a mini-batch to each gpu
