Hey @pl3, sorry about the delay.
For DataParallel (DP), it can become slow when the model is large, as DP needs to replicate the model in every forward pass.
For DistributedDataParallel (DDP), I would expect it is faster than local training. Which of the numbers shown above are DistributedDataParallel? And how did you initialize DDP module? When using DDP, did you reduce the per-process batch_size to batch_size / world_size?