Same here. Performance of DDP model is weaker than one trained on a single GPU. Playing with lr/bs does not help. As number of GPUs in DPP training grows - performance degrades.
Has anyone found the solution ?
UPDT: the reason was found for my case. When training DDP model we need to use DistributedSampler which is passed to Dataloader. We need to train_dataloader.sampler.set_epoch(epoch) on every epoch start.