Unfortunately I don’t really have sample code, what I mean is that I am receiving consistently slightly worse results in terms of loss.

Ah my bad, sorry I misread the original question. I’m wondering if you could confirm that the only change between the two runs is that in one case we wrap the model with DDP() and in the other with DP(). Are you using a single process per GPU for DDP or one process driving all GPUs? Also, are you using GLOO or NCCL for the process group backend?