Could you provide a script that can reproduce this issue?
Are you sure that you used the same learning rate, loss function, optimizer, same number of epochs, etc?
Both DP and DDP should be able to produce a model that can be also output by PyTorch w/o using DP or DDP, as long as the gradients are synced at every step. What about the performance w/o using DP or DDP?