Hi,
I’ve seen some discussions about DDP vs DP here but mainly focused around the learning rate. In my case both are taking the mean of the gradients from the GPU but I am seeing consistently somewhat worse performance in terms of loss and additional metrics from DDP than with DP. I am using same # of GPUs, same BS, same CrossEntropy Loss and other hyperparameters are kept the same as well. The epoch is set during every new epoch of DDP as well.
This is on a GPT-2 line language model so batch-norm cannot be the answer. In my understanding if the gradients are both averaged across GPUs and there is no batch-norm the two methods should be consistent except DDP is just much faster, however, obviously my understanding here is wrong I just haven’t found any explanation as to why this wouldn’t be the case.
Additionally, the seed is set for model initialization and many runs have been done as well.
Thanks!