Gradients are averaged (divided by the number of processes) when using nn.DistributedDataParallel. This is not the case when using nn.DataParallel. You can either multiply them after the call to backward to make them equivalent to the output of nn.DataParallel.