henry_Kang:

So Basically DP and DDP do not directly change the weight “but it is a different way to calculate the gradient in multi GPU conditions”.


correct.

The input data goes through the network, and loss calculate based on output and ground truth.
During this loss calculation, DP or DDP work differently.

correct.

Each loss in the GPU has the different loss result.
DP used mean value because DP send every output result to main GPU and calculate the loss.

This is incorrect. DP’s forward pass 1) create a model replica on every GPU, 2) scatters input to every GPU 3) feed one input shard to a different model replica 4) use one thread per model replica to create output on each GPU 5) gather all outputs from different GPUs to one GPU and return. The loss with DP is calculated based on that gathered output, and hence there is only one loss with DP.

  
      github.com
  
  
    pytorch/pytorch/blob/d06f1818ada6405a30943f58548af958c2b83ff6/torch/nn/parallel/data_parallel.py#L147-L162 2

def forward(self, *inputs, **kwargs):
    if not self.device_ids:
        return self.module(*inputs, **kwargs)

    for t in chain(self.module.parameters(), self.module.buffers()):
        if t.device != self.src_device_obj:
            raise RuntimeError("module must have its parameters and buffers "
                               "on device {} (device_ids[0]) but found one of "
                               "them on device: {}".format(self.src_device_obj, t.device))

    inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids)
    if len(self.device_ids) == 1:
        return self.module(*inputs[0], **kwargs[0])
    replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
    outputs = self.parallel_apply(replicas, inputs, kwargs)
    return self.gather(outputs, self.output_device)



  
  
    
    
  
  


DDP is multi-processing parallel, and hence it can scale across multiple machines. In this case, every process has its own loss and so there are multiple different losses. Gradients are synchronized during the backward pass using autograd hook and allreduce. For more details, I recommend reading the paper I linked above.

What is different between mean calculation and syncronized calculation?

Because DP is single-process-multi-thread, the scatter, parallel_apply, gather ops used in the forward pass are automatically recorded by the autograd graph. So during the backward pass, the gradients will be accumulated to the .grad feld. There is no grad synchronization in DP, because autograd engine does all grad accumulation already.
As DP is multi-process and every process has its own autograd engine, we need additional code to synchronize grad.

So the performance difference might come from LR difference?

Yep, that’s one possible source. It also relates to what loss function you are using. If the loss function cannot guarantee f([x, y]) == (f(x) + f(y)) /2, then the result can also be different, as it is not compatible with gradient averaging used in DDP.