what is different between reducing gradients and weight update.

There are many weight updating algorithms, e.g., Adam, SGD, Adagrad, etc. (see more here 1). And they are all independent from DP or DDP. So even if the gradient is the same, different optimizers can update the weight to a different value.
Reducing gradients in DDP basically means communicating gradients across processes.

Do you mean, DP and DDP exactly update the same weight and same updated each layer right?

Neither DP nor DDP touches model weight. In the following code, it is the optimzer.step() that updates model weights. What DP and DDP do are preparing the .grad field for all parameters.
output = model(input)
output.sum().backward()
# DP and DDP not involved in the below this point.
opt.step()


It is also confusing to me. Do you mean Batch size or LR size? You link the batch size about it.

Quoting some discussion from that link. If you search for “lr”, you will find almost all comments in that thread discusses how to configure LR and batch size.

  
    
    
    Should we split batch_size according to ngpu_per_node when DistributedDataparallel distributed
  
  
    Another question is if we do not divide batch-size by 8, the total images processed in one epoch will be the same as usual or eight times? 
As for learning rate, if we have 8-gpus in total, there wiil be 8 DDP instances. If the batch-size in each DDP distances is 64 (has been divides manually), then one iteration will process 64×4=256 images per node. Taking all gpu into account (2 nodes, 4gpus per node), then one iteration will process 64×8=512 images.  Assuming in one-gpu-one-node scenario, we…
  



  
    
    
    Should we split batch_size according to ngpu_per_node when DistributedDataparallel distributed
  
  
    I agree with all your analysis on the magnitude of the gradients, and I agree that it depends on the loss function. But even with MSE loss fn, it can lead to different conclusions: 

If the fw-bw has processed 8X data, we should set lr to 8X, meaning that the model should take a larger step if it has processed more data as the gradient is more accurate. (IIUC, this is what you advocate for)
If the gradient is of the same magnitude, we should use 1X lr, especially when approaching convergence. Ot…
  



I face that there is no improvement when I use the DDP with synchronized BN. That is why I am asking third question.

Right, SyncBatchNorm has its own way for communication, which is out of control of DDP. Using DDP won’t change how SyncBatchNorm behaves.

  
      github.com
  
  
    pytorch/pytorch/blob/f64d24c941a00bc81b3017008ae212cca761d393/torch/nn/modules/_functions.py#L79-L81 4

torch.distributed.all_reduce(
    combined, torch.distributed.ReduceOp.SUM, process_group, async_op=False)
sum_dy, sum_dy_xmu = torch.split(combined, num_channels)