@makslevental It’s done not in the autograd code but in the DDP reducer code:

  
      github.com
  
  
    pytorch/pytorch/blob/8a026d4f74b71944ac2860c315996165a40f5626/torch/csrc/distributed/c10d/reducer.cpp#L321-L322 10

// Prescale bucket contents to turn the global sum into the global average.
replica.contents.div_(process_group_->getSize());