Hey @hij
Buffers are broadcast from rank 0 to other processes in the beginning of every forward pass. See the code below:

  
      github.com
  
  
    pytorch/pytorch/blob/1f8267931176cc9ecbf00493e5a359a57baca3df/torch/nn/parallel/distributed.py#L509-L513 10

# Synchronize buffers across processes.
# The process with rank 0 is considered the authoritative copy.
self._distributed_broadcast_coalesced(
    self.modules_buffers[0],
    self.broadcast_bucket_size)



  
  
    
    
  
  


If this is what you need, you can register the stddev as a buffer. If you need sth different (e.g., square stddev and sum it across all processes over the wire), you can call allreduce 9 or allgather 4 in the forward function to do that.