As you observe, by default, the buffers are local to the GPU.
If you wanted to synchronize your buffer across GPUs, you need to provide the structure to do that, which is a bit tricky in general.
SyncBatchNorm might be some inspiration (note that it also imports a SyncBatchNorm Function).

  

      github.com
  

  
    pytorch/pytorch/blob/e86d8323cb872a835ba0845e3519079f25e204fc/torch/nn/modules/batchnorm.py#L728



      
      
          # If buffers are not to be tracked, ensure that they won't be updated
          running_mean = (
              self.running_mean if not self.training or self.track_running_stats else None
          )
          running_var = (
              self.running_var if not self.training or self.track_running_stats else None
          )
          
          
# Don't sync batchnorm stats in inference mode (model.eval()).
          need_sync = (bn_training and self.training)
          if need_sync:
              process_group = torch.distributed.group.WORLD
              if self.process_group:
                  process_group = self.process_group
              world_size = torch.distributed.get_world_size(process_group)
              need_sync = world_size > 1
          
          
# fallback to framework BN when synchronization is not necessary
          if not need_sync:
              return F.batch_norm(
                  input,
      
    
  


  

  
    
    
  

  


Best regards
Thomas