The buffers in batch norm are synchronized between processes if broadcast_buffers=True, yes. This means that all processes get a copy of the buffers on process with rank 0. If you want to use a synchronized batch norm, check out nn.SyncBatchNorm 301.