Hi @vadimkantorov I think only SyncBatchNorm will broadcast and sync when using DDP, but they are syncing mean/invstd/count across GPUs instead of syncing buffers, the buffers maintained locally and updated locally. You can refer to the detailed implementaion in pytorch/_functions.py at 251686fc4cb1962944ed99c938df2d54f3d62e46 · pytorch/pytorch · GitHub
For SyncBatchNormed, broadcast/sync will not happen in eval mode, only in train mode.