When using torch.nn.parallel.DistributedDataParallel to parallelize the network on multiple GPUs, do nn.BatchNorm become synchronized among GPUs ?
I suppose it is, because there is a broadcast_buffers flag in DistributedDataParallel defaulted to True.
Do anyone has any thoughts or confirmation on this ?