Hi Alex,
So it really depends on  your application. Generally speaking, if your batchsize is large enough (but not too large), there’s not problem running batchnorm in the “data-parallel” way (i.e., the current pytorch batchnorm behavoir)
Assume your batches were too small (i.e., 1 sample per GPU), then the mean-var-stats (with the current batchnorm behavoir) during training would be useless. But if they were large enough, then the batchnorm stats (as given by data parallel) would be just fine.
On the flip side, if there were hundreds of samples in each GPU and the mean-var-stats were calculated as you suggested, then there are some works, like Goyal et. al. show that this gives undesirable results: