See section 2.3. But you are right. In the paper they wrote:

We also note that the BN statistics
should not be computed across all workers, not only for
the sake of reducing communication, but also for maintaining
the same underlying loss function being optimized.

But the point is, if BN was properly synced (which that repo above does) - it’s like effectively increasing the batch size. And, in general, I think there are concerns about having too large batch sizes. I don’t know if it’s conclusive or not, but yann lecunn said:

Training with large minibatches is bad for your health.
More importantly, it’s bad for your test error.
Friends dont let friends use minibatches larger than 32

when he posted this paper 5 on twitter 4. In my experience, there is no hard-and-fast rules and really each network / data / problem is different. Unfortunately, what that means is that the best theory is just “try both and see what works better”.