I’m not sure on the implementation, but for what it’s worth it’s actually recommended NOT 200 to try and synchronize batchnorm for multi-node training. Not to deter you from exploring it, but unless you’re specifically investigating it I’d say it’s not the best place to devote effort for performance improvements.