Thanks!
I will try to vary the normalization momentum.
Ideally, I would like to reduce the batch size to 1, and then the minibatch standard deviations cannot even be estimated for a single batch, so neither momentum adjustment nor syncing over GPUs is meaningful. This is why I would like to estimate the normalization parameters (mu and sd) over ‘batches of minibatches’.