Looking thoroughly, the trick is written in loffe’s batchnorm paper (page 4).

… their sample variances. Using moving averages instead,
we can track the accuracy of a model as it trains.
Since the means and …