I looked up the original Batch Normalization paper, and its authors mention moving averages in passing, for the purpose of tracking the accuracy of the model.  They suggest using only the batch mean and variance for producing the layer output.
However, the torch implemention 8 indicates that the mean and the variance are estimated using moving averages.
Are you saying that the BN code does not keep the gradients of the moving average buffers, and uses only the current batch to calculate the derivatives of the mean and average?
Did I get this correctly?
This would add random noise to the output of backward().  Interesting.  This requires batches to be large enough for the noise not to be too large.