Hi, @SimonW of course, you are right. I was confused thinking that the running averages were used in the computation during training, but that is only during evaluation (in which case you need no grad). So, if you ask me, the grad will be correct as the batch statistics are calculated twice and won’t change.
However, the running average will indeed be updated twice. This means, if we substitute alpha = 1 - momentum:
Single update: x_avg <- x_avg * (1-momentum) + x_hat * momentum = x_avg * alpha + x_hat * (1 - alpha)
Double update: x_avg <- (x_avg * alpha + x_hat * (1 - alpha)) * alpha + x_hat * (1 - alpha)  = x_avg(n) * alpha^2 + x_hat * (1 - alpha) * (alpha + 1) = x_avg(n) * alpha^2 + x_hat (1 - alpha^2).
I think therefore we can correct the effect of updating twice by taking alpha_corr = sqrt(alpha) or momentum_corr = 1 - alpha_corr = 1 - sqrt(alpha) = 1 - sqrt(1 - momentum), but correct me if I’m wrong. To achieve an effective momentum of 0.1 (the default), this means you should use 1 - sqrt(0.9) ~= 0.051, although I assume the effect will be very small. A solution like @11173 may work as well, although it may be easier to just pass in a dummy running_mean/running_var placeholder the second time, instead of copying back and forth?