Thanks for your reply.
first I know it is not the true mean, that is why I ask how good that estimator approximates the true mean. It is true that it is infeasible to compute it, as we need to do a forward with all the training set for each parameter update.
The thing about the low pass filter is an alternative way of seeing it. What I mean is that this running average is making an smooth average estimate from different parameter updates, so we reduce peaks in the function (as a low pass filter does). It is the same with momentum and SGD.
Finally what I mean using the validation is normalize using the validation statistics only when BN takes places (the input is normalized using training statistics). I now this it’s not the best way, but as long as i do not now how this estimator approximate the true mean and true std, maybe using the mean and variance from my validation set to normalize each layer could be a quick approximation.