ptrblck:

bn.running_mean


Thanks for always answering quickly and accurateky, ptrblck!
I’ve tried to dig into the issue some more, and in another post I found this code where you implement this stuff: “https://github.com/ptrblck/pytorch_misc/blob/master/batch_norm_manual.py”.
If momentum is in use, you set
exponential_average_factor = self.momentum
Your update of the running mean looks like this:
self.running_mean = exponential_average_factor * mean
+ (1 - exponential_average_factor) * self.running_mean
If momentum is close to 1 (typically 0.9 in my application), doesn’t this calculation place most of the weight on the latest observation, and close to none on the history?
Shouldn’t it be the other way around for this calculation to accumulate a stable Ornstein-Uhlenbeck type average?:
exponential_average_factor = 1-self.momentum
Or should momentum be close to 0?