Passing a single instance to BatchNorm was just for demonstration purposes in my toy example. I’m not actually doing this for the real problem I was trying to solve.
Thanks for the update calculation! It looks like running_mean is initialized at zero. So, even if we have a static current_mean does not change between updates, running_mean will have to slowly update from the initial zero to the static current_mean value.