after reading the BN code in detail and posts around here + the original paper the conclusion is here Inconsistent Batchnorm behavior in eval and training modes - #4 by Brando_Miranda 1  in summary:

BN intended behaviour:

Importantly, during inference (eval/testing) running_mean, running_std is used (because they want a deterministic output and to use estimates of the population statistics).
During training the batch statistics is used but a population statistic is estimated with running averages. I assume the reason batch_stats is used during training is to introduce noise that regularizes training (noise robustness)


ref: [1502.03167] Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift 1

So the main mystery is to figure out why my models were saved this way and their running averages from training removed.

meta-leanring context: Is there data leakage in the maml-omniglot example? · Issue #107 · facebookresearch/higher · GitHub 1