I wanted to use the means, stds from training rather than batch stats since it seems if I use batch statistics my model diverges (as outline here machine learning - When should one call .eval() and .train() when doing MAML with the PyTorch higher library? - Stack Overflow). How does one do that?
I am asking since my model seems to have them be zero despite no training having been done yet:
Out[1]: BatchNorm2d(32, eps=0.001, momentum=0.95, affine=True, track_running_stats=True)
args.base_model.model.features.norm1.running_mean
Out[2]: 
tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0.])

are these not saved in a ckpt after training? Should they have been saved?
Docs say they should have:

Also by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation.

by running means are zero vectors…  ?

related:

machine learning - How does one use the mean and std from training in Batch Norm? - Stack Overflow
What is the proper way to do evaluation (e.g. testing, validation) of deep learning model in PyTorch so that checkpoints are saved correctly? - Stack Overflow
Inconsistent Batchnorm behavior in eval and training modes