blackberry:

I wish, it was possible to also save the running mean/std of batchnorm layers from training and use them at test time with a small momentum. This would mitigate the problem.


If you are loading the state_dict, the running stats will also be restored.
Since the validation and test cases are executed via model.eval(), the momentum term won’t be used. Instead the running stats will be used directly to normalize the data.
Could you post the model architecture, so that we could have a look?