Hi,
for models using batch normalization, most example code takes the mean and variance estimated during training for the evaluation of the model.
However, suppose now we have a different test set with different statistics than the training set. Or assume one wants to implement something as in the paper Averaging Weights Leads to Wider Optima and Better Generalization 11 (see chapter 3.3). There, we need to estimate the mean and variance again by passing every sample from our dataset though the model once.
My question is if there is already a build-in functionality for this. And if not, maybe we can start a discussion how this could be achieved.
The most “primitive” idea would be to first reset all bn related parameters, freeze the neural network weights except bn parameters, then do one “training” epoch. The question now is if this has to be implemented all from scratch or if there is build-in functionality that can help with this goal. One person suggested that require_grad=False set for some variables (the input?) could be a promising start.