AndersML:

Are there any recommendations for using BatchNorm2d to get similar behaviour in training and evaluation mode?


A bad evaluation performance could point towards running stats, which don’t represent the dataset well. This would have different reasons, such as different data domains for the training and validation sets. E.g. batchnorm layers would most likely “break”, e.g. if you forget to normalize the validation inputs (while it was performed during training).



 AndersML:

If I evaluate in training mode, then I get better results and the network have more similar environment. Why isn’t there a similar mode for evaluation?


You could disable the running stats during evaluation and also use the batch statistics. However, this would also mean that your validation outputs and predictions depend on the batch size and you might get worse results for different batch sizes or your model might even raise an error, e.g. if stats cannot be computed from a single sample, which is often not desired in the validation/test use case.



 AndersML:

If I evaluate in training mode but with torch.no_grad(), does the running stats update and get used and what does happen to the affine values?


Yes, the running stats will be updated, which could be considered a data leak. The affine parameters won’t be updated, since the forward pass was performed in the no_grad() context and you are most likely not calling loss.backward() and optimizer.step() to update the affine parameters.



 AndersML:

Could I change track_running_stats=False when using the model in evaluation and use more samples in the batch or larger images? (Is there an easy way to change that for one layer if I load a model? (I can iterate the layers.))


Yes, you could, but see the previous point for some disadvantages.
I don’t know, if you can switch the behavior of batchnorm layers after they were initialized (I think this option wasn’t working recently, but you should double check and in doubt initialize these layers directly in the desired mode).



 AndersML:

Could I warm up the batch normalization for the evaluation before looking at the results?


I would also consider this a data leak. The validation samples might perform better, but you would have to ask yourself, if you could even apply the same once this model is deployed. E.g. would you also use a warmup phase? If not, then the validation results would be biased and your model could perform badly on real unseen samples.



 AndersML:

Should I try a larger momentum so that the affine isn’t too influenced by the learning last iteration?


Yes, changing the momentum might help in your use case.