During fine-tuning, if I set the BatchNorm2d in evaluation mode during fine-tuning on different datasets, the accuracy is increased from 63% to 80%.
However, if I leave the BatchNorm2d in the training mode, the performance drops drastically from 63% (before fine-tuning started) to only 39% in the first iteration.


How if I wanted to maintain the performance while leaving the BatchNorm2d in the training mode during fine-tuning?


What’s the BatchNorm2d setting equivalence if we set it to evaluation mode?