Hi thank you for quick response.


The batch size is set to 64 during fine-tuning in my experiments. And yes, I tried different values of momentum from 0.01, 0.1, 0.2, up to 1, and it is observed that the best accuracy is only 72% (momentum = 0.1) compared to 80% for BatchNorm2d in evaluation mode.


Yes, I tried to manually code the evaluation mode to see how batch-norm works by setting child_layer.momentum = 0, child_layer.weight,requires_grad = False, child_layer.bias.requires_grad = False, but it seems like the results are totally different with that of simply setting child_layer.eval(). May I have your advice on this?