Hi. I am wondering if it is possible to train the batchnorm2d layer using a very low learning rate?
Assuming that we are to train an CNN with a batchnorm2d layer, i.e. bn_layer.
To freeze the bn_layer from learning in the training stage, are the following two settings equivalent?
Setting 1 : let the bn_layer in evaluation mode

bn_layer.eval()

Setting 2 : let the bn_layer in training mode

bn_layer.train()
bn_layer.momentum = 0
bn_layer.weight.requires_grad = False
bn_layer.biast.requires_grad = False

Based on my understanding, both settings should be the same, thereby returning the similar results. Unfortunately, the setting 2 doesn’t freeze the bn_layer from learning at all.
However, I observe all parameters and running statistics including running_mean, running_var, weight, and bias are remained unchanged.
Any reason why the two settings are different?
Any code snippets if only weight and bias are trained? but fixing both running statistics unchanged?
Thank you.