I wish to finetune pretrained networks with batchnorm layer for fully convolutional networks. Because of  the limit of GPU memory, we often use batchsize=1 when training FCN. So I wish to freeze the batchnorm parameters (including BN weight and bias, and the running mean and variance). I set the value of requires_grad as False for all the BN layers.
However, when I start training the FCN, the running mean and variance change after one iteration. So what should I do to freeze the running mean and variance during finetuning?