I want to train fully convolutional networks for pixel-wise semantic segmentation. And I want to use vgg_bn or resnet (which have batchNorm layers in them) as initialization. However, training FCN is memory extensive, so batchsize = 1 is often chosen. So finetuning the BatchNorm layer is senseless and I wish to freeze the parameters and moving averages of the BN layers during finetuning.
My question is how to realize this.
I have tried setting the requires_grad as False for all BN layers, but the moving averages changes after each forward propagation.
Anyone can help?