Thank you very much. I tried this. Setting them to requires_grad = False in the initial training causes the batch norm layers to not train and they later can be trained for finetuning with requires_grad = True.
However, I noticed that even if the batch norm layers are not being trained with requires_grad = False, I think all the BN gradient calculations are still happening and a lot of memory is being occupied. In fact, I cant train a complete model because I am getting out of memory error very often. I am guessing if there is a way to shut down the mean and variance calculations.
BTW, I was using the SyncBatchNorm class because I am using 2 GPUs for training with DataParallel.