Solved by baromri in post #8 
              
              
                @albanD, I add the same issue and when setting param.grad = None it force the optimizer to skip this param update at step. Thus, also optimizers with momentum will keep those layers params fixed.