I don’t know the exact problem but optimizer updates only those parameters which leads to loss computation by some means and whose requires_grad is set to True. Try setting  requires_grad of the parameters you don’t want to update, to False or even detach() those parameters from backward computation graph if you also want layers below it  not to be updated.