Solved by ptrblck in post #2 
              
              
                If these parameters were updated previously using an optimizer with running estimates, changing the requires_grad attribute to False might still result in updates due to the running estimates of the optimizer. 
If that’s the case, you might need to set these gradients to None, so that they will be s…