KFrank:

First, are you using non-zero momentum or weight_decay ?


First of all, i am using the momentum in optimizer.
So i understand, the optimizer could update the parameters after i changed the requires_grad=False.
When i check the gradient is “after” calling optimizer.step().
As you mentioned, if the optimizer with momentum modify the gradients of parameters with requires_grad=False, the phenomenon is understanble.
So i will check the gradients before optimizer.step() and will additionally attach the reply.