I would recommend to check the optimizer first and see if its using internal running estimates (e.g. as Adam does).
If that’s the case, the parameters might be updated regardless of their gradients, and you should set the gradients to None.
Let me know, if that helps somehow.