Hello Valiox!



 Valiox:

I zero out the gradient and step
…
Even though the gradient was 0, the tensor values changed a bit:


One possibility – this would depend on the details of what you’re
doing – is that some optimizers have memory, e.g., SGD with
momentum or Adam.  Therefore, even though the current
gradient is zero, the optimizer might be remembering previous
steps with a non-zero gradient, and therefore still be changing
the parameters.


And the gradient is no longer 0, but very small values:


I don’t think that momentum use in an optimizer changes the
gradient.  (Hypothetically, it could be implemented that way,
but I don’t think it is.)
However, at the risk of being completely wrong, I seem to recall
that weight decay might be implemented by actually changing
the gradient.  So turning on weight decay in your optimizer could
explain both the changing parameters and the non-zero gradient.
(!!! Possible misinformation alert !!!)


I this normal behaviour? Why were the tensor values changed at all?


I could see your parameter values changing (under the right
conditions) if your optimizer uses momentum.  My speculation
about the non-zero gradient is more suspect.
Would it be possible to re-run your test using plain-vanilla
torch.optim.SGD, making sure that momentum and
weight_decay are turned off?  (I believe that they are off
by default.)
Best.
K. Frank