Hi Doyup!



 LeeDoYup:

I also checked that the requires_grad=False but, still there are some values in weight.grad.


First, are you using non-zero momentum or weight_decay?
I should have asked my question about non-zero gradients more
carefully:
I’m reasonably sure that SGD implements weight_decay (and
maybe momentum) by modifying the gradients before applying
the actual update step – even if the gradients are zero, or have
requires_grad = False.
So, I should aks:  Are the gradients in question zero after calling
loss.backward(), but before calling optimizer.step()?
Best.
K. Frank