Gradient equal to zero doesn’t stop optimizer inertia (momentum and so on).
make param.grad=None to avoid that.
In short:
for SGD for example if gradient is zero there is still an update.
imagen678×506 28.8 KB