Solved by KFrank in post #2 
              
              
                Hello Valiox! 

One possibility – this would depend on the details of what you’re 
doing – is that some optimizers have memory, e.g., SGD with 
momentum or Adam.  Therefore, even though the current 
gradient is zero, the optimizer might be remembering previous 
steps with a non-zero gradient, and th…