Solved by tom in post #2 
              
              
                You win the “exceptionally bad luck in picking a small example” award . 
Your parameters do change, just your loss does not. 
So what happens is that the gradient of (x-1)**2 is 2*(x-1) and with an SGD lr or 1, you get an oscillation between your initial value and 2-x which both happen to have the s…