Solved by tom in post #3 
              
              
                Hi, 
for p in model.parameters():
    p = (p - lr * p.grad).detach()

No! Don’t do that! Apologies for the bold font. If you assign p, you will just overwrite the name and the model parameter will be unchanged. 
Now, if you look at the source code of an optimizer, say SGD, you find that the update r…