then how should I be updating the variables if I want to do it manually? (note I choose SGD as an example since I knew what should happen but I really wanted to play around with different update rules)

Note: I now tried the update rule u suggested, didn’t seem to work:
W.data.sub_(eta*W.grad.data)

I wonder if there is something really small and weird that I am doing that makes code thats seems that it should work not work…