Thanks that makes sense conceptually.
Also, now I actually figured out what was wrong with some other code I was talking about. If I change the update rule to:
W = W - eta*W.grad

it doesn’t work. The reason is because the above is actually nested in a loop that fetches parameters from my Sequential model, so doing (first):
#1st
for W in mdl.parameters():
    W = W - eta*W.grad

vs
#2nd
for W in mdl.parameters():
    W.data = W.data - eta*W.grad.data

even if both are conceptually wrong its semantically very different. The first one replaces a temporary variable/name with a new variable. Since the original variable held in mdl is never updated the model looks as if it never trains. Thus the 2nd method makes the model actually “work” and get trained but the first one does not. This is because the second actually changes variables inside of mdl.
I know neither is the way pytorch is meant to work (it seems) but at least I understand the behaviour I am seeing now as I change the lines between:
    for W in mdl_sgd.parameters():
        #print(W.grad.data)
        W = W - eta*W.grad
        #W.data.sub_(eta*W.grad.data)
        #W.data = W.data - eta*W.grad.data

from https://stackoverflow.com/questions/45626848/how-does-one-make-sure-that-the-parameters-are-update-manually-in-pytorch-using#comment78249041_45626848 77