Solved by zhl515 in post #2 
              
              
                You should use in-place operation on leaf node of computation graph(W, b are leaf variable) 
    W.sub_(0.001*W.grad)
    b.sub_(0.001*b.grad)

And backward()  accumulates gradients in the leaves - you might need to zero them before calling it.