Solved by albanD in post #4 
              
              
                So first, you never want to use .data. 
If you want to do your weight update manually, you can do: 
loss.backward()
with torch.no_grad():
  w1 -=  learning_rate * w1.grad