Make sure to zero the gradients (if x.grad: x.grad.zero_()) before calculating new y.
Some more info:

  
    
    
    Why do we need to set the gradients manually to zero in pytorch? 
  
  
    y.backward() doesn’t just assign the value of y’(x) to  x.grad (say y depends on x). It actually adds y’(x) to the current value of x.grad (think it as x.grad += true_gradient). 
In the following example, y.backward() is called 5 times, so the final value of x.grad will be 5*cos(0)=5. 
import torch
from torch.autograd import Variable

x = Variable(torch.Tensor([[0]]), requires_grad=True)

for t in range(5):
    y = x.sin() 
    y.backward()
    
print(x.grad) # shows 5

Calling x.grad.data.zero_…