Solved by albanD in post #2 
              
              
                Hi, 
I’m afraid it cannot be as .backward() accumulates in the .grad field all the time. 
You can avoid this though by computing gradients as: 
grad = torch.autograd.grad(loss, model.parameters()).