Solved by MariosOreo in post #3 
              
              
                Hi, 
Before you call .backward(), the gradient of each tensor which requires_grad=True are all None. 
Like the case you posted, you could calculate a.grad firstly and then zero_() its grad. 
In opt.zero_grad() it declare explicity: 
    def zero_grad(self):
        r"""Clears the gradients of all op…