Solved by ptrblck in post #4 
              
              
                torch.autograd.grad should remove all intermediate tensors after its operation, if retain_graph is not set to True (in the same way tensor.backward() operates).