Solved by tom in post #6 
              
              
                Then I would try something along the lines that you suggested. 
Usually I advise against using .backward for anything except model parameters (and use torch.autograd.grad instead, but this seems to be the exception to the rule because we don’t want to duplicate the work of doing grad and backward. 
S…