Solved by albanD in post #4 
              
              
                In most cases, you can do without it, but for example, you can replace: 
loss = l1 + 2 * l2
autograd.grad(loss, inp)

by 
autograd.grad((l1, l2), inp, grad_outputs=(torch.ones_like(l1), 2 * torch.ones_like(l2))

Which is going to be slightly faster. 
Also some algorithms require you to compute x * J …