The whole point of autograd.grad is that you specify what you want gradient for. If you want all, use .backward(). Otherwise how do you know which tensors the gradients returned by autograd.grad correspond to?