Dear @evcu, in your example, loss is scalar and parameters is a vector.  However, the question is for multiple losses, and computing these without summation.  For me, @smth’s reference to torch.autograd.grad wasn’t helpful either.  Its docs start with: “Computes and returns the sum of gradients of outputs w.r.t. the inputs”, and that seems to be what it does.  Under what configuration does grad return individual gradients (at the expense of memory)?
As aside, This is a pretty useful feature for differential privacy.