A more precise mathematical definition would be to say that what we often refer to as “gradients” are Jacobian-vector products, and the gradient is the vector you want to multiply with it (for the loss it’s initialized as [1] by default, so it gives the Jacobian itself).