Hi, I just read through this clear and straightforward tutorial on defining custom Autograd functions in Pytorch:
https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html 5
The code is very clear, but one thing I’d like to clarify on is the comment in the code that says “In the backward pass we receive a Tensor containing the gradient of the loss
with respect to the output, and we need to compute the gradient of the loss
with respect to the input.”
Shouldn’t it be “we need to compute the gradient of the output
with respect to the input”?
I’m confused how we could define the gradient of the loss with respect to the input, since typically we think of the gradient of a function as the gradient of that function’s output with respect to its input.
Forgive me if I misunderstood, but I’m also somewhat confused about what exactly is referred to by “gradient of the loss with respect to the output”. In the formulation of backprop that I’m familiar with, the function’s gradient is simply multiplied to the recursive sum of forward gradients (to put it into very loose English). I could give a very short and hopefully simple formal summary of that formulation if needed; it would definitely help me to understand how Pytorch’s autograd fits into my existing understanding.
Thanks so much for any further clarifications!