Hi,
for 1, yes, that would be the general idea.



 Frazer:

And if I print out the grad_output value, it will be 1 if this is the final scalar loss layer, right ?


It is a good idea to multiply grad_output[0] to the input gradient you return to be consistent here, even if you envision that it usually is one. You might have a use case where you want to weight your loss with others and then you’ll be happy if it just works.
A final advice about implementing Functions that I forgot: There is a function autograd.gradcheck that will test your implementation (e.g. code cell 7 in the implicit function example) of the gradient against the numeric derivative. It’s always good to use that, but you might get false positives due to numerical errors (or extreme nonlinearity).
For 2:



 Frazer:

When do we have to write the backward function ourselves ?


You only write the backward function if you can’t express what you want to compute with what’s already there. So in your example you did the right thing by just writing a nn.Module and have the backward be calculated step by step automatically.
I rarely find myself missing many things.
In the two examples, implicit functions use an iteration to find the value and that is not good to backpropagate through naïvely. For the Cholesky decomposition, the situation is somewhat similar, although it is mainstream enough to eventually be included in pytorch properly (I guess, other libs also have it) after solving the performance issue in the backward.
Best regards
Thomas