Thanks for your reply.
I guess I was buried in the idea that we usually only want the derivatives with respect to the learnable parameters but did not think that sometimes we might want derivatives with respect to intermediate results too. Thanks for the help  Another related question, in cases when two input tensors have identical shapes, how does grad know which inputs we are specifying? Does it look at grad_fn to do that?