Aha! This is highly illuminating! Thank you. One thing I don’t understand though: what is the meaning of the argument given to backward when the output leaf (sorry, I don’t know the proper terminology) of the graph is not a scalar? I agree with your comment that having a scalar-valued loss function is what makes sense for deep learning, but it seems to be that the AD framework allows for an output leaf that is vector-valued.