As a quick illustrative example, let’s start with two abstract functions, c = f(a, b), and e = g(c, d).
If we know dg/de and dg/dd and dc/da and dc/da, we can compute dg/da and dg/db, because of the classic chain rule:
de/da = (de/dc)(dc/da) + (de/dd)(dd/da) = (de/dc)(dc/da)
de/db = (de/dc)(dc/db) + (de/dd)(dd/db) = (de/dc)(dc/db)
Note that de/dc will have an explicit concrete value without needing to know dc/da or dc/db as we traverse backward through the graph. This de/dc is the grad_output as we view the computation graph from the perspective of f(a,b). We multiply this value with dc/da to get de/da and dc/db to get de/db. In the concrete case where f(a,b), this would be multiplying with b to get de/da and multiplying with a to get de/db. Note that this pattern is general. It does not matter if the graph extends past e(c, d) with more functions, or if there are previous functions before c = f(a, b), because this pattern can be extended until all gradients are computed.
You might want to take a look at some references on reverse-mode automatic differentiation, which is the typical way this is done on most frameworks (Reverse Accumulation) in this article : Automatic differentiation - Wikipedia 2. The CS231n also has some great hands on examples with a “real” computation (matmul) at the end. CS231n Convolutional Neural Networks for Visual Recognition 2