Variables have backward references. So conceptually a list of “producers” not “consumers”. The reference chain is Variable -> Function -> … -> Function -> (root) Variable. The function objects also have references to tensors needed to compute derivatives: So for the matrix-multiply of Variables C = A @ B, C.creator (a Function) has references to A.data and B.data (tensors).


The gradients of intermediate Variables are accumulated in a C++ grad_buffer 78 object. It’s not exposed in Python except indirectly through gradient hooks. The .grad of intermediate Variables is always None.


I the ordering of “hooks registered by modules” and “registered by variables” is swapped, but I’m not 100% sure. (Test it)


Some of the automatic differentiating behavior will change a bit in an upcoming PR 107