albanD:

MB.next_functions


This explanation solves a lot of my problems, thanks. But I still got a little bit of confusion. Let’s come back to the toy example. I understand now that we can trace back from the next_functions graph-sort of thing to the x. But as we know in this example, x.grad will be a 4x4 matrix of '2’s. How can we retrieve this number ‘2’ information from the backward pass? For example, in the MulBackward, which correponds to y = x*2. How does it know we are using what to multiply what? The x can be further traced from the AccumelateGrad.x, but where is the information of 2? Since MB.next_functions[1] is just (None, 0). There is no Function subclass in it anymore.
So, my actually final quesion is how can we mannually use the z.grad_fn instance to calculate the grads for all the inputs. I just need to see the actual dataflow. Not only the backward functions called, you still need to store the inputs to the backward functions, I think. But I don’t know where to find it.
Another question is can we directly call the grad_fn() to calculate something?
I tried to use grad_fn(), and I clealy see that if I don’t pass the correct shape, it will throw an error. So the Function subclass instance does store some shape information, right? But I don’t see how to access it.