Thanks, in the case of addcdiv, when I do gradgradcheck, that is to calculate the second order of derivatives. I just successfully manually used grad_fn(torch.ones(1, device=‘cuda:0’)) to get the grad to the inputs of this grad_fn. And by looking at the next_functions, and then pass the grad of the inputs from last step as the input to this next_functions’ input, I can manually get the gradients to the leafs eventually. However, for some of the scalars, I have to do a manual tensor.sum() to get the scalar gradients in the end. So I think the grad_fn class does contain the information including what shapes are accepted, what the constants to mul/div are.(e.g. in y = 2 * x, the 2 is a constant)
So my manual calculations using grad_fn does coninside with what should be. But on my machine, the gradgrad results for addcdiv is not all correct. So I’m stuck at how to debug this. In the addcdiv example, the backward-functions are all very simple, like SumBackward, MulBackward, NegBackward, DivBackward, AccumulateGrad. And I already got the graph of how they are chained together. But I don’t know where can I set a breakpoint in the c++/cuda source code in order to debug this. For example, if DivBackward for cuda tensor has some bug, where should I put a breakpoint for this? Much appreciated.