ptrblck:

Alternatively, you could clone the gradient after the first backward pass and compare it to the gradient after the second backward call. However, for the sake of debugging I would personally prefer the first approach.


Thanks. But my loss is actually the overall loss, which means that I assume it will remember all the operations done on that parameter.  I’m not sure how I can clone the gradient after the first backward pass…