Unfortunately, I suspect that will not suffice. I need to modify the way pytorch is computing these gradients in the graph for a given mini-batch of data.
What I mean is, I would ideally want to buffer up some gradients (corresponding to data in a mini-batch) right before it back propagates those through layer i. Hence, just like the forward pass has been explicitly exposed (meaning that you can see how the data is flowing through the layers), I would ideally like to see the backward pass too.