Do you need to call backward at some point and want Autograd to use all computation graphs?
I’m not familiar with your use case, but for RNNs you could detach the last state(s) so that the backward pass only calculates the gradients for the current step.