Your described issue is often caused by an increasing computation graph, which would not only increase the computation time but also the memory usage.
This could happen, if you are keeping the computation graph alive by using backward(retain_graph=True), which won’t delete the computation graph after the backward pass.
Often this issue is caused by reusing the same hidden tensors without detaching them. While this could fit your use case, you would have to check “how far” into the past iterations the gradients should be computed.