Well, the usual thing to do is .detach() 10 steps before the end.
This won’t give you overlapping windows to backprop through, but it’s rather common (e.g. fast.ai’s language models do this and Jeremy explains it in the corresponding lecture).
Best regards
Thomas