Very interesting. Adding:
loss = window.mean()
loss.backward(retain_graph=True)

to the loop fixes the leak. Can you (or someone) explain what exactly is happening here?
It’s true that this is a min repro of a bigger codebase (though I did not intend it as pseudocode), but my actual code also does not run backward terribly frequently at the moment, by design, and it would be useful if there was a way I could prevent this memory accumulation between backwards in some way? (Or if it’s unavoidable I’d like to understand why, at least.)
What would the ramifications be of doing backward on some, essentially, fake loss, and then later (when I’m ready) actually doing the full zero_grad/backward/step with my real loss? What is backward(retain_graph=True) actually doing here to free my memory? (I initially assumed retain_graph would mean there was no benefit to running it, but I seem to have been wrong about that.) Is there some less hacky way of doing this?
Regarding your other question: With the detach it does not run out of memory, but it does break the graph chain I’m trying to construct, I’m pretty sure. But anyway the backward seems to have been the issue.
Thanks so much for your insight, I definitely assumed that backward only freed the tensors created by the previous backward; I did not know it also impacted tensors created during the forward pass.