Oh, I was wrong. My apologies. Adding the backward just slowed the network down so much it just looks like the memory usage was flat, and I was just not paying enough attention - by step 4k the memory was half consumed (doing a backward every 1000).
My issue is basically that I know what I need to remove, I just don’t have any way to sever it completely from the graph as far as I can tell. The links are interesting but are really not quite what I’m looking for. And window[-1] = window[-1].detach() doesn’t do much as far as I can tell, when window is a tensor.
I think I’ve decided what I want basically just can’t be done for some reason. Once an element is put into a tensor, it seems to live in some form in the graph that I can’t purge even when the element is no longer in the tensor.
This would have been convenient for other reasons in my graph, but I think I’ll just have to make do with a list and torch.stack.
Anyway, thanks for the help.