albanD:

Python is refcounted.


I understand that this guarantees no leaks, but does that guarantee that GC will always be on time?



 albanD:

You accumulate your loss (for later printing) in a differentiable manner like all_loss += loss .


All of my loss accumulation is already using loss.item().



 albanD:

You save some state in a list in a differentiable manner


I was actually guilty of this. I am training a ‘hard’ attention recurrent model that directs a ‘window’ around an image to produce ‘glimpses’ 2, and I was storing the past glimpse locations in a memory tensor. I instantiated this memory with torch.zeros(), which sets requires_gradients to False by default, but I didn’t realize that setting an element of this memory with a non-detached tensor would attach the entire memory tensor. I now call .detach() on the input tensor before inserting it into memory. I also posted about this 42 on the repo I based my code on for anyone curious.
I’m restarting training now, and I’ll report back on whether this helped once training is over. Each epoch takes 30+ minutes so it will be a while . Thanks for the help, saved me a day or two of pulling my hair out.