Hi

Could faulty garbage collection be responsible?

Python is refcounted. So no issue here 
The usual reason this happens is:

You accumulate your loss (for later printing) in a differentiable manner like all_loss += loss. This means that all_loss keeps all the history of all the previous iterations. You can fix it by doing all_loss += loss.item() to get a python number that does not track gradients.
You reuse some Tensor that requires_grad from one iteration to the next. Effectively growing the graph of history needed to compute gradient at each iteration. You need to check elements that are reused from one iteration to the next.
You save some state in a list in a differentiable manner, add .detach() to break the history graph and only store the value of the Tensor (and not all its history).

Let me know if it helps !