Based on your post I went back and took another look at it, and I narrowed down my issue. I was wrong in the memory not being freed it is instead the eval mode of my code that is requiring more memory than my training mode.
The problem is that in eval mode I’m still running the code with autograd, which is needed since I explicitly need to get gradient information even in eval mode. However I guess this gradient information is accumulating, so I need a way to zero it after each iteration.
Using zero_grad on the optimizer doesn’t solve the problem, guess I need to use it on each variable individually