I removed t".data" and requires_grad = False. No change.
I am measuring memory using ubuntu’s “system monitor” and glances.
I don’t think my code has any states. It has a  couple of static variables other than nn.modules.
The model is mostly sequential convolutions, normalizations, multiplications, sums…etc. Nothing too crazy. It’s not an ensemble or GAN or anything like that.
When training I was using a GPU and it would run iterations including validations for days without issues.

image.png848×243 10.9 KB

This is an example of the first 5 or so iterations. It continues to go up until the process dies from memory issues.