I tried using:
import gc
for obj in gc.get_objects():
       if torch.is_tensor(obj) or (hasattr(obj, 'data') and torch.is_tensor(obj.data)):
            print(type(obj), obj.size())

After inspecting the resulting output, I see that two variables are increasing in their number over iterations:
Iterations 1-2:
0 mentions of: (<class 'torch.Tensor'>, (3723264,))
7 mentions of (<class 'torch.Tensor'>, ())
Iterations 2-3:
3 mentions of (<class 'torch.Tensor'>, (3723264,))
11 mentions of (<class 'torch.Tensor'>, ())
Iterations 4-5:
8 mentions of (<class 'torch.Tensor'>, (3723264,))
17 mentions of (<class 'torch.Tensor'>, ())
Iterations 34-35:
64 mentions of: (<class 'torch.Tensor'>, (3723264,))
77 mentions of: (<class 'torch.Tensor'>, ())
Iterations 35-36:
70 mentions of (<class 'torch.Tensor'>, (3723264,))
79 mentions of (<class 'torch.Tensor'>, ())
Iterations 47-48:
94 mentions of (<class 'torch.Tensor'>, (3723264,))
103 mentions of (<class 'torch.Tensor'>, ())
Iterations 96-97:
192 mentions of: (<class 'torch.Tensor'>, (3723264,))
201 mentions of: (<class 'torch.Tensor'>, ())
I’m not entirely sure what I happening, but I suspect this might be the cause of my issue. I wish I could see exactly which Variables are responsible for (<class 'torch.Tensor'>, (3723264,)), and (<class 'torch.Tensor'>, ()).
Removing my print and save functions do nothing to stop the issue.