I’m not sure anything in your code immediately jumps into the eye that is bad.
So there a couple of things to keep in mind here:


PyTorch has a caching GPU memory allocator, this means that it will not return memory to the system when freeing tensors but instead keep it and re-use it for the next tensor (but torch.cuda.memory_allocated will give you the memory that is currently allocated to tensors. You can try releasing cached memory (with torch.cuda.empty_cache 1), but there are circumstances in which it is impossible to release all unused memory from the cache.


There are two typical things where you would see something like this:


When doing something like
   a = torch.randn(5, 5, device="cuda")
   a = torch.randn(5, 5, device="cuda")

the first tensor a will only go be freed after the second line has been completed (if there were an exception, you would still have the old value for a). This means that while the second line is executed, both tensors will be there at the same time. Similar effects can also happen with more complex code.


The caching and re-using can have the effect that after running the first model, there are “gaps” from releasing unused tensors that are too small for the tensors of the second model (e.g. because part of the memory is used by another tensor that was allocated in between). This is a bit of “bad luck” and often such effects get smaller over time (because we do not have bad luck all the time).




So I am not sure you have a memory leak here, from your description it might be benign.
Best regards
Thomas