I think I’m missing something in my understanding of the CUDA memory management. I was under the impression that if one deletes all references to objects that were stored on a GPU and subsequently frees the cache, the allocated memory should be zero.
Since my code is part of a larger project and I was until now unable to reproduce the behaviour with a minimal example, I’ll show you a simplified version of what my code is doing. Consider the following snippet:
import torch
from torchvision.models import vgg19

device = torch.device("cuda:0")

def display_memory():
    torch.cuda.empty_cache()
    memory = torch.cuda.memory_allocated(device)
    print("{:.3f} GB".format(memory / 1024 ** 3))

def fn():
    input = torch.rand(16, 3, 512, 512, device=device)
    model = vgg19().features.to(device)
    return model(input).detach()

display_memory()

output = fn()
print(output.size())
display_memory()

del output
display_memory()

This prints the following:
0.000 GB
0.009 GB
0.001 GB

As expected the detached output adds almost no extra memory. After the del statement the memory is almost completely freed. Since this could be related to my later question: Can someone explain the remaining 1 MB after all Tensors and nn.Modules are deleted?
For my actual code the following is displayed if I insert the display_memory() function at the appropriate places:
0.000 GB
0.956 GB
0.946 GB

Even after deletion of every Tensor a large part of the memory remains occupied.
Can someone think of something what I could be doing wrong?