This function finds the dummy tensors that still stuck in the GPU memory and not cleared:
def pretty_size(size):
    """Pretty prints a torch.Size object"""
    assert(isinstance(size, torch.Size))
    return " × ".join(map(str, size))

def dump_tensors(gpu_only=True):
    """Prints a list of the Tensors being tracked by the garbage collector."""
    import gc
    total_size = 0
    for obj in gc.get_objects():
        try:
            if torch.is_tensor(obj):
                if not gpu_only or obj.is_cuda:
                    print("%s:%s%s %s" % (type(obj).__name__, 
                                          " GPU" if obj.is_cuda else "",
                                          " pinned" if obj.is_pinned else "",
                                          pretty_size(obj.size())))
                    total_size += obj.numel()
            elif hasattr(obj, "data") and torch.is_tensor(obj.data):
                if not gpu_only or obj.is_cuda:
                    print("%s → %s:%s%s%s%s %s" % (type(obj).__name__, 
                                                   type(obj.data).__name__, 
                                                   " GPU" if obj.is_cuda else "",
                                                   " pinned" if obj.data.is_pinned else "",
                                                   " grad" if obj.requires_grad else "", 
                                                   " volatile" if obj.volatile else "",
                                                   pretty_size(obj.data.size())))
                    total_size += obj.data.numel()
        except Exception as e:
            pass        
    print("Total size:", total_size)

and here’s the output:
Tensor: GPU pinned 
Tensor: GPU pinned 32 × 3 × 3 × 3
Tensor: GPU pinned 32
Tensor: GPU pinned 32 × 32 × 3 × 3
Tensor: GPU pinned 32
Tensor: GPU pinned 64 × 32 × 3 × 3
Tensor: GPU pinned 64
Tensor: GPU pinned 64 × 64 × 3 × 3
Tensor: GPU pinned 64
Tensor: GPU pinned 128 × 64 × 3 × 3
Tensor: GPU pinned 128
Tensor: GPU pinned 128 × 128 × 3 × 3
Tensor: GPU pinned 128
Tensor: GPU pinned 256 × 128 × 3 × 3
Tensor: GPU pinned 256
Tensor: GPU pinned 256 × 256 × 3 × 3
Tensor: GPU pinned 256
Tensor: GPU pinned 4096 × 36864
Tensor: GPU pinned 4096
Tensor: GPU pinned 5 × 4096
Tensor: GPU pinned 5
Tensor: GPU pinned 5 × 3 × 256 × 256
Tensor: GPU pinned 5
Tensor: GPU pinned 5 × 5
Tensor: GPU pinned 
Tensor: GPU pinned 32 × 3 × 3 × 3
Tensor: GPU pinned 32
Tensor: GPU pinned 32 × 32 × 3 × 3
Tensor: GPU pinned 32
Tensor: GPU pinned 64 × 32 × 3 × 3
Tensor: GPU pinned 64
Tensor: GPU pinned 64 × 64 × 3 × 3
Tensor: GPU pinned 64
Tensor: GPU pinned 128 × 64 × 3 × 3
Tensor: GPU pinned 128
Tensor: GPU pinned 128 × 128 × 3 × 3
Tensor: GPU pinned 128
Tensor: GPU pinned 256 × 128 × 3 × 3
Tensor: GPU pinned 256
Tensor: GPU pinned 256 × 256 × 3 × 3
Tensor: GPU pinned 256
Tensor: GPU pinned 4096 × 36864
Tensor: GPU pinned 4096
Tensor: GPU pinned 5 × 4096
Tensor: GPU pinned 5
Tensor: GPU pinned 
Parameter: GPU pinned 32 × 3 × 3 × 3
Parameter: GPU pinned 32
Parameter: GPU pinned 32 × 32 × 3 × 3
Parameter: GPU pinned 32
Parameter: GPU pinned 64 × 32 × 3 × 3
Parameter: GPU pinned 64
Parameter: GPU pinned 64 × 64 × 3 × 3
Parameter: GPU pinned 64
Parameter: GPU pinned 128 × 64 × 3 × 3
Parameter: GPU pinned 128
Parameter: GPU pinned 128 × 128 × 3 × 3
Parameter: GPU pinned 128
Parameter: GPU pinned 256 × 128 × 3 × 3
Parameter: GPU pinned 256
Parameter: GPU pinned 256 × 256 × 3 × 3
Parameter: GPU pinned 256
Parameter: GPU pinned 4096 × 36864
Parameter: GPU pinned 4096
Parameter: GPU pinned 5 × 4096
Parameter: GPU pinned 5
('Total size:', 457558416)

sounds like the model’s parameters and tensors are not freed. Any suggestions i can do to avoid this?