Mr_Tajniak:

What is wrong with this


Please check out the CUDA semantics document.
Instead, torch.cuda.set_device("cuda0") I would use torch.cuda.set_device("cuda:0"), but in general the code you provided in your last update @Mr_Tajniak would not work for the case of multiple GPUs.
In case you have a single GPU (the case I would assume) based on your hardware, what @ptrblck said:

.empty_cache will only clear the cache, if no references are stored anymore to any of the data. If you don’t see any memory release after the call, you would have to delete some tensors before.

This basically means PyTorch torch.cuda.empty_cache() would clear the PyTorch cache area inside the GPU.
You can check out the size of this area with this code:
import torch

import gc
def p():
    c = torch.cuda.memory_cached()
    print(f'cached   :{c}')
    a = torch.cuda.memory_allocated()
    print(f'allocated:{a}')
    f = torch.cuda.memory_cached()-torch.cuda.memory_allocated()
    print(f'free     :{f}')
    
torch.cuda.empty_cache()
p()
r = torch.randn(1, 128).cuda()
p()

Out:
cached   :0
allocated:0
free     :0
cached   :2097152
allocated:512
free     :2096640

See how PyTorch allocated 2Mb of cache just for storing this 128 floats. If you would del r followed by p() the GPU memory will be free again.
If you would have some objects you haven’t deleted make sure you delete them if they are not needed.
Why did PyTorch cached the memory in advance?
To reuse it later. This is the idea of the cache. We assume this precious resource will be used later.