Thanks for your response. If I run your code I get the exact same results but for some reason nvidia-smi doesn’t seem to notice that the memory was deallocated.
If I run this code
def nvidia_smi():
    out = subprocess.Popen(['nvidia-smi'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    sout, serr = out.communicate()
    res = re.findall("([0-9]*)MiB / 16130MiB", str(sout))
    res = [int(x) for x in res]
    return res

os.environ['CUDA_VISIBLE_DEVICES'] = '2'
print(f'{nvidia_smi()[2]} MiB')

t = torch.zeros((1, 2**6, 2**6)).to(f'cuda')
print(f'{nvidia_smi()[2]} MiB')

del t
print(f'{nvidia_smi()[2]} MiB')

torch.cuda.empty_cache()
print(f'{nvidia_smi()[2]} MiB')

The result is
470 MiB
1473 MiB
1473 MiB
1471 MiB

As the process terminate the used memory goes down to 470 MiB again.
For some reason empty_cache() manages to deallocate 2 MiB (this is consistent and not due to other processes on the same GPU I’ve tried it multiple times). Thinkig about it I guess that those 2 MiB are the size of the tensor I allocate. The other 1001 MiB are probably allocated by the CUDA backend for some internal functioning reason.
I was gonna ask if there’s a way to prevent that but I don’t think so.