I’m trying to free some GPU memory so that other processes can use it. I tried to do that by executing torch.cuda.empty_cache() after deleting the tensor but for some reason it doesn’t seem to work.
I wrote this small script to replicate the problem
os.environ['CUDA_VISIBLE_DEVICES'] = '0'
showUtilization()
t = torch.zeros((1, 2**6, 2**6)).to(f'cuda')
showUtilization()
del t
torch.cuda.empty_cache()
showUtilization()

The memory utilization grows from 5% to 12% after allocating the tensor and stays to 12% even after emptying the cache.
Of course as the process terminates the memory is released but I’d need to do that while the process is running. Does anyone have any idea about how to solve this?