If you are checking the memory via nvidia-smi, note that the CUDA context will also use memory.
The allocated and cached memory will be freed using your code snippet:
print(torch.cuda.memory_allocated())
print(torch.cuda.memory_cached())
> 0
> 0

a = torch.rand(10000, 10000).cuda()  
print(torch.cuda.memory_allocated())
print(torch.cuda.memory_cached())
> 400556032
> 400556032

del a
print(torch.cuda.memory_allocated())
print(torch.cuda.memory_cached())
> 0
> 400556032

torch.cuda.empty_cache()   
print(torch.cuda.memory_allocated())
print(torch.cuda.memory_cached())
> 0
> 0