Hi, here is one toy code about this issue:
import torch 

torch.cuda.set_device(3)
a = torch.rand(10000, 10000).cuda()
# monitor cuda:3 by "watch -n 0.01 nvidia-smi"
a = torch.add(a, 0.0)
# keep monitoring

When using same variable name “a” in torch.add, I find the old a’s memory is not freed in cuda, it still exists even though the reference is updated, and I cannot reach tp original memory since the reference is gone. How could I make the memory of old reference automatically freed when use same variable name in left/right of torch operations?