I guess the answer is just have to be careful
if you declare
t = torch.randn((1,3,1024,1024),device="cuda")
f = [t]
def something(f):
  f.mul_(0.001)
  return f
f1 = something(f)

to clear cuda memory you have to
del t
del f
del f1

same goes for tuples, dictionaries. I even tried building my own tensorset class in cpp + pybind, still, as soon as it is used in python, passing it to a function and returning it creates a reference. To clear VRAM you have to delete all references.
If anyone has a better solution, please shout.
Thanks