Im rewriting my question because I have the answer.
Problem: if you have declare cuda tensors in python and you pass them to functions using containers, and then need to delete them, you have to delete all references. see example below
or look at the gist

  
      gist.github.com
  
  
    https://gist.github.com/xvdp/5ea32d4baff801c8d80960a8fa7b4594 16
memory_tests.py
"""testing vram in pytorch cuda
every time a variable is put inside a container in python, to remove it completely
one needs to delete variable and container,
this can be problematic when using pytorch cuda if one doesnt clear all containers
Three tests:

>>> python memory_tests list
    # creates 2 tensors puts them in a list, modifies them in place, deletes them
    # in place mod changes original tensors
    # list and both tensors need to be deleted
This file has been truncated. show original