Solved by ptrblck in post #2 
              
              
                The memory should be used by the CUDA context, which is initialized with the first CUDA operation, e.g. creating a tensor. 
You cannot release it during the runtime of your script.