Hi,
I was wondering who have “memory rights” on tensors created in a pytorch Cuda extension.
I have created several custom cuda extension following this tutorial: https://pytorch.org/tutorials/advanced/cpp_extension.html.
I am now using those extensions in a training loop (i.e. a network is using some of my custom operations) and I noticed that the memory usage strictly increase at each training iteration. I think it might come from those custom operations so I would like to know:


If a tensor is created in Cuda code binded to python and then returned by the custom function, if then the python script delete this tensor, does it effectively clear the memory?


What about if a tensor is created in Cuda code but just contains intermediate results and so is not return by the Cuda function, is it deleted once we exit the function scope like in Python or do we have to delete it manually? If so and if the answer to the first question is yes, is the recommended practice to return all created tensors to then delete it in python or to handle those in the Cuda code itself?


I hope my question is clear, thank you for your help in advance.
Samuel