Hello!
I’m currently having some CUDA “out of memory” errors during training after a certain number of iterations, and only when back-propagation is being used and precisely when trying to allocate new tensors in the backward methods of my autograd::Function subclasses. I’m using custom activation functions & layers for which I wrote custom CUDA code which is then wrapped in autograd::Function subclasses. I am 99% sure it is due to memory leaks and not problem size as I did something pretty similar in Python/PyTorch and had no issues at all, and on my C++ code it happens only after a certain number of iterations.
So the thing is that I’d like to know how one should go in general about identifying memory leaks in a libtorch program because right now a direct use of say valgrind doesn’t help me much and debugging/guessing is currently a real pain. Say for example since, from my understanding, a caching CUDA allocator is used by libtorch, I’d like to find (for example during debugging) whether some tensors are indeed being freed (as in their space is marked as “available” by the caching allocator) after they’re being dereferenced, because if they aren’t then that can help me indeed to localize a bug in my code causing those tensors to not be properly dereferenced.
Thank you!