Thanks for showing that piece of code. It’s very nice to know where that happens. But that was already pointed by @ptrblck. Now I am wondering if there is a way to free these tensors (the ones saved. for backwards) and also (if possible) the grad_output somewhere inside the backward function.
One of the issues I face is that the calculations I do in the backward use a lot of memory. It is not really that much considering only a single layer. But, considering that some CNN architecture can get really deep and wide, I just run out of memory in such scenarios. I am looking for releasing those tensors right after using then, while still having other computations left.
Maybe my only hope for those scenarios is to be heavily based on in-place operations.