SingleShockControl:

So you’re saying that some extra memory is allocated to the autograd graph gradients that is never released? And cannot be released in any way?


Some extra memory is allocated during forward to be able to compute the backward yes.
This memory is released when all the references to it are dead (in general when your output/loss goes out of scope).
On top of that, the GPU memory is cached and is not returned to the OS even though it is available to allocate more Tensors within pytorch.