Hi,
Two things:

We have a custom allocator, so even when the memory is released, you won’t see it available on nvidia-smi but you will be able to use it in pytorch.
The memory is realeased only when you don’t reference it anymore. You might want to wrap the content of your inner loop in a function so that all the intermediary results go out of scope (and are thus released) between loop iterations.