Raphikowski:

Or could you imagine that actually another error is the root of all evil and is only enhanced by the torch cholesky call?


Given that you still see issues without it, it is possible yes.

Does calling gc.collect() at every iteration solves the issues (both with and without the cholesky)?
What do you use when you remove the cholesky, do you link the output with the input with a differentiable manner? That could explain the reduction in memory when you remove it.
Do you use .backward(create_graph=True) ?