So I tried what you suggested and just always created a random tensor (in each iteration a new random tensor is created) with torch.randn of the same shape as the data. Then making this random tensor positive definite by matrix multiplying it with its transpose and then computing torch.cholesky of that random matrix instead of calculating torch cholesky of the data. And the result in terms of memory usage is the same: I.e., the memory usage still increased every epoch.
And in case that I comment out that torch.choleksy call, the memory usage does still increase but only marginally and only every 20 epochs or so and orders of magnitude smaller. So to me it seems like torch.cholesky was the culprit.
Or could you imagine that actually another error is the root of all evil and is only enhanced by the torch cholesky call? I.e., could you imagine that still one of the other possible issues you mentioned could be at the heart of the problem?