Thanks for your reply!
What you mean is that if intermediate_result requires gradients, it will not only maintain two tensors (in intermediate_result currently), and it will also track all the preceding tensors in the computational graph, right? If so, the steadily growing memory usage the a normal case.



 albanD:

What you need to make sure is how big your computational graph is.


As you said, the memory usage increases in the first epoch, and it becomes stable in the following phase.