thanks so much for your prompt reply!
I understand what you mean. So this is one of the cases where “retain_graph = True” cannot be avoided.
However for me this does not work. Any batchsize at a certain point of the training (few to many k iterations leads to GPU OOM, for a model which requires 3 Gb on average without retain graph.
I have tried to avoid all memory leaks, and still the GPU is full when using retain graph. Why does the network grow so much in size? and why does it keep track of all previous iterations of the network? if I can ask?
Additional question: what is the recommended way to analyze tensor and graph memory usage for debugging?
Thanks a lot!