Hi Kushaj,
Thanks for the prompt reply.
However, in my case, I think the memory footprint can be further optimized.
Basically, I have a function f, that I compute gradient g. Then I compute gradient of g with regard to different vectors in the for loop.
I believe that each iteration in the for loop depends on the computation of g, but doesn’t depend on each other. However, pyTorch is asking me to retain_graph for all computations, which I think is not necessary.
Do you see the special situation I am having here?
Fei