Can I ask one more question?
If I understand correctly from here 8, the backward creates a directed graph that will do the computations and give the output and I think that even though retain_variables are set True, the graph is not retained. Now the docs page of autograd 1 show that there will a retain_graph option in the new version of backward(I think). Do you know if that will help? I know I still have to call multiple backwards, but will that make consecutive backwards quicker?