Solved by ptrblck in post #6 
              
              
                That sounds valid and I don’t know why retain_graph=True would be needed. 
Based on your description you are executing forward passes through the models, calculate losses, accumulate them, and call backward on the final loss. 
I don’t see why the graph should be kept alive and where the “backward th…