That sounds valid and I don’t know why retain_graph=True would be needed.
Based on your description you are executing forward passes through the models, calculate losses, accumulate them, and call backward on the final loss.
I don’t see why the graph should be kept alive and where the “backward through the graph a second time” issue would be raised.