exactly! that is what I was trying to do! and as you mentioned a manual check of this operation is a must.
about the second part of the question I think now I can formulate it in a better way:
to avoid grpah_retain=True, especially for big size models, it is possible to run loss2.backward() only on a subset of the model’s leaves. I think this helps us avoid allocating memory for retain_grad & avoid the wasted effort of calculating their gradient and then setting them back to 0 with optimizer2.zero_grad().