I would guess that providing some kind of information from the higher level models to the sub models might increase the performance. If you are creating a big model with different outputs, you might calculate the losses separately, add them together, and call backward on the final loss.