Did you find that there were issues with overtraining the network?  For example, say that y1 was very easy to learn, but y3 was hard to learn.  Was there a single loss function being minimized, or was there a separate loss function for each subnetwork?
If there was a single loss function, I worry then that y1 being easy to learn would result in it being overtrained, due to the difficulty in learning y3.