I have 2 different models as shown above and I want to take features from FC layer of each model and concatenate them and pass it through another FC layer and train them. The question I have is if I should have 3 loss functions (one loss function for each network ) or one loss function at the end of the whole network. The concern I have about using one loss function is while backward propagation does the network choose to change the weights of only one network because it might lead to more decrease in loss