You could try to scale the activations from the left branch to try to force the model to focus more on this branch. However, this would mean that the magnitude of the input features would directly correspond to the importance of these features.
Do you know any publications for this claim?