Generally, you would want to have autonomous submodels before creating an “ensemble” - you can then take the weighted average of predictions with softmax-ed trained combination weights.
Without this - i.e. if you train everything at once with one objective(loss) across all parts, you’ll just get a higher capacity model where parts (and combination weights) train to mutually compensate output errors from other parts.