Consequently, the overall model can demonstrate a higher degree of adaptability and precision in
tackling diverse and complex tasks. We also observe that after instruction-tuning, the MoE models
exhibit better expert usage, which may help prevent the expert collapse for generalization after
instruction-tuning as in [57].
Mixture-of-Experts