Finetuning v.s. Instruction Finetuning. To compare the gap between finetuning MoE directly and
FLAN-MOE, we experiment with single-task finetuned MoE, single-task finetuned FLAN-MOE, and
dense counterparts in Figure 6. We perform hyper-parameter search for each finetuning setting.
For the examined Held-Out tasks, we observed that the improvement of FLAN-MOE over finetuning
MoE is noticeably larger compared to the performance gap between FLAN-T5 and T5. This difference
becomes even more pronounced when there is a scarcity of labeled data or when the model size is
increased. These observations confirm the benefits of FLAN-MOE in mitigating overfitting issues
associated with directly finetuning MoE.
Despite their advantages such as increased adaptability and efficiency in managing complex tasks,
MoE architectures are prone to overfitting during the finetuning process, as discussed in citation. This
Mixture-of-Experts