Failure Cases. The fine-grained specialization of
FLAN-MOE models, particularly when fine-tuned on
English-only instructions, can inadvertently lead to a
narrowing of the model’s capacity to effectively pro-
cess and generate content in multiple languages. We
found all the FLAN-MOE perform poorly on multi-
lingual benchmarks including TyDiQA and MGSM.
Even the largest FLAN-ST32B only achieves 15.5%
on MGSM and 25.1% on TyDiQA, which is only
comparable to the vanilla PaLM62B with 18.2% on
MSGM, and PaLM8B with 25.0% on TyDiQA. It also
underperform FLAN-PALMvariants. We hypothe-
ses that this issue may stes from the model’s over-
optimization towards the specificities of the English
language during finetuning, which can impede its
ability to navigate the complexities of other languages. Consequently, while MoE models offer
significant benefits in terms of task-specific adaptability and efficiency, their potential shortcomings
Mixture-of-Experts