2

Figure 1: The effect of instruction tuning on MOE models versus dense counterparts for base-size
models (same flops across all models in this figure). We perform single-task finetuning for each
model on held-out benchmarks. Compared to dense models, MoE models benefit more from
instruction-tuning, and are more sensitive to the number of instruction-tuning tasks. Overall,
the performance of MoE models scales better with respect to the number of tasks, than the number of
experts.
Mixture-of-Experts