can be seen in Figures 6 and 1, where single-task fine-tuned MoE models sometimes underperform
their dense T5 counterparts.
Interestingly, compared to dense models, MoE models derive greater benefits from instruction-tuning
and are more sensitive to the number of instruction-tuning tasks. In general, MoE model performance
scales better with respect to the number of tasks rather than the number of experts. We hypothesize
this is primarily due to the specialized nature of individual experts, which can lead to heightened
sensitivity to noise and limited generalization capabilities when exposed to unseen data.
Mixture-of-Experts