have sought to enhance the gating mechanism itself [15, 24, 43, 55], MoE models have also been
explored in the context of multitask learning [15, 22]. Typically, a shared pool of experts is used,
although there has been investigation into per-task routers [30]. This essentially permits an input to
choose the most relevant expert(s) for a given task, thereby optimizing the processing and results.
Nevertheless, the instability of MoE models during fine-tuning or multitask learning has consistently
been a challenge. Our study aims to investigate whether instruction fine-tuning with scaled tasks
might contribute to mitigating the generalization issues inherent to MoE models.
Mixture-of-Experts