4 Discussion

4.1 Finetuing Strategy

Sparse models have performed remarkably well in the regime of large datasets, but have sometimes
performed poorly when finetuning data is limited [56, 12]. Instruction finetuning can also be viewed
as a continual finetuning stage, so we present a detailed study about how different factors impact the
instruct finetuning performance of FLAN-MOE and offer a practical recipe. All the discussion here is
based on instruction finetuning FLAN-ECBASE/FLAN-STBASE for 100k steps.
Mixture-of-Experts