Sparse Mixture-of-Experts (MoE) is a neural architecture design that can be uti-
lized to add learnable parameters to Large Language Models (LLMs) without
increasing inference cost. Instruction tuning is a technique for training LLMs to
follow instructions. We advocate combining these two approaches, as we find that
MoE models benefit more from instruction tuning than dense models. In particular,
we conduct empirical studies across three experimental setups: (i) Direct finetun-
ing on individual downstream tasks devoid of instruction tuning; (ii) Instruction
tuning followed by in-context few-shot or zero-shot generalization on downstream
tasks; and (iii) Instruction tuning supplemented by further finetuning on individual
downstream tasks. In the first scenario, MoE models overall underperform dense
models of identical computational capacity. This narrative, however, dramatically
changes with the introduction of instruction tuning (second and third scenario),
Mixture-of-Experts