55, 56]

5

Arch:Type:10¤1100101GFlops Per Token Prediction20304050Avg. Held-Out ScoreMMLU-Direct 0Shot10¤1100101GFlops Per Token Prediction202530354045Avg. Held-Out ScoreBBH-Direct 0ShotFigure 3: Learning efficiency comparison. Average zero-shot, and few-shot performance of FLAN-
MOE models versus FLAN-T5 dense models as more tokens are processed during training on FLAN
Tasks.

experts continues to grow, the performance gains begin to diminish, eventually reaching a point of
saturation for BASE-sized model.
Mixture-of-Experts