+1%
+1%

+3%
+2%
+10%
+1%
+5%
+3%
−2%
+4%
−2%
+10%
+2%

+8%
+6%
+19%
+13%

+20%
+8%
+9%

Table 10: Fine-tuning performance of FLOP-matched dense and sparse models. Comparison
of the dense-L baseline and the sparse FLOP-matched version (higher numbers better). We observe
consistent gains across diverse tasks, using approximately the same amount of computation. The
only two tasks without improvement from the sparse model are the two smallest: CB with 250
training examples and WSC with 259.
ST-MOE- DESIGNING STABLE AND TRANSFERABLE SPARSE EXPERT MODELS