16

ST-MoE-L (↑) Gain (%)

Metric
Name
F1
SQuADv2
acc
SQuADv2
avg
SuperGLUE
acc
BoolQ
acc
Copa
acc
RTE
acc
WiC
F1
MultiRC
acc
WSC
acc
ReCoRD
acc
CB
ROUGE-2
XSum
ROUGE-2
CNN-DM
acc
WinoGrande (XL)
acc
ANLI (R3)
acc
ARC-Easy
ARC-Challenge
acc
Closed Book TriviaQA acc
acc
Closed Book NatQA
Closed Book WebQA
acc

Split Dense-L (↑)
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev

94.0
87.6
85.1
87.1
83.0
91.0
70.4
83.9
95.2
85.7
100
19.9
20.3
75.4
54.3
63.5
50.2
28.1
27.2
30.5

94.5
88.1
87.4
88.6
91.0
92.1
74.0
86.0
93.3
88.9
98.2
21.8
20.7
81.7
57.3
75.4
56.9
33.8
29.5
33.2

+1%
+1%

+3%
+2%
+10%
+1%
+5%
+3%
−2%
+4%
−2%
+10%
+2%

+8%
+6%
+19%
+13%

+20%
+8%
+9%
ST-MOE- DESIGNING STABLE AND TRANSFERABLE SPARSE EXPERT MODELS