We consider a few small architecture variations here. The ﬁrst modiﬁcation was adding additional
FFN layers (feed-forward network, see Table 1 for more details) immediately before or after each
MoE layer (referred to as Sparse-Dense). Table 16 reveals the effectiveness of an FFN layer im-
mediately preceding or following each sparse layer and that these extra FFN layers help less when
added elsewhere in the network. Guaranteeing all tokens have at least one FFN applied to them
between each attention layer appears useful.

Model
Dense model (baseline)
Dense model w/ extra FFN layers
Sparse model (baseline)
Sparse model w/ extra FFN layer after each sparse layer
Sparse model w/ extra FFN layer before each sparse layer
Sparse model w/ extra FNN layers placed randomly in the network

Neg. Log Perp. (↑)

-1.474
-1.452
-1.383
-1.369
-1.369
-1.376

∆
-

-

0.022

0.014
0.014
0.007
ST-MOE- DESIGNING STABLE AND TRANSFERABLE SPARSE EXPERT MODELS