1.7
3.4

2.2
3.4

1.6
3.4

1.8
3.4

Encoder
Decoder
Table 14: Entropy of routed sentinel tokens across encoder and decoder layers. We support our
qualitative observation that encoder experts specialize, but decoder expert don’t by computing the
entropy over the routing for sentinel tokens. The encoder routing entropy is low, but the decoder
router is high entropy, and nearly equal to uniform routing. Because each layer has 32-experts, a
completely uniform distribution has entropy of 3.5.

3.5
3.5

1.2
3.4

1.7
3.4

20

7.3 MULTILINGUAL EXPERTS SPECIALIZE, BUT NOT BY LANGUAGE

We next consider a multilingual sparse model pretrained on a mixture of different languages and
inspect the expert specialization in the encoder. As in the monolingual case, we ﬁnd strong evidence
of expert specialization. Table 15 presents some examples of experts specializing in sentinel tokens,
numbers, conjunctions & articles and proper names.

Expert specialization
Sentinel tokens

Routed tokens

Numbers
ST-MOE- DESIGNING STABLE AND TRANSFERABLE SPARSE EXPERT MODELS