7.1 ENCODER EXPERTS EXHIBIT SPECIALIZATION

Our ﬁrst observation is that, at each layer, at least one expert specializes in sentinel tokens (mask to-
kens that represent blanks to ﬁll-in). Additionally, some encoder experts exhibit clear specialization,
with some experts primarily operating on punctuation, verbs, proper names, counting, etc. Table 13
presents a few notable example of specialization across encoder experts. And while we ﬁnd many
instances of specialization, these have been speciﬁcally extracted from many examples without a
clear semantic or syntactic specialization.

7.2 DECODER EXPERTS LACK SPECIALIZATION
ST-MOE- DESIGNING STABLE AND TRANSFERABLE SPARSE EXPERT MODELS