7 TRACING TOKENS THROUGH THE MODEL

Thus far we have presented quantitative measures and performance metrics. We change tack to
explore qualitative features by visualizing how tokens are routed among the experts. We do so by
passing a batch of tokens to the model and manually inspecting token assignment at each layer. We
consider our ST-MoE-L model pre-trained either on the monolingual C4 corpus (Raffel et al., 2019)
or on the multilingual mC4 corpus (Xue et al., 2020). On both the encoder and the decoder, the
model has six sparse layers, each with 32 experts.

The span corruption objective is to recover spans of variable-length contiguous segments
masked out in the inputs. This is formatted as:

Preliminaries

Inputs: I went to <extra id 0> to buy <extra id 1>

Targets: <extra id 0> the store <extra id 1> milk

In our encoder-decoder architecture, the inputs will be passed to the encoder and targets to
the decoder.
ST-MOE- DESIGNING STABLE AND TRANSFERABLE SPARSE EXPERT MODELS