One might expect experts to specialize in languages, which appears as a natural criterion for divvying
up batches of data among experts. However, we ﬁnd no evidence of language specialization (see
Table 15). Routers instead pass tokens from English, Japanese, French and Chinese indiscriminately
and the experts appear to be multilingual. But this lack of language specialization is less surprising
when considering the mechanism of token routing and load balancing. Since each group of tokens
may only contain one, to at most a few, languages (a group usually consists of 2-4 sequences in
our setup), then all experts are encouraged to handle tokens from all languages. We experimented
with a global load balance loss, however, this usually results in worse load-balance and worse model
performance, so we leave further improving multilingual expert models as an area of open work
(Section 9).
Our visualization reveals apparent specialization learned in our models (Tables 13, 15) for the en-
ST-MOE- DESIGNING STABLE AND TRANSFERABLE SPARSE EXPERT MODELS