(Zoph et al., 2016; Nguyen and Chiang, 2017). An
extreme case of such transfer learning is zero-shot
translation (Johnson et al., 2017), where multilingual models are able to translate between language
pairs that were never seen during training.
While very promising, it is still unclear how far
one can scale multilingual NMT in terms of the
number of languages involved. Previous works
on multilingual NMT typically trained models
with up to 7 languages (Dong et al., 2015; Firat et al., 2016b; Ha et al., 2016; Johnson et al.,
2017; Gu et al., 2018) and up to 20 trained directions (Cettolo et al., 2017) simultaneously. One
recent exception is Neubig and Hu (2018) who
trained many-to-one models from 58 languages
into English. While utilizing signiﬁcantly more
languages than previous works, their experiments
were restricted to many-to-one models in a lowresource setting with up to 214k examples per
language-pair and were evaluated only on four
translation directions.
In this work, we take a step towards practical
“universal” NMT – training massively multilingual models which support up to 102 languages
and with up to one million examples per languagepair simultaneously. Speciﬁcally, we focus on
Multilingual neural machine translation (NMT) enables training a single model
that supports translation from multiple source languages into multiple target
languages. In this paper, we push the limits of multilingual NMT in terms of
number of languages being used. We perform extensive experiments in training
massively multilingual NMT models, translating up to 102 languages to and from
English within a single model. We explore different setups for training such
models and analyze the trade-offs between translation quality and various
modeling decisions. We report results on the publicly available TED talks
multilingual corpus where we show that massively multilingual many-to-many
models are effective in low resource settings, outperforming the previous
state-of-the-art while supporting up to 59 languages. Our experiments on a
large-scale dataset with 102 languages to and from English and up to one
million examples per direction also show promising results, surpassing strong
bilingual baselines and encouraging future work on massively multilingual NMT.
Roee Aharoni Melvin Johnson Orhan Firat