one million examples per direction also show
promising results, surpassing strong bilingual
baselines and encouraging future work on
massively multilingual NMT.
1 Introduction
Neural machine translation (NMT) (Kalchbrenner and Blunsom, 2013; Bahdanau et al., 2014;
Sutskever et al., 2014) is the current state-ofthe-art approach for machine translation in both
academia (Bojar et al., 2016, 2017, 2018) and industry (Wu et al., 2016; Hassan et al., 2018). Recent works (Dong et al., 2015; Firat et al., 2016a;
Ha et al., 2016; Johnson et al., 2017) extended the
approach to support multilingual translation, i.e.
training a single model that is capable of translating between multiple language pairs.
Multilingual models are appealing for several
reasons. First, they are more efﬁcient in terms
Work carried out during an internship at Google AI.of the number of required models and model parameters, enabling simpler deployment. Another
beneﬁt is transfer learning; when low-resource
language pairs are trained together with highresource ones, the translation quality may improve
(Zoph et al., 2016; Nguyen and Chiang, 2017). An
extreme case of such transfer learning is zero-shot
Multilingual neural machine translation (NMT) enables training a single model
that supports translation from multiple source languages into multiple target
languages. In this paper, we push the limits of multilingual NMT in terms of
number of languages being used. We perform extensive experiments in training
massively multilingual NMT models, translating up to 102 languages to and from
English within a single model. We explore different setups for training such
models and analyze the trade-offs between translation quality and various
modeling decisions. We report results on the publicly available TED talks
multilingual corpus where we show that massively multilingual many-to-many
models are effective in low resource settings, outperforming the previous
state-of-the-art while supporting up to 59 languages. Our experiments on a
large-scale dataset with 102 languages to and from English and up to one
million examples per direction also show promising results, surpassing strong
bilingual baselines and encouraging future work on massively multilingual NMT.
Roee Aharoni Melvin Johnson Orhan Firat