models can scale to 59 languages in a lowresource, imbalanced, English-centric setting,
with the following observations: (1) massively
multilingual many-to-many models outperform
many-to-one and bilingual models with similar capacity and identical training conditions when averaged over 8 language pairs into English. We
attribute this improvement over the many-to-one
models to the multiple target language pairs which
may act as regularizers, especially in this lowresource multi-way-parallel setting that is prone to
memorization. (2) many-to-many models are inferior in performance when going out-of-English
in comparison to a one-to-many model. We attribute this to English being over-represented in
the English-centric many-to-many setting, where
it appears as a target language in 58 out of 116
trained directions, which may harm the performance on the rest of the target languages as the
model capacity is limited.3
It is important to stress the fact that we compared the different models under identical training
conditions and did not perform extensive hyperparameter tuning for each setting separately. However, we believe that such tuning may improve
performance even further, as the diversity in each
training batch is very different between the different settings. For example, while the baseline
model batches include only one language in the
Multilingual neural machine translation (NMT) enables training a single model
that supports translation from multiple source languages into multiple target
languages. In this paper, we push the limits of multilingual NMT in terms of
number of languages being used. We perform extensive experiments in training
massively multilingual NMT models, translating up to 102 languages to and from
English within a single model. We explore different setups for training such
models and analyze the trade-offs between translation quality and various
modeling decisions. We report results on the publicly available TED talks
multilingual corpus where we show that massively multilingual many-to-many
models are effective in low resource settings, outperforming the previous
state-of-the-art while supporting up to 59 languages. Our experiments on a
large-scale dataset with 102 languages to and from English and up to one
million examples per direction also show promising results, surpassing strong
bilingual baselines and encouraging future work on massively multilingual NMT.
Roee Aharoni Melvin Johnson Orhan Firat