worse than the one-to-many model by 2.53 BLEU
on average. While previous works (Wang et al.,
2018; Sachan and Neubig, 2018) discuss the phenomena of quality degradation in English-to-many
settings, this shows that increasing the number of
source languages also causes additional degradation in a many-to-many model. This degradation
may be due to the English-centric setting: since
most of the translation directions the model is
trained on are into English, this leaves less capacity for the other target languages (while still performing better than the bilingual baselines on all 8
language pairs). We also note that in this case the
results are consistent among the higher and lower
resource pairs – the one-to-many model is better
than the many-to-many model, which outperforms
the bilingual baselines in all cases. This is unlike
the difference we saw in the X !En experiments
since here we do not have the multi-way-parallel
overﬁtting issue.
2.4 Discussion
From the above experiments we learn that NMT
models can scale to 59 languages in a lowresource, imbalanced, English-centric setting,
with the following observations: (1) massively
multilingual many-to-many models outperform
Multilingual neural machine translation (NMT) enables training a single model
that supports translation from multiple source languages into multiple target
languages. In this paper, we push the limits of multilingual NMT in terms of
number of languages being used. We perform extensive experiments in training
massively multilingual NMT models, translating up to 102 languages to and from
English within a single model. We explore different setups for training such
models and analyze the trade-offs between translation quality and various
modeling decisions. We report results on the publicly available TED talks
multilingual corpus where we show that massively multilingual many-to-many
models are effective in low resource settings, outperforming the previous
state-of-the-art while supporting up to 59 languages. Our experiments on a
large-scale dataset with 102 languages to and from English and up to one
million examples per direction also show promising results, surpassing strong
bilingual baselines and encouraging future work on massively multilingual NMT.
Roee Aharoni Melvin Johnson Orhan Firat