to 58 languages to-and-from English are very effective in low resource settings, allowing to use
high-capacity models while avoiding overﬁtting
and achieving superior results to the current stateof-the-art on this dataset (Neubig and Hu, 2018;
Wang et al., 2019) when translating into English.
We then turn to experiment with models trained
on 103 languages in a high-resource setting. For
this purpose we compile an English-centric inhouse dataset, including 102 languages aligned
to-and-from English with up to one million examples per language pair. We then train a single model on the resulting 204 translation directions and ﬁnd that such models outperform strong
bilingual baselines by more than 2 BLEU averaged across 10 diverse language pairs, both toand-from English. Finally, we analyze the tradeoffs between the number of involved languages
and translation accuracy in such settings, showing
that massively multilingual models generalize better to zero-shot scenarios. We hope these results
will encourage future research on massively multilingual NMT.
2 Low-Resource Setting: 59 Languages
2.1 Experimental Setup
The main question we wish to answer in this work
is how well a single NMT model can scale to
support a very large number of language pairs.
The answer is not trivial: on the one hand, training multiple language pairs together may result
Multilingual neural machine translation (NMT) enables training a single model
that supports translation from multiple source languages into multiple target
languages. In this paper, we push the limits of multilingual NMT in terms of
number of languages being used. We perform extensive experiments in training
massively multilingual NMT models, translating up to 102 languages to and from
English within a single model. We explore different setups for training such
models and analyze the trade-offs between translation quality and various
modeling decisions. We report results on the publicly available TED talks
multilingual corpus where we show that massively multilingual many-to-many
models are effective in low resource settings, outperforming the previous
state-of-the-art while supporting up to 59 languages. Our experiments on a
large-scale dataset with 102 languages to and from English and up to one
million examples per direction also show promising results, surpassing strong
bilingual baselines and encouraging future work on massively multilingual NMT.
Roee Aharoni Melvin Johnson Orhan Firat