and with up to one million examples per languagepair simultaneously. Speciﬁcally, we focus on
training “English-centric” many-to-many models,
in which the training data is composed of many
language pairs that contain English either on the
source side or the target side. This is a realistic
setting since English parallel data is widely available for many language pairs. We restrict our experiments to Transformer models (Vaswani et al.,
2017) as they were shown to be very effective
in recent benchmarks (Ott et al., 2018), also inarXiv:1903.00089v3  [cs.CL]  2 Jul 2019
the context of multilingual models (Lakew et al.,
2018; Sachan and Neubig, 2018).
We evaluate the performance of such massively
multilingual models while varying factors like
model capacity, the number of trained directions
(tasks) and low-resource vs. high-resource settings. Our experiments on the publicly available
TED talks dataset (Qi et al., 2018) show that massively multilingual many-to-many models with up
to 58 languages to-and-from English are very effective in low resource settings, allowing to use
high-capacity models while avoiding overﬁtting
Multilingual neural machine translation (NMT) enables training a single model
that supports translation from multiple source languages into multiple target
languages. In this paper, we push the limits of multilingual NMT in terms of
number of languages being used. We perform extensive experiments in training
massively multilingual NMT models, translating up to 102 languages to and from
English within a single model. We explore different setups for training such
models and analyze the trade-offs between translation quality and various
modeling decisions. We report results on the publicly available TED talks
multilingual corpus where we show that massively multilingual many-to-many
models are effective in low resource settings, outperforming the previous
state-of-the-art while supporting up to 59 languages. Our experiments on a
large-scale dataset with 102 languages to and from English and up to one
million examples per direction also show promising results, surpassing strong
bilingual baselines and encouraging future work on massively multilingual NMT.
Roee Aharoni Melvin Johnson Orhan Firat