is how well a single NMT model can scale to
support a very large number of language pairs.
The answer is not trivial: on the one hand, training multiple language pairs together may result
in transfer learning (Zoph et al., 2016; Nguyen
and Chiang, 2017). This may improve performance as we increase the number of language
pairs, since more information can be shared between the different translation tasks, allowing the
model to learn which information to share. On the
other hand, adding many language pairs may result in a bottleneck; the model has a limited capacity while it needs to handle this large number
of translation tasks, and sharing all parameters between the different languages can be sub-optimal(Wang et al., 2018) especially if they are not from
the same typological language family (Sachan and
Neubig, 2018).
We begin tackling this question by experimenting with the TED Talks parallel corpus compiled
by Qi et al. (2018)1, which is unique in that it includes parallel data from 59 languages. For comparison, this is signiﬁcantly “more multilingual”
than the data available from all previous WMT
news translation shared task evaluations throughout the years – the latest being Bojar et al. (2016,
2017, 2018), which included 14 languages so far.2
We focus on the setting where we train
Multilingual neural machine translation (NMT) enables training a single model
that supports translation from multiple source languages into multiple target
languages. In this paper, we push the limits of multilingual NMT in terms of
number of languages being used. We perform extensive experiments in training
massively multilingual NMT models, translating up to 102 languages to and from
English within a single model. We explore different setups for training such
models and analyze the trade-offs between translation quality and various
modeling decisions. We report results on the publicly available TED talks
multilingual corpus where we show that massively multilingual many-to-many
models are effective in low resource settings, outperforming the previous
state-of-the-art while supporting up to 59 languages. Our experiments on a
large-scale dataset with 102 languages to and from English and up to one
million examples per direction also show promising results, surpassing strong
bilingual baselines and encouraging future work on massively multilingual NMT.
Roee Aharoni Melvin Johnson Orhan Firat