et al. (2019) which use ”Soft Decoupled Encoding” for the input tokens, while our models use a
simple subword segmentation.
One possible explanation is that the many-toone model overﬁts the English side of the corpus
as it is multi-way-parallel: in such setting the English sentences are overlapping across the different language pairs, making it much easier for the
model to memorize the training set instead of generalizing (when enough capacity is available). On
the other hand, the many-to-many model is trained
on additional target languages other than English,
which can act as regularizers for the X !En tasks,
reducing such overﬁtting.
To further illustrate this, Figure 1 tracks the
BLEU scores on the individual development sets
during training for Italian (It), Romanian (Ro),
Dutch (Nl), German (De) and Arabic (Ar) into English (left), together with BLEU scores on a subset of the training set for each model. We can
see that while the many-to-one model degrades in
performance on the development set, the manyto-many model still improves. Note the large
gap in the many-to-one model between the training set BLEU and the development set BLEU,
which points on the generalization issue that is
Multilingual neural machine translation (NMT) enables training a single model
that supports translation from multiple source languages into multiple target
languages. In this paper, we push the limits of multilingual NMT in terms of
number of languages being used. We perform extensive experiments in training
massively multilingual NMT models, translating up to 102 languages to and from
English within a single model. We explore different setups for training such
models and analyze the trade-offs between translation quality and various
modeling decisions. We report results on the publicly available TED talks
multilingual corpus where we show that massively multilingual many-to-many
models are effective in low resource settings, outperforming the previous
state-of-the-art while supporting up to 59 languages. Our experiments on a
large-scale dataset with 102 languages to and from English and up to one
million examples per direction also show promising results, surpassing strong
bilingual baselines and encouraging future work on massively multilingual NMT.
Roee Aharoni Melvin Johnson Orhan Firat