languages into English.
While the mentioned studies provide valuable
contributions to improving multilingual models,
they apply their models on only up to 7 languages
(Johnson et al., 2017) and 20 trained directions
(Cettolo et al., 2017) in a single model, whereas
we focus on scaling NMT to much larger numbers of languages and trained directions. Regarding massively multilingual models, Neubig and
Hu (2018) explored methods for rapid adaptation
of NMT to new languages by training multilingual models on the 59-language TED Talks corpus and ﬁne-tuning them using data from the new
languages. While modeling signiﬁcantly more
languages than previous studies, they only train
many-to-one models, which we show are inferior
in comparison to our proposed massively multilingual many-to-many models when evaluated into
English on this dataset.
Tiedemann (2018) trained an English-centric
many-to-many model on translations of the bible
including 927 languages. While this work pointed
to an interesting phenomena in the latent space
learned by the model where it clusters representations of typologically-similar languages together, it did not include any evaluation of the
produced translations. Similarly, Malaviya et al.
(2017) trained a many-to-English system including 1017 languages from bible translations, and
Multilingual neural machine translation (NMT) enables training a single model
that supports translation from multiple source languages into multiple target
languages. In this paper, we push the limits of multilingual NMT in terms of
number of languages being used. We perform extensive experiments in training
massively multilingual NMT models, translating up to 102 languages to and from
English within a single model. We explore different setups for training such
models and analyze the trade-offs between translation quality and various
modeling decisions. We report results on the publicly available TED talks
multilingual corpus where we show that massively multilingual many-to-many
models are effective in low resource settings, outperforming the previous
state-of-the-art while supporting up to 59 languages. Our experiments on a
large-scale dataset with 102 languages to and from English and up to one
million examples per direction also show promising results, surpassing strong
bilingual baselines and encouraging future work on massively multilingual NMT.
Roee Aharoni Melvin Johnson Orhan Firat