the model and data are much larger in this case,
we used a dropout rate of 0.1 for our multilingual
models and tuned it to 0.3 for our baseline models
as it improved the translation quality on the development set.
We evaluate our models on 10 languages from
different typological families: Semitic – Arabic
(Ar), Hebrew (He), Romance – Galician (Gl),
Italian (It), Romanian (Ro), Germanic – German
(De), Dutch (Nl), Slavic – Belarusian (Be), Slovak (Sk) and Turkic – Azerbaijani (Az) and Turkish (Tr). We evaluate both to-and-from English,
where each language pair is trained on up to one
million examples. As in the previous experiment,
we report test results from the model that performed best in terms of BLEU on the development
set.
4The average number of examples per language pair is
940k, as for 13 out of the 102 pairs we had less than one
million examples available.
5This is larger than the Transformer “Big” conﬁguration,
which includes approximately 213M trained parameters.
# of language pairs 102
examples per pair
min 63,879
max 1,000,000
average 940,087
std. deviation 188,194
Multilingual neural machine translation (NMT) enables training a single model
that supports translation from multiple source languages into multiple target
languages. In this paper, we push the limits of multilingual NMT in terms of
number of languages being used. We perform extensive experiments in training
massively multilingual NMT models, translating up to 102 languages to and from
English within a single model. We explore different setups for training such
models and analyze the trade-offs between translation quality and various
modeling decisions. We report results on the publicly available TED talks
multilingual corpus where we show that massively multilingual many-to-many
models are effective in low resource settings, outperforming the previous
state-of-the-art while supporting up to 59 languages. Our experiments on a
large-scale dataset with 102 languages to and from English and up to one
million examples per direction also show promising results, surpassing strong
bilingual baselines and encouraging future work on massively multilingual NMT.
Roee Aharoni Melvin Johnson Orhan Firat