400000 4.301280901 6.631205231 16.16190225 17.38892049 17.55904406
450000 3.882381693 6.804813445 18.20554733 17.48778224 17.9339543
500000 3.45445089 6.358428299 17.32598543 16.71108902 15.97367823
550000 3.18 6.38 18.65 18.28 17.4
600000 2.86 9.5 18.46 14.92 17.12
650000 2.55 12.2 18.98 15.68 16.19
700000 2.98 8.44 20.16 15.4 18.52
Figure 2: Zero-shot BLEU during training for Ukranian to Russian
els, it is much better for the 50-to-50, 75-to-75 and
103-to-103 models. Given these results we can
say that the balance between capacity and generalization here favors the mid range 50-to-50 model,
even when using models with more than 473M
trained parameters. This may hint at the necessity of even larger models for such settings, which
is a challenging avenue for future work. We also
note that our 103 language corpus includes up to
Multilingual neural machine translation (NMT) enables training a single model
that supports translation from multiple source languages into multiple target
languages. In this paper, we push the limits of multilingual NMT in terms of
number of languages being used. We perform extensive experiments in training
massively multilingual NMT models, translating up to 102 languages to and from
English within a single model. We explore different setups for training such
models and analyze the trade-offs between translation quality and various
modeling decisions. We report results on the publicly available TED talks
multilingual corpus where we show that massively multilingual many-to-many
models are effective in low resource settings, outperforming the previous
state-of-the-art while supporting up to 59 languages. Our experiments on a
large-scale dataset with 102 languages to and from English and up to one
million examples per direction also show promising results, surpassing strong
bilingual baselines and encouraging future work on massively multilingual NMT.
Roee Aharoni Melvin Johnson Orhan Firat