of these subsets and measure the performance for
each model on the 8 supervised language pairs
from the smallest subset – fArabic, French, Russian, Ukrainian g$English. In this way we can
analyze to what extent adding more languages improves or harms translation quality while keeping
the model capacity ﬁxed, testing the capacity vs.
accuracy “saturation point”.
Table 7 shows the results of this experiment,
reporting the test results for the models that performed best on the multilingual development set.
We can see that in most cases the best results
are obtained using the 5-to-5 model, showing that
there is indeed a trade off between the number
of languages and translation accuracy when using a ﬁxed model capacity and the same training setup. One may expect that the gaps between
the different models should become smaller and
even close with more updates, as the models with
more languages see less examples per language
in each batch, thus requiring more updates to improve in terms of BLEU. However, in our setting
these gaps did not close even after the models converged, leaving 2.73 average BLEU difference be-tween the 5-to-5 and the 103-to-103 model.
4.2 Multilinguality & Zero-Shot
Performance
We then study the effect of the number of languages on zero-shot translation accuracy. Since
Multilingual neural machine translation (NMT) enables training a single model
that supports translation from multiple source languages into multiple target
languages. In this paper, we push the limits of multilingual NMT in terms of
number of languages being used. We perform extensive experiments in training
massively multilingual NMT models, translating up to 102 languages to and from
English within a single model. We explore different setups for training such
models and analyze the trade-offs between translation quality and various
modeling decisions. We report results on the publicly available TED talks
multilingual corpus where we show that massively multilingual many-to-many
models are effective in low resource settings, outperforming the previous
state-of-the-art while supporting up to 59 languages. Our experiments on a
large-scale dataset with 102 languages to and from English and up to one
million examples per direction also show promising results, surpassing strong
bilingual baselines and encouraging future work on massively multilingual NMT.
Roee Aharoni Melvin Johnson Orhan Firat