to one million examples per language pair. This
results in 103 languages in total, and 204 translation directions which we train simultaneously.
3This issue may be alleviated by over-sampling the nonEnglish-target pairs, but we leave this for future work.More details about this dataset are available in Table 4, and Table 10 in the supplementary material
details all the languages in the dataset.4
Similarly to our previous experiments, we compare the massively multilingual models to bilingual baselines trained on the same data. We tokenize the data using an in-house tokenizer and then
apply joint subword segmentation to achieve an
open-vocabulary. In this setting we used a vocabulary of 64k subwords rather than 32k. Since the
dataset contains 24k unique characters, a 32k symbol vocabulary will consist of mostly characters,
thereby increasing the average sequence length.
Regarding the model, for these experiments we
use a larger Transformer model with 6 layers in
both the encoder and the decoder, model dimension set to 1024, hidden dimension size of 8192,
and 16 attention heads. This results in a model
with approximately 473.7M parameters.5Since
the model and data are much larger in this case,
we used a dropout rate of 0.1 for our multilingual
models and tuned it to 0.3 for our baseline models
Multilingual neural machine translation (NMT) enables training a single model
that supports translation from multiple source languages into multiple target
languages. In this paper, we push the limits of multilingual NMT in terms of
number of languages being used. We perform extensive experiments in training
massively multilingual NMT models, translating up to 102 languages to and from
English within a single model. We explore different setups for training such
models and analyze the trade-offs between translation quality and various
modeling decisions. We report results on the publicly available TED talks
multilingual corpus where we show that massively multilingual many-to-many
models are effective in low resource settings, outperforming the previous
state-of-the-art while supporting up to 59 languages. Our experiments on a
large-scale dataset with 102 languages to and from English and up to one
million examples per direction also show promising results, surpassing strong
bilingual baselines and encouraging future work on massively multilingual NMT.
Roee Aharoni Melvin Johnson Orhan Firat