performance even further, as the diversity in each
training batch is very different between the different settings. For example, while the baseline
model batches include only one language in the
source and one language in the target, the manyto-many model includes 59 languages in each
side with a strong bias towards English. These
differences may require tailored hyper-parameter
choices for each settings (i.e. different batch sizes,
learning rate schedules, dropout rates etc.) which
would be interesting to explore in future work.
In the following experiments we investigate
whether these observations hold using (1) an even
larger set of languages, and (2) a much larger,
balanced training corpus that is not multi-wayparallel.
3 High-Resource Setting: 103 Languages
3.1 Experimental Setup
In this setting we scale the number of languages
and examples per language pair further when
training a single massively multilingual model.
Since we are not aware of a publicly available resource for this purpose, we construct an in-house
dataset. This dataset includes 102 language pairs
which we “mirror” to-and-from English, with up
to one million examples per language pair. This
results in 103 languages in total, and 204 translation directions which we train simultaneously.
Multilingual neural machine translation (NMT) enables training a single model
that supports translation from multiple source languages into multiple target
languages. In this paper, we push the limits of multilingual NMT in terms of
number of languages being used. We perform extensive experiments in training
massively multilingual NMT models, translating up to 102 languages to and from
English within a single model. We explore different setups for training such
models and analyze the trade-offs between translation quality and various
modeling decisions. We report results on the publicly available TED talks
multilingual corpus where we show that massively multilingual many-to-many
models are effective in low resource settings, outperforming the previous
state-of-the-art while supporting up to 59 languages. Our experiments on a
large-scale dataset with 102 languages to and from English and up to one
million examples per direction also show promising results, surpassing strong
bilingual baselines and encouraging future work on massively multilingual NMT.
Roee Aharoni Melvin Johnson Orhan Firat