trained parameters. This may hint at the necessity of even larger models for such settings, which
is a challenging avenue for future work. We also
note that our 103 language corpus includes up to
one million examples per language pair – while in
real-world MT deployments, systems are trained
on much more examples per pair. This again emphasizes the need for better techniques for training
such massively multilingual models as we may already be hitting the capacity barrier in our setting.
5 Related Work
Dong et al. (2015) extended the NMT model of
Bahdanau et al. (2014) to one-to-many translation
(from English into 4 languages) by adding a dedicated decoder per target language, showing improvements over strong single-pair baselines. Firat et al. (2016a,b) proposed many-to-many models (with up to 6 languages) by using separate encoders and decoders per language while sharing
the attention mechanism. They also introduced
the notion of zero-resource translation, where they
use synthetic training data generated through pivoting to train translation directions without available training data. Ha et al. (2016) and Johnson
et al. (2017) proposed to use a shared encoderdecoder-attention model for many-to-many translation (with up to 7 languages in the latter). In
order to determine the target language in such
Multilingual neural machine translation (NMT) enables training a single model
that supports translation from multiple source languages into multiple target
languages. In this paper, we push the limits of multilingual NMT in terms of
number of languages being used. We perform extensive experiments in training
massively multilingual NMT models, translating up to 102 languages to and from
English within a single model. We explore different setups for training such
models and analyze the trade-offs between translation quality and various
modeling decisions. We report results on the publicly available TED talks
multilingual corpus where we show that massively multilingual many-to-many
models are effective in low resource settings, outperforming the previous
state-of-the-art while supporting up to 59 languages. Our experiments on a
large-scale dataset with 102 languages to and from English and up to one
million examples per direction also show promising results, surpassing strong
bilingual baselines and encouraging future work on massively multilingual NMT.
Roee Aharoni Melvin Johnson Orhan Firat