order to determine the target language in such
scenarios they proposed adding dedicated targetlanguage symbols to the source. This method enabled zero-shot translation, showing the ability of
the model to generalize to unseen pairs.
Recent works propose different methods for parameter sharing between language pairs in multilingual NMT. Blackwood et al. (2018) propose
sharing all parameters but the attention mechanism
and show improvements over sharing all parameters. Sachan and Neubig (2018) explore sharing various components in self-attentional (Transformer) models. Lu et al. (2018) add a shared “interlingua” layer while using separate encoders and
decoders. Zaremoodi et al. (2018) utilize recurrent
units with multiple blocks together with a trainable
routing network. Platanios et al. (2018) propose
to share the entire network, while using a contex-tual parameter generator that learns to generate the
parameters of the system given the desired source
and target languages. Gu et al. (2018) propose
a “Universal Language Representation” layer together with a Mixture-of-Language-Experts component to improve a many-to-one model from 5
languages into English.
While the mentioned studies provide valuable
contributions to improving multilingual models,
they apply their models on only up to 7 languages
Multilingual neural machine translation (NMT) enables training a single model
that supports translation from multiple source languages into multiple target
languages. In this paper, we push the limits of multilingual NMT in terms of
number of languages being used. We perform extensive experiments in training
massively multilingual NMT models, translating up to 102 languages to and from
English within a single model. We explore different setups for training such
models and analyze the trade-offs between translation quality and various
modeling decisions. We report results on the publicly available TED talks
multilingual corpus where we show that massively multilingual many-to-many
models are effective in low resource settings, outperforming the previous
state-of-the-art while supporting up to 59 languages. Our experiments on a
large-scale dataset with 102 languages to and from English and up to one
million examples per direction also show promising results, surpassing strong
bilingual baselines and encouraging future work on massively multilingual NMT.
Roee Aharoni Melvin Johnson Orhan Firat