2017, 2018), which included 14 languages so far.2
We focus on the setting where we train
“English-centric” models, i.e. training on all
language pairs that contain English in either the
source or the target, resulting in 116 translation
directions. This dataset is also highly imbalanced, with language pairs including between 3.3k
to 214k sentence pairs for training. Table 9 in
the supplementary material details the languages
and training set sizes for this dataset. Since the
dataset is already tokenized we did not apply additional preprocessing other than applying joint
subword segmentation (Sennrich et al., 2016) with
32k symbols.
Regarding the languages we evaluate on, we begin with the same four languages as Neubig and
Hu (2018) – Azerbeijani (Az), Belarusian (Be),
Galician (Gl) and Slovak (Sk). These languages
present an extreme low-resource case, with as few
as 4.5k training examples for Belarusian-English.
In order to better understand the effect of training
set size in these settings, we evaluate on four additional languages that have more than 167k training examples each – Arabic (Ar), German (De),
Hebrew (He) and Italian (It).
2.2 Model Details
Using the same data, we trained three massively
Multilingual neural machine translation (NMT) enables training a single model
that supports translation from multiple source languages into multiple target
languages. In this paper, we push the limits of multilingual NMT in terms of
number of languages being used. We perform extensive experiments in training
massively multilingual NMT models, translating up to 102 languages to and from
English within a single model. We explore different setups for training such
models and analyze the trade-offs between translation quality and various
modeling decisions. We report results on the publicly available TED talks
multilingual corpus where we show that massively multilingual many-to-many
models are effective in low resource settings, outperforming the previous
state-of-the-art while supporting up to 59 languages. Our experiments on a
large-scale dataset with 102 languages to and from English and up to one
million examples per direction also show promising results, surpassing strong
bilingual baselines and encouraging future work on massively multilingual NMT.
Roee Aharoni Melvin Johnson Orhan Firat