implemented in Tensorﬂow-Lingvo (Shen et al.,
2019).
In all cases we report test results for the checkpoint that performed best on the development set
in terms of BLEU. For the multilingual models we
create a development set that includes examples
we uniformly sample from a concatenation of all
the individual language pair development sets, resulting in 13k development examples per model.
Another important detail regarding multilingual
training is the batching scheme. In all of our multilingual models we use heterogeneous batching,
where each batch contains examples which are
uniformly sampled from a concatenation of all the
language pairs the model is trained on. Speciﬁcally, we use batches of 64 examples for sequences
shorter than 69 tokens and batches of 16 examples for longer sequences. We did not use oversampling as the dataset is relatively small.
2.3 Results
We use tokenized BLEU in order to be comparable with Neubig and Hu (2018). Table 1 showsAz-En Be-En Gl-En Sk-En Avg.
# of examples 5.9k 4.5k 10k 61k 20.3k
Neubig & Hu 18
baselines 2.7 2.8 16.2 24 11.42
Multilingual neural machine translation (NMT) enables training a single model
that supports translation from multiple source languages into multiple target
languages. In this paper, we push the limits of multilingual NMT in terms of
number of languages being used. We perform extensive experiments in training
massively multilingual NMT models, translating up to 102 languages to and from
English within a single model. We explore different setups for training such
models and analyze the trade-offs between translation quality and various
modeling decisions. We report results on the publicly available TED talks
multilingual corpus where we show that massively multilingual many-to-many
models are effective in low resource settings, outperforming the previous
state-of-the-art while supporting up to 59 languages. Our experiments on a
large-scale dataset with 102 languages to and from English and up to one
million examples per direction also show promising results, surpassing strong
bilingual baselines and encouraging future work on massively multilingual NMT.
Roee Aharoni Melvin Johnson Orhan Firat