bilingual baselines and to two recently published
results on this dataset (Neubig and Hu (2018);
Wang et al. (2019)).
Regarding the models, we focused on the Transformer in the “Base” conﬁguration. We refer the
reader to Vaswani et al. (2017) for more details
on the model architecture. Speciﬁcally, we use 6
layers in both the encoder and the decoder, with
model dimension set at 512, hidden dimension
size of 2048 and 8 attention heads. We also applied dropout at a rate of 0.2 in the following components: on the sum of the input embeddings and
the positional embeddings, on the output of each
sub-layer before added to the previous layer input
(residual connection), on the inner layer output after the ReLU activation in each feed-forward sublayer, and to the attention weight in each attention
sub-layer. This results in a model with approximately 93M trainable parameters. For all models we used the inverse square root learning rate
schedule from Vaswani et al. (2017) with learningrate set at 3 and 40k warmup steps. All models are
implemented in Tensorﬂow-Lingvo (Shen et al.,
2019).
Multilingual neural machine translation (NMT) enables training a single model
that supports translation from multiple source languages into multiple target
languages. In this paper, we push the limits of multilingual NMT in terms of
number of languages being used. We perform extensive experiments in training
massively multilingual NMT models, translating up to 102 languages to and from
English within a single model. We explore different setups for training such
models and analyze the trade-offs between translation quality and various
modeling decisions. We report results on the publicly available TED talks
multilingual corpus where we show that massively multilingual many-to-many
models are effective in low resource settings, outperforming the previous
state-of-the-art while supporting up to 59 languages. Our experiments on a
large-scale dataset with 102 languages to and from English and up to one
million examples per direction also show promising results, surpassing strong
bilingual baselines and encouraging future work on massively multilingual NMT.
Roee Aharoni Melvin Johnson Orhan Firat