summaries: In this paper, the authors present a study of different aspects of language-specific model capacity for massively multilingual machine translation. To this end, language-specific behaviour is achieved via a combination of conditional computation to decide whether to use language-specific parameters or not and statically assigning experts for each languages. The language specific sub-layers are incorporated throughout the network. The training objective allow budgetary constraints on the amount of language-specific parameters. The paper does a systematic analysis on the role of language specific parameters using the proposed architecture. Based on the analysis, recommendations on design of multilingual NMT architectures are proposed and their efficacy validated experimentally. The study sheds light on the amount of language specific parameter sharing, their distribution in the network, impact of language, etc.

summaries: The work proposes a hybrid architecture that has: (1) language-specific (LS) components; (2) as well as the components that are shared across all the languages -- a trade-off between specificity and generality.  A key conclusion of the work is that the best architectures typically are. the ones that have ~10-30% language-specific capacity.

summaries: In this work, the authors present a conditional language-specific routing (CLSR) scheme for transformer-based multilingual NMT systems. They introduce a CLSR layer after every transformer encoder and decoder layer; each such layer is made up of hard gating functions conditioned on token representations that will either select a language-specific projection layer or a shared projection layer. Further, a budget is imposed on the language-specific capacity measured by aggregating the number of gates that allow for language-specific computations; this budget constraint forces the network to identify the sub-layers that will benefit most from being language-specific.

abstractText: Using a mix of shared and language-specific (LS) parameters has shown promise in multilingual neural machine translation (MNMT), but the question of when and where LS capacity matters most is still under-studied. We offer such a study by proposing conditional language-specific routing (CLSR). CLSR employs hard binary gates conditioned on token representations to dynamically select LS or shared paths. By manipulating these gates, it can schedule LS capacity across sub-layers in MNMT subject to the guidance of translation signals and budget constraints. Moreover, CLSR can easily scale up to massively multilingual settings. Experiments with Transformer on OPUS-100 and WMT datasets show that: 1) MNMT is sensitive to both the amount and the position of LS modeling: distributing 10%-30% LS computation to the top and/or bottom encoder/decoder layers delivers the best performance; and 2) one-to-many translation benefits more from CLSR compared to many-to-one translation, particularly with unbalanced training data. Our study further verifies the trade-off between the shared capacity and LS capacity for multilingual translation. We corroborate our analysis by confirming the soundness of our findings as foundation of our improved multilingual Transformers. Source code and models are available at https://github.com/bzhangGo/zero/tree/iclr2021_clsr.

authors:
Roee Aharoni
Melvin Johnson
Orhan Firat

title: Massively multilingual neural machine translation

authors:
Naveen Arivazhagan
Ankur Bapna
Orhan Firat
Dmitry Lepikhin
Melvin Johnson
Maxim Krikun
Mia Xu Chen
Yuan Cao
George Foster
Colin Cherry
Wolfgang Macherey
Zhifeng Chen
Yonghui Wu

title: Massively multilingual neural machine translation in the wild: Findings and challenges

venue: URL http://arxiv.org/abs/1907.05019

authors:
Ankur Bapna
Orhan Firat

title: Simple, scalable adaptation for neural machine translation

venue: In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),

authors:
Ankur Bapna
Naveen Arivazhagan
Orhan Firat

title: Controlling computation versus quality for neural sequence models

venue: arXiv preprint arXiv:2002.07106,

authors:
Yoshua Bengio
Nicholas Léonard
Aaron Courville

title: Estimating or propagating gradients through stochastic neurons for conditional computation

venue: arXiv preprint arXiv:1308.3432,

authors:
Graeme Blackwood
Miguel Ballesteros
Todd Ward

title: Multilingual neural machine translation with task-specific attention

venue: In Proceedings of the 27th International Conference on Computational Linguistics,

authors:
Chung-Cheng Chiu
Colin Raffel

title: Monotonic chunkwise attention

authors:
Andrew Davis
Itamar Arel

title: Low-rank approximations for conditional feedforward computation in deep neural networks

authors:
Mostafa Dehghani
Stephan Gouws
Oriol Vinyals
Jakob Uszkoreit
Lukasz Kaiser

title: Universal transformers

authors:
Daxiang Dong
Hua Wu
Wei He
Dianhai Yu
Haifeng Wang

title: Multi-task learning for multiple language translation

venue: In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers),

authors:
Maha Elbayad
Jiatao Gu
Edouard Grave
Michael Auli

title: Depth-adaptive transformer

authors:
Carlos Escolano
Marta R Costa-jussà
José AR Fonollosa
Mikel Artetxe

title: Multilingual machine translation: Closing the gap between shared and language-specific encoder-decoders

venue: arXiv preprint arXiv:2004.06575,

authors:
Angela Fan
Edouard Grave
Armand Joulin

title: Reducing transformer depth on demand with structured dropout

authors:
Orhan Firat
Kyunghyun Cho
Yoshua Bengio

title: Multi-way, multilingual neural machine translation with a shared attention mechanism

venue: In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,

authors:
Orhan Firat
Baskaran Sankaran
Yaser Al-onaizan
Fatos T. Yarman Vural
Kyunghyun Cho

title: Zero-resource translation with multi-lingual neural machine translation

venue: In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing,

title: Adaptive computation time for recurrent neural networks

venue: arXiv preprint arXiv:1603.08983,

authors:
Thanh-Le Ha
Jan Niehues
Alexander Waibel

title: Toward multilingual neural machine translation with universal encoder and decoder

venue: In Proceedings of the 13th International Workshop on Spoken Language Translation (IWSLT),

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

authors:
Yanping Huang
Youlong Cheng
Ankur Bapna
Orhan Firat
Dehao Chen
Mia Chen
HyoukJoong Lee
Jiquan Ngiam
Quoc V Le
Yonghui Wu

title: Gpipe: Efficient training of giant neural networks using pipeline parallelism

venue: In Advances in neural information processing systems,

authors:
Melvin Johnson
Mike Schuster
Quoc V. Le
Maxim Krikun
Yonghui Wu
Zhifeng Chen
Nikhil Thorat
Fernanda Viégas
Martin Wattenberg
Greg Corrado
Macduff Hughes
Jeffrey Dean

title: Google’s multilingual neural machine translation system: Enabling zero-shot translation

venue: Transactions of the Association for Computational Linguistics,

authors:
Diederik P Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

authors:
Taku Kudo
John Richardson

title: SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing

venue: In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,

authors:
Jason Lee
Kyunghyun Cho
Thomas Hofmann

title: Fully character-level neural machine translation without explicit segmentation

authors:
Dmitry Lepikhin
HyoukJoong Lee
Yuanzhong Xu
Dehao Chen
Orhan Firat
Yanping Huang
Maxim Krikun
Noam Shazeer
Zhifeng Chen

title: Gshard: Scaling giant models with conditional computation and automatic sharding

authors:
Hanxiao Liu
Karen Simonyan
Yiming Yang

title: DARTS: Differentiable architecture search

title: The need for biases in learning generalizations. Department of Computer Science, Laboratory for Computer Science Research

authors:
Kishore Papineni
Salim Roukos
Todd Ward
Wei-Jing Zhu

title: Bleu: a method for automatic evaluation of machine translation

venue: In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics,

authors:
Emmanouil Antonios Platanios
Mrinmaya Sachan
Graham Neubig
Tom Mitchell

title: Contextual parameter generation for universal neural machine translation

venue: In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,

title: A call for clarity in reporting BLEU scores

venue: In Proceedings of the Third Conference on Machine Translation: Research Papers,

authors:
Sebastian Ruder
Joachim Bingel
Isabelle Augenstein
Anders Søgaard

title: Latent multi-task architecture learning

venue: Proceedings of the AAAI Conference on Artificial Intelligence,

authors:
Devendra Sachan
Graham Neubig

title: Parameter sharing methods for multilingual self-attentional translation models

authors:
Rico Sennrich
Barry Haddow
Alexandra Birch

title: Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1715–1725, Berlin, Germany, August 2016

venue: Association for Computational Linguistics. doi: 10.18653/v1/P16-1162. URL https://www.aclweb. org/anthology/P16-1162

authors:
Noam Shazeer
Azalia Mirhoseini
Krzysztof Maziarz
Andy Davis
Quoc Le
Geoffrey Hinton
Jeff Dean

title: Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

venue: arXiv preprint arXiv:1701.06538,

authors:
Aditya Siddhant
Ankur Bapna
Yuan Cao
Orhan Firat
Mia Chen
Sneha Kudugunta
Naveen Arivazhagan
Yonghui Wu

title: Leveraging monolingual data with self-supervision for multilingual neural machine translation

authors:
Sainbayar Sukhbaatar
Edouard Grave
Piotr Bojanowski
Armand Joulin

title: Adaptive attention span in transformers

venue: In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,

authors:
Xu Tan
Jiale Chen
Di He
Yingce Xia
Tao QIN
Tie-Yan Liu

title: Multilingual neural machine translation with language clustering

title: Parallel data, tools and interfaces in opus

venue: Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12),

authors:
Ashish Vaswani
Noam Shazeer
Niki Parmar
Jakob Uszkoreit
Llion Jones
Aidan N Gomez
Ł ukasz Kaiser
Illia Polosukhin

title: Attention is all you need

venue: Advances in Neural Information Processing Systems

authors:
Raúl Vázquez
Alessandro Raganato
Jörg Tiedemann
Mathias Creutz

title: Multilingual NMT with a language-independent attention bridge

venue: In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019),

authors:
Xinyi Wang
Yulia Tsvetkov
Graham Neubig

title: Balancing training for multilingual neural machine translation

venue: arXiv preprint arXiv:2004.06748,

authors:
Yining Wang
Jiajun Zhang
Feifei Zhai
Jingfang Xu
Chengqing Zong

title: Three strategies to improve one-to-many multilingual translation

authors:
Yining Wang
Long Zhou
Jiajun Zhang
Feifei Zhai
Jingfang Xu
Chengqing Zong

title: A compact and language-sensitive multilingual translation method

authors:
Zirui Wang
Zachary C. Lipton
Yulia Tsvetkov

title: On negative interference in multilingual models: Findings and a meta-learning treatment

venue: In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),

authors:
Biao Zhang
Ivan Titov
Rico Sennrich

title: On sparsifying encoder outputs in sequence-to-sequence models

venue: arXiv preprint arXiv:2004.11854,

authors:
Biao Zhang
Philip Williams
Ivan Titov
Rico Sennrich

title: Improving massively multilingual neural machine translation and zero-shot translation

authors:
Changfeng Zhu
Heng Yu
Shanbo Cheng
Weihua Luo

title: Language-aware interlingua for multilingual neural machine translation

venue: In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1650–1655,

authors:
Barret Zoph
Kevin Knight

title: Multi-source neural translation

heading: 1 INTRODUCTION

text:
Model architecture design injects inductive biases to neural network layouts, allowing a learning algorithm to favor certain representations over others, independent of the observed data (Mitchell, 1980). In multilingual neural machine translation (MNMT), where the learning objective is commonly cast as a multi-task learning problem (Firat et al., 2016a; Ha et al., 2016; Johnson et al., 2017), the inductive bias researchers usually study is deciding on which components of the neural network to share between tasks (languages), and which components to leave specific to the task or language. These components can be entire layer stacks, individual layers or even some sub-layers (Sachan & Neubig, 2018; Blackwood et al., 2018; Wang et al., 2019; Zhu et al., 2020). Noticeably, the search space of which parameters to share and at which granularity grows rapidly, as we make neural networks large or increase the number of tasks (languages). This rapid expansion of the search space prevents us from exhaustively exploring the choice of sharing patterns in MNMT.
The incapability of full-space exploration motivates methods relying on heuristics (Sachan & Neubig, 2018), that lack flexibility when more languages are covered, or meta-learning (Platanios et al., 2018), that are often hard to scale. These limitations hinder their generalization to large-scale multilingual models, which is the very focus of our study. In large scale multilingual models, also known as massively multilingual models (Aharoni et al., 2019; Arivazhagan et al., 2019; Zhang et al., 2020b), hundreds of languages with varying amounts of training data, difficulty and linguistic properties are jointly trained together in a multi-task setup. While the joint training enables positive
∗Work done while Biao Zhang was interning at Google Research.
transfer across languages, it also introduces task-interference between dissimilar languages (Arivazhagan et al., 2019; Wang et al., 2020a;b) and a capacity bottleneck emerges due to the increased number of languages and data (Huang et al., 2019; Zhang et al., 2020b).
In this paper we adopt an end-to-end data driven approach (conditional language-specific routing, or CLSR) which permits directly probing a large section of the search space. We let the network learn the sharing structure from the data itself, by learning to route between language-specific (LS) or shared pathways. These two routes determine the mode of operation for the network: when the LS branch is selected, the model is given access to a set of LS layers (implemented as simple projections per language) and when the shared branch is chosen, the computation is routed to a layer that is used by all languages. By guiding the (gating) decision process with token level activation information, the network flexibly learns to alternate between the two modes and naturally lends itself to a conditional computation approach for multilingual processing (Bengio et al., 2013; Davis & Arel, 2013; Bapna et al., 2020). The gate states are optimized towards maximizing translation quality, but regularized with a budget constraint to control the amount of LS capacity1. Reducing the available budget results in fewer gates routing through the LS paths, enforcing CLSR to identify the most crucial sub-layers which allows us to observe and study the importance of each sub-layer for multilingual processing. Our approach is visually depicted in Figure 1.
We verify our proposal on WMT and the massively multilingual OPUS-100 dataset, with models building on the Transformer architecture (Vaswani et al., 2017). We explore target-specific and source-specific modeling for one-to-many2 and many-to-one translation, respectively. To measure the degree of each sub-layer’s tendency to be language-specific, we propose LSScore metric. Our results show that CLSR successfully navigates the trade-offs in LS modeling, outperforming several strong baselines. Our main findings are summarized below:
• Both the amount and the position of LS layers matter for MNMT. The best performance is achieved by distributing 10%-30% LS computation to the top and/or bottom encoder/decoder layers.
• Feed-forward sub-layers utilize more LS capacity compared to other sub-layers on one-tomany translation.
• One-to-many translation benefits more from CLSR (with target LS parameters) compared to many-to-one translation (with source LS parameters), particularly when the training data is imbalanced.
• The induced sharing pattern learned by CLSR is highly similar across languages. 1We use the term “the amount of LS capacity” to refer to the proportion of open gates where CLSR selects to route information through the LS path instead of its shared counterpart, which is directly regularized and guided by the budget constraint p as in Eq. 6.
2In a one-to-many machine translation setup, a single source side language (commonly English) is tasked to be translated into multiple target languages, one at a time.
• We can use the learned patterns to hard-code better parameter sharing strategies for multilingual Transformers.

heading: 2 RELATED WORK

text:
Our work closely relates to language-specific (LS) modeling for multilingual NMT and conditional computation for sequential data which we will recap both here. Early research on MNMT focused on improving shared capacity for separate bilingual models to enhance cross-lingual transfer. These efforts included sharing encoders for one-to-many translation (Dong et al., 2015), sharing decoders for many-to-one translation (Zoph & Knight, 2016; Lee et al., 2017) and sharing sub-layers (attention) for many-to-many translation (Firat et al., 2016a;b). These studies corroborated the feasibility of accommodating multiple languages with shared NMT sub-components, motivating researchers to explore universal MNMT. Ha et al. (2016) and Johnson et al. (2017) proposed such an implementation that performs multilingual translation with a single monolithic NMT model where the entire network is shared across languages, thanks to a target language token informing the model which language to translate into. Although this paradigm shows great scalability (Aharoni et al., 2019), the language token alone affords little flexibility in handling language diversity with a rigid share-all layout. Follow-up studies thus resort to LS modeling in an attempt to seek a better trade-off between sharing and not sharing. Methods in this category involve specializing neural attentions (Blackwood et al., 2018; Sachan & Neubig, 2018; Wang et al., 2019), broadening encoder outputs and normalizations (Zhang et al., 2020b), decoupling multilingual encoders and/or decoders (Vázquez et al., 2019; Escolano et al., 2020), using a fixed mix of LS and shared parameters (Wang et al., 2018), inserting lightweight adapters (Bapna & Firat, 2019) and separately modeling languages for different clusters (Tan et al., 2019), to name a few. Nevertheless, these methods heavily depend on heuristics, providing little evidence about how to optimally distribute LS capacity across the model.
By contrast, our proposed CLSR forces the model to learn LS behaviour. It can be treated as a simplified differentiable neural architecture search (NAS) model (Liu et al., 2019) with a search space defined by the presence/absence of LS projections after every transformer sub-layer. However, in contrast with NAS, we utilize conditional computation (Bengio et al., 2013) to make the choice of executing the shared or LS path conditional on the input representations. This allows us to compare and contrast the choice of LS vs shared paths on different inputs and languages. Conditional computation has previously been successfully applied to adapt the amount of computation to the input in recurrent models (Graves, 2016) and transformers (Dehghani et al., 2019; Elbayad et al., 2020), or to significantly scale up model capacity by utilizing sparsely-gated Mixture-of-Experts layers (Shazeer et al., 2017; Lepikhin et al., 2020). Zhang et al. (2020a) applied conditional computation to sparsify encoder outputs in sequence-to-sequence models in order to reduce attention costs, while Bapna et al. (2020) introduced conditional execution of Transformer sub-layers to control the amount of computation expended by the model at inference. Sukhbaatar et al. (2019) learn parameters that limit the attention spans, in order to make the attention operation more efficient, while Fan et al. (2020) utilize structured dropout to prune transformer layers at inference. Ruder et al. (2019) propose the sluice network that learns the inter-task (shared) sub-spaces on top of task-specific models. By contrast, CLSR starts with a totally shared model, and learns how to inject task-specific projections into it, which scales more easily to massively multilingual settings. Different from previous studies, we explore conditional computation as an analysis tool to understand the ideal arrangement of LS capacity for MNMT. Utilizing conditional computation to search for better LS sharing patterns in multilingual translation, to the best of our knowledge, has never been investigated before.

heading: 3 BACKGROUND: MNMT

text:
Given a source sentence X ′ = {x1, x2, . . . , xI} and its target translation Y = {y1, y2, . . . , yJ}, we follow Johnson et al. (2017) to reuse standard bilingual NMT models for multilingual translation by altering the source input with a language token lang, i.e. changing X ′ to X = {lang, x1, x2, . . . , xI}. Note that lang denotes the target language in one-to-many translation but source language in many-to-one translation.
We model translation from X to Y with Transformer (Vaswani et al., 2017). Transformer relies on the following residual-normalization structure (He et al., 2015; Ba et al., 2016) to smooth informa-
tion flow and avoid gradient vanishing and explosion: zl+1 = LN ( zl + f ( zl )) , (1)
where l denotes layer depth and LN(·) is layer normalization (Ba et al., 2016). Function f(·) represents the basic building block in Transformer, such as attention network or feed-forward network. The encoder in Transformer is a stack of L identical layers, with each layer involving a self-attention sub-layer (SAN) and a feed-forward sub-layer (FFN). The decoder uses a similar structure except for an extra cross-attention sub-layer (CAN) inserted in-between the above two sub-layers.

heading: 4 CONDITIONAL LANGUAGE-SPECIFIC ROUTING (CLSR)

text:
The success of MNMT comes at the cost of expressivity and model’s ability to capture languagespecific characteristics. It has been empirically shown that the language signals from language indicator tokens alone are not sufficient (Arivazhagan et al., 2019), making architectures dedicated to LS modeling a necessity (Blackwood et al., 2018; Sachan & Neubig, 2018; Zhang et al., 2020b). Nevertheless, the question when and where LS modeling matters most in MNMT still remains to be answered. To this end, we propose conditional language-specific routing (CLSR) which specializes f(·) and changes the formulation in Equation 1 as follows:
zl+1 = LN ( zl + CLSR ( f ( zl ))) . (2)
CLSR learns a hard binary (scalar-valued) gate g(·) for each input token based on its hidden representation zl ∈ Rd. These gates endow each sub-layer in Transformer with the capability of routing information selectively through either LS path hlang or shared path hshared:
CLSR ( f ( zl ))
= g(zl)hlang + (1− g(zl))hshared, (3) with hlang = f ( zl ) Wlang, hshared = f ( zl ) Wshared, (4)
where Wshared is a weight matrix shared across languages, while parameter Wlang is only used for modeling language lang which endows NMT with source or target LS modeling capacity.3 Intuitively, a closed gate, corresponding to shared capacity, encourages maximal cross-lingual information transfer; an open gate, corresponding to LS capacity instead, improves language awareness for translation albeit it blocks knowledge transfer. CLSR balances between the two modes as controlled by the gates.
Following Bapna et al. (2020), we parameterize the gate g(·) with a two-layer feed-forward network G(·), and inject zero-mean Gaussian noise during training to discretize it (Chiu & Raffel, 2018) :
g(zl) = σ ( G(zl) + α(t)N (0, 1) ) , G(zl) = Relu ( zlW1 + b ) w2, (5)
where σ(·) is the logistic-sigmoid function, dg is the gating feed-forward hidden dimension, and W1 ∈ Rd×dg ,w2 ∈ Rdg are trainable parameters. α is linearly increased along with training steps t. In this way, the gating parameters can be optimized with accurate gradients when α is small at the beginning so as to measure the degree to which each position in each sub-layer benefits from LS modeling. As training progresses, α grows larger, forcing the gating network to emit hard outputs. At inference time, we discretize the gate based on a simple decision rule: g(zl) = δ ( G(zl) ≥ 0 ) , where δ(·) is a Dirac measure. We train the gates based on the standard maximum likelihood objective, along with an additional budget regularization term that enables control over the amount of LS capacity used for translation. Let the set of all CLSR layers in the encoder beMenc and the decoder beMdec. Then the amount of LS computation utilized by a sentence pair (X,Y ) is given byG(X,Y ) = ∑ x∈X ∑ m∈Menc gm(x)+∑
y∈Y ∑
m∈Mdec gm(y). Given a budget constraint p ∈ [0, 1] and a batch of sentence pairs, B, the training loss function of CLSR is formulated below:
L (B) = ∑
(X,Y )∈B
MLE (X,Y ) + ∣∣∣∣∣ ∑ (X,Y )∈BG(X,Y )∑ (X,Y )∈B(|X||Menc|+ |Y ||Mdec|) − p ∣∣∣∣∣ , (6)
3We maintain a set of language-specific weight matrices in order to compute hlang for each language. To make the number of parameters manageable, we share the set of LS matrices Wlang across all encoder or decoder sub-layers, but distinguish Wlangenc and W lang dec , LS matrices for the encoder and decoder, respectively.
Intuitively, the budget constraint tries to regulate the amount of LS computation available to all tokens in the batch as a fraction p of the total LS computation in the model. Given that we make a binary decision for every input for every CLSR layer, this corresponds to a search space of O(2|X||Menc|+|Y ||Mdec|). This gating space not only grows with respect to the model depth and sub-layer types, but also is highly input dependent. This dependency makes it difficult to search the entire space using heuristic methods (Blackwood et al., 2018).
The constraint in Equation 6 is imposed upon the aggregated gates. As a consequence, the model can learn to trade-off LS capacity for certain layers and inputs for others. Decreasing the budget encourages gate closure, such that the LS path is chosen only in the critical sub-layers. Thus, a properly learned gating function reveals the activations of LS paths and helps gain insights into the model behavior.

heading: 5 EXPERIMENTS

text:
Data and Evaluation We report results on two benchmarks: OPUS-100 (Zhang et al., 2020b) and WMT-14 (Barrault et al., 2019). OPUS-100 is a massively multilingual dataset collected from OPUS (Tiedemann, 2012), including 100 languages in total with 99 languages to-and-from English.4 It consists of 55M training sentence pairs with up to 1M samples per language pair, and covers 94 dev/test language pairs, each with 2000 samples at most. WMT-14 is another multilingual Englishcentric dataset composed of 13 widely-used WMT benchmarks following Siddhant et al. (2020) but excluding Kazakh and Gujarati due to their poor parallel resource. Compared to OPUS-100, WMT14 involves much fewer languages but its training data distribution is highly skewed across diverse language pairs, ranging from 0.2M (En-Tr) to 60M (En-Cs) training examples, thus posing severe challenges. We show more details about the train, dev and test data for WMT-14 in Appendix A.
We apply byte pair encoding (BPE) algorithm (Sennrich et al., 2016) using SentencePiece (Kudo & Richardson, 2018) to preprocess multilingual sentences with a vocabulary size of 64K. We use BLEU (Papineni et al., 2002), offered by SacreBLEU (Post, 2018)5, for translation evaluation. Following Zhang et al. (2020b), we split the 94 test language pairs in OPUS-100 into three groups based on their training data size to ease model evaluation: high resource (>0.9M, 45 languages), low resource (<0.1M, 26 languages) and medium resource (others, 28 languages). Similarly, we split the 13 test language pairs in WMT-14 as follows: High (>10M, 5), Low (<1M, 5) and Med (others, 3).
We perform experiments for one-to-many translation (O2M) and many-to-one translation (M2O). In addition to using the original training data as is, we also report results with a temperature-based strategy to balance the training data distribution by over-sampling low-resource languages with a temperature of T = 5 (Arivazhagan et al., 2019).
We report average BLEU, and also show win ratio (Zhang et al., 2020b, WR), informing the proportion of language pairs on which our method beats our baseline. To evaluate how often each sub-layer is LS, we introduce a new metric, LSScore, formulated as follows:
LSScoref (l, p) = 1 |P| ∑ p∈P g̃pl,f − p, (7)
where g̃pl,f denotes the gating value averaged over all test tokens for the l-th sub-layer f(·) trained under a budget of p. Recall that we pose the budget constraint to the summed gate value instead of each individual gate, as in Equation 6. This gives CLSR the freedom to close more gates in some language-insensitive sub-layers (i.e. g̃pl,f < p) while preserve more for the others (i.e. g̃ p l,f > p). A larger LSScore, > 0 in particular, indicates that this sub-layer utilizes more LS modeling.
Model Settings We adapt Transformer base for our experiments: L = 6, d = 512, 8 attention heads with FFN middle size of 2048. Dropout of rate 0.1 is applied to residual connections and attention weights. We optimize parameters using Adam (Kingma & Ba, 2015) (β1 = 0.9, β2 = 0.98) with label smoothing of 0.1. Learning rate is scheduled according to the inverse square root of running steps with a warmup step of 4K (Vaswani et al., 2017). We limit training sequence length
4http://opus.nlpl.eu/opus-100.php 5Signature: BLEU+case.mixed+numrefs.1+smooth.exp+tok.13a+version.1.4.10.
to 100, and train all models with a batch size of 1920. We set the maximum running step to 500K and 600K for OPUS-100 and WMT-14, respectively. We perform beam search decoding with beam size of 4 and length penalty of 0.6. We average the last 5 checkpoints for evaluation. For CLSR, we set dg to 128, and linearly increase α from 0 to 5 along training steps (Bapna et al., 2020).6 We vary the budget p in the range of P = {0.0, 0.1, 0.3, 0.5, 0.7, 0.9, 1.0} to study its impact on model performance.

heading: 5.1 RESULTS ON OPUS-100

text:
On the trade-off between shared and language-specific capacity. Using more LS modeling fails to deliver increasingly better translation performance, as shown in Figure 2(a) and 2(b) when p approaches 1.0. At the point of full LS capacity for Transformer, i.e. p = 1.0, CLSR even underperforms its corresponding multilingual baseline by a large margin of 1.0-2.0 BLEU on M2O, Figure 2(b). Similarly, sharing all model parameters across language pairs, p → 0.0, also yields suboptimal performance with either original or oversampled training data. CLSR achieves its best translation quality at p = 0.3 (+2.0/3.0 BLEU) and p = 0.1 (+0.5/1.5 BLEU) for O2M and M2O, respectively. The win ratio curves in Figure 2(c) and 2(d) further confirm the robustness of these quality improvements, where properly scheduling LS capacity outperforms the baselines on >∼ 80% language pairs. These results clearly show the trade-off between these two kinds of capacity.
When does language-specific capacity matter for multilingual translation? Results in Figure 2 show that O2M favors more LS modeling and benefits more from it compared to M2O (p = 0.3 vs. p = 0.1, and +3.0 vs. +1.5 BLEU on the original training data). We conjecture that translations on M2O share the same target language (English), so information can be easily transferred through shared modeling; by contrast, MNMT has to handle languages of diverse typological features for O2M, demanding stronger capacity delivered to each translation direction. Results in Figure 2 also show that CLSR yields more aggressive improvements on the original training data, compared to the oversampled counterpart (+3.0 vs. +2.0 BLEU on O2M and +1.5 vs. +0.5 BLEU on M2O). Finegrained analysis on each resource group, as shown in Figure 7 (Appendix B), reveals that CLSR yields more improvements for low-resource translation where the oversampling strategy partially offsets these improvements.
Where should we add language-specific capacity in multilingual Transformer? Figure 2 suggests that CLSR uses 10-30% LS capacity to reach its best performance. We next study how CLSR schedules this capacity across all Transformer sub-layers, in order to determine the ideal arrangement of LS layers for translation. Figure 3 shows the results. Regardless of layer types, CLSR schedules more LS capacity to the top and/or bottom encoder/decoder layers rather than the middle ones. We find that more LS capacity is allocated to feed-forward sub-layers for O2M (both encoder Figure 3(a) and decoder Figure 3(b)), while the cross-attention sub-layers use more LS capacity for M2O (decoder, Figure 3(d)). Regarding the encoder for M2O, we observe no significant LSScore
6Note that, there are 99×2 and 13×2 independent Wlang weight matrices when CLSR is used for OPUS100 and WMT-14, respectively. This corresponds to each language having a weight matrix in the encoder and the decoder.
difference among different sub-layers as in Figure 3(c). Overall, CLSR tends to make the “M” side more LS, i.e. the encoder side of M2O (Figure 3(c)) and the decoder side of O2M (Figure 3(b)).
Does CLSR schedule language-specific capacity based on linguistic similarity? It is intriguing to explore how the capacity scheduled by CLSR is actualized, especially whether CLSR learns to organize LS capacity according to linguistic characteristics or not. The heatmap in Figure 6, Appendix B shows that the LSScore distribution over different sub-layers (y-axis) has only subtle difference across different language pairs (x-axis). This suggests that the capacity schedule has little to do with linguistic characteristics. More results in other settings are given in Figure 6, Appendix B, which reflect similar observation. In short, CLSR allocates LS capacity to specific sub-layers rather than specific languages. This could be ascribed to the design of CLSR. CLSR shares the gating parameters and the budget, p, across all languages, which might impose some inductive bias discouraging its LS behavior. Besides, the structure of the gating in CLSR (Eq. 5) might fail to offer enough flexibility for controlling the gates in different ways across languages and layers. We leave further study of LS gating to future work.
On detailed results and comparison to other baselines. Table 1 summarizes our results.7 Although LS (Zhang et al., 2020b) improves O2M, it fails to surpass the vanilla Baseline on M2O, indicating that the position of its LS layer, i.e. on top of the encoder outputs, is sub-optimal for
7We list the number of trainable parameters for each model in Table 6, Appendix D.
many-to-one translation. Compared to LS , CLSR-S uses no LS modeling while CLSR-L injects LS projection into each sub-layer, both of which delivers inferior performance on O2M (against LS and Baseline) and M2O (against Baseline), echoing our findings from Figure 2. By contrast, CLSR?, which uses an optimized budget p, yields promising results, beating Baseline on O2M and M2O, underlining the importance of seeking balances between sharing and not sharing.
Can we use these insights to improve multilingual Transformer? We answer this question by transferring our findings from Figure 3 into the following two Transformer variants: one enhances the top and bottom encoder/decoder layers with LS projections (Top-Bottom), while the other makes high-LSScore sub-layers LS (Dedicated).8 Results of these experiments, in Table 1, demonstrate that incorporating our findings into Transformer helps to improve translation performance. The dedicated model in particular recovers and even surpasses the performance of CLSR?.

heading: 5.2 RESULTS ON WMT-14

text:
Figure 4 shows the capacity trade-off on WMT-14, reconfirming the ability of CLSR. One noticeable difference is that the relative improvements become smaller. We ascribe this to the smaller number of language pairs in WMT-14, where the effect of introducing LS capacity into the model is smaller compared to the massively multilingual setting (Arivazhagan et al., 2019). Table 2 shows similar results as Table 1, where CLSR? outperforms both fully-shared (CLSR-S) and fully-LS (CLSRL) baselines, and both Top-Bottom and Dedicated improve multilingual translation compared to Baseline (except for M2O with oversampling). More results are given in Appendix C. Our results demonstrate that the CLSR approach generalizes to different datasets, with varying number of lan-
8Detail about which sub-layers are specialized in Dedicated is given in Table 4, Appendix B.
guages and resource sizes. We provide additional experiments ablating different elements on CLSR in Table 7, Appendix E.

heading: 6 CONCLUSION AND FUTURE WORK

text:
Share or not? This is an open question when developing MNMT models. In this paper, we attempt to answer this question by proposing conditional language-specific routing (CLSR). Our empirical results demonstrate that CLSR learns to balance between shared and LS paths across all NMT sublayers, improving the quality of multilingual translation. Our analysis on OPUS-100 and WMT-14 suggest that both the position and the amount of LS capacity greatly affects MNMT. Scheduling 10%-30% LS layers to the top and/or bottom encoder/decoder layers reaches CLSR’s peak performance. We also demonstrate how our findings can be leveraged to design a multilingual Transformer with an optimal sharing pattern. We believe that our work improves our understanding on the tradeoff between sharing and not sharing, paving the way for better multilingual models.
In the future, we plan to extend our study to many-to-many translation as well as larger-capacity models. We also plan to adapt CLSR to other multilingual multi-task learning settings to better handle knowledge transfer among different tasks, especially for cross-lingual downstream transfer.

heading: ACKNOWLEDGEMENTS

text: We would like to thank Yuan Cao for his valuable feedback. We would also like to thank the Google Translate team for their constructive discussions and comments. We thank the reviewers for their insightful comments. Rico Sennrich has received funding from the Swiss National Science Foundation (project no. 176727).

heading: A DATASET DETAILS

text: Table 3 lists the statistics for WMT-14. We collect these datasets following Siddhant et al. (2020). Figure 5 shows the training data distribution. Training data in WMT-14 is more imbalanced across different language pairs compared to OPUS-100.

heading: B MORE RESULTS ON OPUS-100

text:
Figure 7 shows the translation performance of CLSR on each resource group when varying the budget p. We find that LS modeling greatly improves translation on low-resource settings, although
this improvement is partially offset by the oversampling strategy. We observe that the success of oversampling is built on top of sacrificing quality on high-resource languages, where CLSR largely narrows or even closes the performance gap against the multilingual baseline trained on the original
data. In addition, we find no significant difference in term of the trade-off curve across resource groups compared to Figure 2.
Figure 6 shows the heatmaps for LSScore distribution. Regardless of translation directions (O2M or M2O) and data settings (Original or Oversample), there is no clear linguistic pattern behind these heatmaps. The learning of CLSR is not linguistic driven.

heading: C MORE RESULTS ON WMT-14

text:
We show more results on WMT-14 here. Both heatmaps in Figure 9 and fine-grained trade-off curves in Figure 8 show a very similar story to the one on OPUS-100. In terms of LSScore distribution in Figure 10(d), we also have similar observation except that the cross-attention sub-layer in the decoder on M2O shows no clear LS preference.
These results suggest the high generalization of CLSR to different data settings.

heading: D MODEL PARAMETERS

text: We list model parameters in Table 6. Note that the source/target vocabulary for O2M is the same as the target/source vocabulary for M2O, thus models for O2M and M2O have the same number of trainable parameters except for Dedicated where the network structure differs.

heading: E ABLATION STUDY

text:
We provide two variants of CLSR for ablation. The first one examines whether CLSR could induce an optimal budget automatically by eschewing the budget constraint in Eq. 6 (CLSR w/o p); the second one further replaces the hard gating network with a vanilla soft gating network (CLSR-Gate).
Table 7 summarizes the translation results. Without the regularization term, CLSR w/o p yields an average gating value of 0.37/0.16 and 0.39/0.19 for O2M/M2O on OPUS-100 and WMT-14, respectively9, resonating with our finding that O2M requires more LS capacity compared to M2O. Compared to CLSR?, CLSR w/o p produces more open gates, achieving comparable but overall worse performance in all data settings. This highlights the importance of the added regularization term, where without such regularization the model is likely spending extra learning cycles to automatically optimize for the budget as with CLSR w/o p. This in return hints at the importance of optimization routines (learning) which we held constant in this study.
By contrast, switching to soft gating network, CLSR-Gate gains translation quality in most settings, although the quality gains are mostly less than 0.1 compared to CLSR?. Note that, CLSR-Gate relies on continuous gates, indicating a much higher flexibility in balancing parameter sharing patterns. However, we also notice that CLSR-Gate performs worse on M2O, even fails to beat the vanilla Baseline (All) on the oversampled WMT-14. Unlike CLSR-Gate, CLSR? uses hard binary gates which eases capacity analysis and achieves consistent improvements across data settings.
9We also average the gating value over the original and oversampled data settings.