summaries: In this paper, the authors present a study of different aspects of language-specific model capacity for massively multilingual machine translation. To this end, language-specific behaviour is achieved via a combination of conditional computation to decide whether to use language-specific parameters or not and statically assigning experts for each languages. The language specific sub-layers are incorporated throughout the network. The training objective allow budgetary constraints on the amount of language-specific parameters. The paper does a systematic analysis on the role of language specific parameters using the proposed architecture. Based on the analysis, recommendations on design of multilingual NMT architectures are proposed and their efficacy validated experimentally. The study sheds light on the amount of language specific parameter sharing, their distribution in the network, impact of language, etc.  The work proposes a hybrid architecture that has: (1) language-specific (LS) components; (2) as well as the components that are shared across all the languages -- a trade-off between specificity and generality.  A key conclusion of the work is that the best architectures typically are. the ones that have ~10-30% language-specific capacity.  In this work, the authors present a conditional language-specific routing (CLSR) scheme for transformer-based multilingual NMT systems. They introduce a CLSR layer after every transformer encoder and decoder layer; each such layer is made up of hard gating functions conditioned on token representations that will either select a language-specific projection layer or a shared projection layer. Further, a budget is imposed on the language-specific capacity measured by aggregating the number of gates that allow for language-specific computations; this budget constraint forces the network to identify the sub-layers that will benefit most from being language-specific.
abstractText: Using a mix of shared and language-specific (LS) parameters has shown promise in multilingual neural machine translation (MNMT), but the question of when and where LS capacity matters most is still under-studied. We offer such a study by proposing conditional language-specific routing (CLSR). CLSR employs hard binary gates conditioned on token representations to dynamically select LS or shared paths. By manipulating these gates, it can schedule LS capacity across sub-layers in MNMT subject to the guidance of translation signals and budget constraints. Moreover, CLSR can easily scale up to massively multilingual settings. Experiments with Transformer on OPUS-100 and WMT datasets show that: 1) MNMT is sensitive to both the amount and the position of LS modeling: distributing 10%-30% LS computation to the top and/or bottom encoder/decoder layers delivers the best performance; and 2) one-to-many translation benefits more from CLSR compared to many-to-one translation, particularly with unbalanced training data. Our study further verifies the trade-off between the shared capacity and LS capacity for multilingual translation. We corroborate our analysis by confirming the soundness of our findings as foundation of our improved multilingual Transformers. Source code and models are available at https://github.com/bzhangGo/zero/tree/iclr2021_clsr.
authors.name: PACITY FOR
authors.name: MULTILINGUAL TRANSLATION
authors.name: Biao Zhang
authors.name: Ankur Bapna
authors.name: Rico Sennrich
authors.name: Orhan Firat
references.authors: Roee Aharoni Melvin Johnson Orhan Firat
references.title: Massively multilingual neural machine translation
references.authors: Naveen Arivazhagan Ankur Bapna Orhan Firat Dmitry Lepikhin Melvin Johnson Maxim Krikun Mia Xu Chen Yuan Cao George Foster Colin Cherry Wolfgang Macherey Zhifeng Chen Yonghui Wu
references.title: Massively multilingual neural machine translation in the wild: Findings and challenges
references.venue: URL http://arxiv.org/abs/1907.05019
references.authors: Ankur Bapna Orhan Firat
references.title: Simple, scalable adaptation for neural machine translation
references.venue: In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),
references.authors: Ankur Bapna Naveen Arivazhagan Orhan Firat
references.title: Controlling computation versus quality for neural sequence models
references.venue: arXiv preprint arXiv:2002.07106,
references.authors: Yoshua Bengio Nicholas Léonard Aaron Courville
references.title: Estimating or propagating gradients through stochastic neurons for conditional computation
references.venue: arXiv preprint arXiv:1308.3432,
references.authors: Graeme Blackwood Miguel Ballesteros Todd Ward
references.title: Multilingual neural machine translation with task-specific attention
references.venue: In Proceedings of the 27th International Conference on Computational Linguistics,
references.authors: Chung-Cheng Chiu Colin Raffel
references.title: Monotonic chunkwise attention
references.venue: In International Conference on Learning Representations,
references.authors: Andrew Davis Itamar Arel
references.title: Low-rank approximations for conditional feedforward computation in deep neural networks
references.authors: Mostafa Dehghani Stephan Gouws Oriol Vinyals Jakob Uszkoreit Lukasz Kaiser
references.title: Universal transformers
references.venue: In International Conference on Learning Representations,
references.authors: Daxiang Dong Hua Wu Wei He Dianhai Yu Haifeng Wang
references.title: Multi-task learning for multiple language translation
references.venue: In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers),
references.authors: Maha Elbayad Jiatao Gu Edouard Grave Michael Auli
references.title: Depth-adaptive transformer
references.venue: In International Conference on Learning Representations,
references.authors: Carlos Escolano Marta R Costa-jussà José AR Fonollosa Mikel Artetxe
references.title: Multilingual machine translation: Closing the gap between shared and language-specific encoder-decoders
references.venue: arXiv preprint arXiv:2004.06575,
references.authors: Angela Fan Edouard Grave Armand Joulin
references.title: Reducing transformer depth on demand with structured dropout
references.venue: In International Conference on Learning Representations,
references.authors: Orhan Firat Kyunghyun Cho Yoshua Bengio
references.title: Multi-way, multilingual neural machine translation with a shared attention mechanism
references.venue: In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,
references.authors: Orhan Firat Baskaran Sankaran Yaser Al-onaizan Fatos T. Yarman Vural Kyunghyun Cho
references.title: Zero-resource translation with multi-lingual neural machine translation
references.venue: In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing,
references.authors: Alex Graves
references.title: Adaptive computation time for recurrent neural networks
references.venue: arXiv preprint arXiv:1603.08983,
references.authors: Thanh-Le Ha Jan Niehues Alexander Waibel
references.title: Toward multilingual neural machine translation with universal encoder and decoder
references.venue: In Proceedings of the 13th International Workshop on Spoken Language Translation (IWSLT),
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.venue: CoRR, abs/1512.03385,
references.authors: Yanping Huang Youlong Cheng Ankur Bapna Orhan Firat Dehao Chen Mia Chen HyoukJoong Lee Jiquan Ngiam Quoc V Le Yonghui Wu
references.title: Gpipe: Efficient training of giant neural networks using pipeline parallelism
references.venue: In Advances in neural information processing systems,
references.authors: Melvin Johnson Mike Schuster Quoc V. Le Maxim Krikun Yonghui Wu Zhifeng Chen Nikhil Thorat Fernanda Viégas Martin Wattenberg Greg Corrado Macduff Hughes Jeffrey Dean
references.title: Google’s multilingual neural machine translation system: Enabling zero-shot translation
references.venue: Transactions of the Association for Computational Linguistics,
references.authors: Diederik P Kingma Jimmy Ba
references.title: Adam: A method for stochastic optimization
references.venue: In International Conference on Learning Representations,
references.authors: Taku Kudo John Richardson
references.title: SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing
references.venue: In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,
references.authors: Jason Lee Kyunghyun Cho Thomas Hofmann
references.title: Fully character-level neural machine translation without explicit segmentation
references.venue: Transactions of the Association for Computational Linguistics,
references.authors: Dmitry Lepikhin HyoukJoong Lee Yuanzhong Xu Dehao Chen Orhan Firat Yanping Huang Maxim Krikun Noam Shazeer Zhifeng Chen
references.title: Gshard: Scaling giant models with conditional computation and automatic sharding
references.authors: Hanxiao Liu Karen Simonyan Yiming Yang
references.title: DARTS: Differentiable architecture search
references.venue: In International Conference on Learning Representations,
references.authors: Tom M Mitchell
references.title: The need for biases in learning generalizations. Department of Computer Science, Laboratory for Computer Science Research
references.authors: Kishore Papineni Salim Roukos Todd Ward Wei-Jing Zhu
references.title: Bleu: a method for automatic evaluation of machine translation
references.venue: In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics,
references.authors: Emmanouil Antonios Platanios Mrinmaya Sachan Graham Neubig Tom Mitchell
references.title: Contextual parameter generation for universal neural machine translation
references.venue: In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,
references.authors: Matt Post
references.title: A call for clarity in reporting BLEU scores
references.venue: In Proceedings of the Third Conference on Machine Translation: Research Papers,
references.authors: Sebastian Ruder Joachim Bingel Isabelle Augenstein Anders Søgaard
references.title: Latent multi-task architecture learning
references.venue: Proceedings of the AAAI Conference on Artificial Intelligence,
references.authors: Devendra Sachan Graham Neubig
references.title: Parameter sharing methods for multilingual self-attentional translation models
references.venue: In Proceedings of the Third Conference on Machine Translation: Research Papers,
references.authors: Rico Sennrich Barry Haddow Alexandra Birch
references.title: Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1715–1725, Berlin, Germany, August 2016
references.venue: Association for Computational Linguistics. doi: 10.18653/v1/P16-1162. URL https://www.aclweb. org/anthology/P16-1162
references.authors: Noam Shazeer Azalia Mirhoseini Krzysztof Maziarz Andy Davis Quoc Le Geoffrey Hinton Jeff Dean
references.title: Outrageously large neural networks: The sparsely-gated mixture-of-experts layer
references.venue: arXiv preprint arXiv:1701.06538,
references.authors: Aditya Siddhant Ankur Bapna Yuan Cao Orhan Firat Mia Chen Sneha Kudugunta Naveen Arivazhagan Yonghui Wu
references.title: Leveraging monolingual data with self-supervision for multilingual neural machine translation
references.venue: In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,
references.authors: Sainbayar Sukhbaatar Edouard Grave Piotr Bojanowski Armand Joulin
references.title: Adaptive attention span in transformers
references.venue: In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,
references.authors: Xu Tan Jiale Chen Di He Yingce Xia Tao QIN Tie-Yan Liu
references.title: Multilingual neural machine translation with language clustering
references.venue: In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),
references.authors: Jörg Tiedemann
references.title: Parallel data, tools and interfaces in opus
references.venue: Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12),
references.authors: Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Ł ukasz Kaiser Illia Polosukhin
references.title: Attention is all you need
references.venue: Advances in Neural Information Processing Systems
references.authors: Raúl Vázquez Alessandro Raganato Jörg Tiedemann Mathias Creutz
references.title: Multilingual NMT with a language-independent attention bridge
references.venue: In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019),
references.authors: Xinyi Wang Yulia Tsvetkov Graham Neubig
references.title: Balancing training for multilingual neural machine translation
references.venue: arXiv preprint arXiv:2004.06748,
references.authors: Yining Wang Jiajun Zhang Feifei Zhai Jingfang Xu Chengqing Zong
references.title: Three strategies to improve one-to-many multilingual translation
references.venue: In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,
references.authors: Yining Wang Long Zhou Jiajun Zhang Feifei Zhai Jingfang Xu Chengqing Zong
references.title: A compact and language-sensitive multilingual translation method
references.venue: In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,
references.authors: Zirui Wang Zachary C. Lipton Yulia Tsvetkov
references.title: On negative interference in multilingual models: Findings and a meta-learning treatment
references.venue: In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),
references.authors: Biao Zhang Ivan Titov Rico Sennrich
references.title: On sparsifying encoder outputs in sequence-to-sequence models
references.venue: arXiv preprint arXiv:2004.11854,
references.authors: Biao Zhang Philip Williams Ivan Titov Rico Sennrich
references.title: Improving massively multilingual neural machine translation and zero-shot translation
references.venue: In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,
references.authors: Changfeng Zhu Heng Yu Shanbo Cheng Weihua Luo
references.title: Language-aware interlingua for multilingual neural machine translation
references.venue: In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1650–1655,
references.authors: Barret Zoph Kevin Knight
references.title: Multi-source neural translation
references.venue: In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,
sections.heading: 1 INTRODUCTION
sections.text: Model architecture design injects inductive biases to neural network layouts, allowing a learning algorithm to favor certain representations over others, independent of the observed data (Mitchell, 1980). In multilingual neural machine translation (MNMT), where the learning objective is commonly cast as a multi-task learning problem (Firat et al., 2016a; Ha et al., 2016; Johnson et al., 2017), the inductive bias researchers usually study is deciding on which components of the neural network to share between tasks (languages), and which components to leave specific to the task or language. These components can be entire layer stacks, individual layers or even some sub-layers (Sachan & Neubig, 2018; Blackwood et al., 2018; Wang et al., 2019; Zhu et al., 2020). Noticeably, the search space of which parameters to share and at which granularity grows rapidly, as we make neural networks large or increase the number of tasks (languages). This rapid expansion of the search space prevents us from exhaustively exploring the choice of sharing patterns in MNMT.
  The incapability of full-space exploration motivates methods relying on heuristics (Sachan & Neubig, 2018), that lack flexibility when more languages are covered, or meta-learning (Platanios et al., 2018), that are often hard to scale. These limitations hinder their generalization to large-scale multilingual models, which is the very focus of our study. In large scale multilingual models, also known as massively multilingual models (Aharoni et al., 2019; Arivazhagan et al., 2019; Zhang et al., 2020b), hundreds of languages with varying amounts of training data, difficulty and linguistic properties are jointly trained together in a multi-task setup. While the joint training enables positive
  ∗Work done while Biao Zhang was interning at Google Research.
  transfer across languages, it also introduces task-interference between dissimilar languages (Arivazhagan et al., 2019; Wang et al., 2020a;b) and a capacity bottleneck emerges due to the increased number of languages and data (Huang et al., 2019; Zhang et al., 2020b).
  In this paper we adopt an end-to-end data driven approach (conditional language-specific routing, or CLSR) which permits directly probing a large section of the search space. We let the network learn the sharing structure from the data itself, by learning to route between language-specific (LS) or shared pathways. These two routes determine the mode of operation for the network: when the LS branch is selected, the model is given access to a set of LS layers (implemented as simple projections per language) and when the shared branch is chosen, the computation is routed to a layer that is used by all languages. By guiding the (gating) decision process with token level activation information, the network flexibly learns to alternate between the two modes and naturally lends itself to a conditional computation approach for multilingual processing (Bengio et al., 2013; Davis & Arel, 2013; Bapna et al., 2020). The gate states are optimized towards maximizing translation quality, but regularized with a budget constraint to control the amount of LS capacity1. Reducing the available budget results in fewer gates routing through the LS paths, enforcing CLSR to identify the most crucial sub-layers which allows us to observe and study the importance of each sub-layer for multilingual processing. Our approach is visually depicted in Figure 1.
  We verify our proposal on WMT and the massively multilingual OPUS-100 dataset, with models building on the Transformer architecture (Vaswani et al., 2017). We explore target-specific and source-specific modeling for one-to-many2 and many-to-one translation, respectively. To measure the degree of each sub-layer’s tendency to be language-specific, we propose LSScore metric. Our results show that CLSR successfully navigates the trade-offs in LS modeling, outperforming several strong baselines. Our main findings are summarized below:
  • Both the amount and the position of LS layers matter for MNMT. The best performance is achieved by distributing 10%-30% LS computation to the top and/or bottom encoder/decoder layers.
  • Feed-forward sub-layers utilize more LS capacity compared to other sub-layers on one-tomany translation.
  • One-to-many translation benefits more from CLSR (with target LS parameters) compared to many-to-one translation (with source LS parameters), particularly when the training data is imbalanced.
  • The induced sharing pattern learned by CLSR is highly similar across languages. 1We use the term “the amount of LS capacity” to refer to the proportion of open gates where CLSR selects to route information through the LS path instead of its shared counterpart, which is directly regularized and guided by the budget constraint p as in Eq. 6.
  2In a one-to-many machine translation setup, a single source side language (commonly English) is tasked to be translated into multiple target languages, one at a time.
  • We can use the learned patterns to hard-code better parameter sharing strategies for multilingual Transformers.
sections.heading: 2 RELATED WORK
sections.text: Our work closely relates to language-specific (LS) modeling for multilingual NMT and conditional computation for sequential data which we will recap both here. Early research on MNMT focused on improving shared capacity for separate bilingual models to enhance cross-lingual transfer. These efforts included sharing encoders for one-to-many translation (Dong et al., 2015), sharing decoders for many-to-one translation (Zoph & Knight, 2016; Lee et al., 2017) and sharing sub-layers (attention) for many-to-many translation (Firat et al., 2016a;b). These studies corroborated the feasibility of accommodating multiple languages with shared NMT sub-components, motivating researchers to explore universal MNMT. Ha et al. (2016) and Johnson et al. (2017) proposed such an implementation that performs multilingual translation with a single monolithic NMT model where the entire network is shared across languages, thanks to a target language token informing the model which language to translate into. Although this paradigm shows great scalability (Aharoni et al., 2019), the language token alone affords little flexibility in handling language diversity with a rigid share-all layout. Follow-up studies thus resort to LS modeling in an attempt to seek a better trade-off between sharing and not sharing. Methods in this category involve specializing neural attentions (Blackwood et al., 2018; Sachan & Neubig, 2018; Wang et al., 2019), broadening encoder outputs and normalizations (Zhang et al., 2020b), decoupling multilingual encoders and/or decoders (Vázquez et al., 2019; Escolano et al., 2020), using a fixed mix of LS and shared parameters (Wang et al., 2018), inserting lightweight adapters (Bapna & Firat, 2019) and separately modeling languages for different clusters (Tan et al., 2019), to name a few. Nevertheless, these methods heavily depend on heuristics, providing little evidence about how to optimally distribute LS capacity across the model.
  By contrast, our proposed CLSR forces the model to learn LS behaviour. It can be treated as a simplified differentiable neural architecture search (NAS) model (Liu et al., 2019) with a search space defined by the presence/absence of LS projections after every transformer sub-layer. However, in contrast with NAS, we utilize conditional computation (Bengio et al., 2013) to make the choice of executing the shared or LS path conditional on the input representations. This allows us to compare and contrast the choice of LS vs shared paths on different inputs and languages. Conditional computation has previously been successfully applied to adapt the amount of computation to the input in recurrent models (Graves, 2016) and transformers (Dehghani et al., 2019; Elbayad et al., 2020), or to significantly scale up model capacity by utilizing sparsely-gated Mixture-of-Experts layers (Shazeer et al., 2017; Lepikhin et al., 2020). Zhang et al. (2020a) applied conditional computation to sparsify encoder outputs in sequence-to-sequence models in order to reduce attention costs, while Bapna et al. (2020) introduced conditional execution of Transformer sub-layers to control the amount of computation expended by the model at inference. Sukhbaatar et al. (2019) learn parameters that limit the attention spans, in order to make the attention operation more efficient, while Fan et al. (2020) utilize structured dropout to prune transformer layers at inference. Ruder et al. (2019) propose the sluice network that learns the inter-task (shared) sub-spaces on top of task-specific models. By contrast, CLSR starts with a totally shared model, and learns how to inject task-specific projections into it, which scales more easily to massively multilingual settings. Different from previous studies, we explore conditional computation as an analysis tool to understand the ideal arrangement of LS capacity for MNMT. Utilizing conditional computation to search for better LS sharing patterns in multilingual translation, to the best of our knowledge, has never been investigated before.
sections.heading: 3 BACKGROUND: MNMT
sections.text: Given a source sentence X ′ = {x1, x2, . . . , xI} and its target translation Y = {y1, y2, . . . , yJ}, we follow Johnson et al. (2017) to reuse standard bilingual NMT models for multilingual translation by altering the source input with a language token lang, i.e. changing X ′ to X = {lang, x1, x2, . . . , xI}. Note that lang denotes the target language in one-to-many translation but source language in many-to-one translation.
  We model translation from X to Y with Transformer (Vaswani et al., 2017). Transformer relies on the following residual-normalization structure (He et al., 2015; Ba et al., 2016) to smooth informa-
  tion flow and avoid gradient vanishing and explosion: zl+1 = LN ( zl + f ( zl )) , (1)
  where l denotes layer depth and LN(·) is layer normalization (Ba et al., 2016). Function f(·) represents the basic building block in Transformer, such as attention network or feed-forward network. The encoder in Transformer is a stack of L identical layers, with each layer involving a self-attention sub-layer (SAN) and a feed-forward sub-layer (FFN). The decoder uses a similar structure except for an extra cross-attention sub-layer (CAN) inserted in-between the above two sub-layers.
sections.heading: 4 CONDITIONAL LANGUAGE-SPECIFIC ROUTING (CLSR)
sections.text: The success of MNMT comes at the cost of expressivity and model’s ability to capture languagespecific characteristics. It has been empirically shown that the language signals from language indicator tokens alone are not sufficient (Arivazhagan et al., 2019), making architectures dedicated to LS modeling a necessity (Blackwood et al., 2018; Sachan & Neubig, 2018; Zhang et al., 2020b). Nevertheless, the question when and where LS modeling matters most in MNMT still remains to be answered. To this end, we propose conditional language-specific routing (CLSR) which specializes f(·) and changes the formulation in Equation 1 as follows:
  zl+1 = LN ( zl + CLSR ( f ( zl ))) . (2)
  CLSR learns a hard binary (scalar-valued) gate g(·) for each input token based on its hidden representation zl ∈ Rd. These gates endow each sub-layer in Transformer with the capability of routing information selectively through either LS path hlang or shared path hshared:
  CLSR ( f ( zl ))
  = g(zl)hlang + (1− g(zl))hshared, (3) with hlang = f ( zl ) Wlang, hshared = f ( zl ) Wshared, (4)
  where Wshared is a weight matrix shared across languages, while parameter Wlang is only used for modeling language lang which endows NMT with source or target LS modeling capacity.3 Intuitively, a closed gate, corresponding to shared capacity, encourages maximal cross-lingual information transfer; an open gate, corresponding to LS capacity instead, improves language awareness for translation albeit it blocks knowledge transfer. CLSR balances between the two modes as controlled by the gates.
  Following Bapna et al. (2020), we parameterize the gate g(·) with a two-layer feed-forward network G(·), and inject zero-mean Gaussian noise during training to discretize it (Chiu & Raffel, 2018) :
  g(zl) = σ ( G(zl) + α(t)N (0, 1) ) , G(zl) = Relu ( zlW1 + b ) w2, (5)
  where σ(·) is the logistic-sigmoid function, dg is the gating feed-forward hidden dimension, and W1 ∈ Rd×dg ,w2 ∈ Rdg are trainable parameters. α is linearly increased along with training steps t. In this way, the gating parameters can be optimized with accurate gradients when α is small at the beginning so as to measure the degree to which each position in each sub-layer benefits from LS modeling. As training progresses, α grows larger, forcing the gating network to emit hard outputs. At inference time, we discretize the gate based on a simple decision rule: g(zl) = δ ( G(zl) ≥ 0 ) , where δ(·) is a Dirac measure. We train the gates based on the standard maximum likelihood objective, along with an additional budget regularization term that enables control over the amount of LS capacity used for translation. Let the set of all CLSR layers in the encoder beMenc and the decoder beMdec. Then the amount of LS computation utilized by a sentence pair (X,Y ) is given byG(X,Y ) = ∑ x∈X ∑ m∈Menc gm(x)+∑
  y∈Y ∑
  m∈Mdec gm(y). Given a budget constraint p ∈ [0, 1] and a batch of sentence pairs, B, the training loss function of CLSR is formulated below:
  L (B) = ∑
  (X,Y )∈B
  MLE (X,Y ) + ∣∣∣∣∣ ∑ (X,Y )∈BG(X,Y )∑ (X,Y )∈B(|X||Menc|+ |Y ||Mdec|) − p ∣∣∣∣∣ , (6)
  3We maintain a set of language-specific weight matrices in order to compute hlang for each language. To make the number of parameters manageable, we share the set of LS matrices Wlang across all encoder or decoder sub-layers, but distinguish Wlangenc and W lang dec , LS matrices for the encoder and decoder, respectively.
  Intuitively, the budget constraint tries to regulate the amount of LS computation available to all tokens in the batch as a fraction p of the total LS computation in the model. Given that we make a binary decision for every input for every CLSR layer, this corresponds to a search space of O(2|X||Menc|+|Y ||Mdec|). This gating space not only grows with respect to the model depth and sub-layer types, but also is highly input dependent. This dependency makes it difficult to search the entire space using heuristic methods (Blackwood et al., 2018).
  The constraint in Equation 6 is imposed upon the aggregated gates. As a consequence, the model can learn to trade-off LS capacity for certain layers and inputs for others. Decreasing the budget encourages gate closure, such that the LS path is chosen only in the critical sub-layers. Thus, a properly learned gating function reveals the activations of LS paths and helps gain insights into the model behavior.
sections.heading: 5 EXPERIMENTS
sections.text: Data and Evaluation We report results on two benchmarks: OPUS-100 (Zhang et al., 2020b) and WMT-14 (Barrault et al., 2019). OPUS-100 is a massively multilingual dataset collected from OPUS (Tiedemann, 2012), including 100 languages in total with 99 languages to-and-from English.4 It consists of 55M training sentence pairs with up to 1M samples per language pair, and covers 94 dev/test language pairs, each with 2000 samples at most. WMT-14 is another multilingual Englishcentric dataset composed of 13 widely-used WMT benchmarks following Siddhant et al. (2020) but excluding Kazakh and Gujarati due to their poor parallel resource. Compared to OPUS-100, WMT14 involves much fewer languages but its training data distribution is highly skewed across diverse language pairs, ranging from 0.2M (En-Tr) to 60M (En-Cs) training examples, thus posing severe challenges. We show more details about the train, dev and test data for WMT-14 in Appendix A.
  We apply byte pair encoding (BPE) algorithm (Sennrich et al., 2016) using SentencePiece (Kudo & Richardson, 2018) to preprocess multilingual sentences with a vocabulary size of 64K. We use BLEU (Papineni et al., 2002), offered by SacreBLEU (Post, 2018)5, for translation evaluation. Following Zhang et al. (2020b), we split the 94 test language pairs in OPUS-100 into three groups based on their training data size to ease model evaluation: high resource (>0.9M, 45 languages), low resource (<0.1M, 26 languages) and medium resource (others, 28 languages). Similarly, we split the 13 test language pairs in WMT-14 as follows: High (>10M, 5), Low (<1M, 5) and Med (others, 3).
  We perform experiments for one-to-many translation (O2M) and many-to-one translation (M2O). In addition to using the original training data as is, we also report results with a temperature-based strategy to balance the training data distribution by over-sampling low-resource languages with a temperature of T = 5 (Arivazhagan et al., 2019).
  We report average BLEU, and also show win ratio (Zhang et al., 2020b, WR), informing the proportion of language pairs on which our method beats our baseline. To evaluate how often each sub-layer is LS, we introduce a new metric, LSScore, formulated as follows:
  LSScoref (l, p) = 1 |P| ∑ p∈P g̃pl,f − p, (7)
  where g̃pl,f denotes the gating value averaged over all test tokens for the l-th sub-layer f(·) trained under a budget of p. Recall that we pose the budget constraint to the summed gate value instead of each individual gate, as in Equation 6. This gives CLSR the freedom to close more gates in some language-insensitive sub-layers (i.e. g̃pl,f < p) while preserve more for the others (i.e. g̃ p l,f > p). A larger LSScore, > 0 in particular, indicates that this sub-layer utilizes more LS modeling.
  Model Settings We adapt Transformer base for our experiments: L = 6, d = 512, 8 attention heads with FFN middle size of 2048. Dropout of rate 0.1 is applied to residual connections and attention weights. We optimize parameters using Adam (Kingma & Ba, 2015) (β1 = 0.9, β2 = 0.98) with label smoothing of 0.1. Learning rate is scheduled according to the inverse square root of running steps with a warmup step of 4K (Vaswani et al., 2017). We limit training sequence length
  4http://opus.nlpl.eu/opus-100.php 5Signature: BLEU+case.mixed+numrefs.1+smooth.exp+tok.13a+version.1.4.10.
  to 100, and train all models with a batch size of 1920. We set the maximum running step to 500K and 600K for OPUS-100 and WMT-14, respectively. We perform beam search decoding with beam size of 4 and length penalty of 0.6. We average the last 5 checkpoints for evaluation. For CLSR, we set dg to 128, and linearly increase α from 0 to 5 along training steps (Bapna et al., 2020).6 We vary the budget p in the range of P = {0.0, 0.1, 0.3, 0.5, 0.7, 0.9, 1.0} to study its impact on model performance.
sections.heading: 5.1 RESULTS ON OPUS-100
sections.text: On the trade-off between shared and language-specific capacity. Using more LS modeling fails to deliver increasingly better translation performance, as shown in Figure 2(a) and 2(b) when p approaches 1.0. At the point of full LS capacity for Transformer, i.e. p = 1.0, CLSR even underperforms its corresponding multilingual baseline by a large margin of 1.0-2.0 BLEU on M2O, Figure 2(b). Similarly, sharing all model parameters across language pairs, p → 0.0, also yields suboptimal performance with either original or oversampled training data. CLSR achieves its best translation quality at p = 0.3 (+2.0/3.0 BLEU) and p = 0.1 (+0.5/1.5 BLEU) for O2M and M2O, respectively. The win ratio curves in Figure 2(c) and 2(d) further confirm the robustness of these quality improvements, where properly scheduling LS capacity outperforms the baselines on >∼ 80% language pairs. These results clearly show the trade-off between these two kinds of capacity.
  When does language-specific capacity matter for multilingual translation? Results in Figure 2 show that O2M favors more LS modeling and benefits more from it compared to M2O (p = 0.3 vs. p = 0.1, and +3.0 vs. +1.5 BLEU on the original training data). We conjecture that translations on M2O share the same target language (English), so information can be easily transferred through shared modeling; by contrast, MNMT has to handle languages of diverse typological features for O2M, demanding stronger capacity delivered to each translation direction. Results in Figure 2 also show that CLSR yields more aggressive improvements on the original training data, compared to the oversampled counterpart (+3.0 vs. +2.0 BLEU on O2M and +1.5 vs. +0.5 BLEU on M2O). Finegrained analysis on each resource group, as shown in Figure 7 (Appendix B), reveals that CLSR yields more improvements for low-resource translation where the oversampling strategy partially offsets these improvements.
  Where should we add language-specific capacity in multilingual Transformer? Figure 2 suggests that CLSR uses 10-30% LS capacity to reach its best performance. We next study how CLSR schedules this capacity across all Transformer sub-layers, in order to determine the ideal arrangement of LS layers for translation. Figure 3 shows the results. Regardless of layer types, CLSR schedules more LS capacity to the top and/or bottom encoder/decoder layers rather than the middle ones. We find that more LS capacity is allocated to feed-forward sub-layers for O2M (both encoder Figure 3(a) and decoder Figure 3(b)), while the cross-attention sub-layers use more LS capacity for M2O (decoder, Figure 3(d)). Regarding the encoder for M2O, we observe no significant LSScore
  6Note that, there are 99×2 and 13×2 independent Wlang weight matrices when CLSR is used for OPUS100 and WMT-14, respectively. This corresponds to each language having a weight matrix in the encoder and the decoder.
  difference among different sub-layers as in Figure 3(c). Overall, CLSR tends to make the “M” side more LS, i.e. the encoder side of M2O (Figure 3(c)) and the decoder side of O2M (Figure 3(b)).
  Does CLSR schedule language-specific capacity based on linguistic similarity? It is intriguing to explore how the capacity scheduled by CLSR is actualized, especially whether CLSR learns to organize LS capacity according to linguistic characteristics or not. The heatmap in Figure 6, Appendix B shows that the LSScore distribution over different sub-layers (y-axis) has only subtle difference across different language pairs (x-axis). This suggests that the capacity schedule has little to do with linguistic characteristics. More results in other settings are given in Figure 6, Appendix B, which reflect similar observation. In short, CLSR allocates LS capacity to specific sub-layers rather than specific languages. This could be ascribed to the design of CLSR. CLSR shares the gating parameters and the budget, p, across all languages, which might impose some inductive bias discouraging its LS behavior. Besides, the structure of the gating in CLSR (Eq. 5) might fail to offer enough flexibility for controlling the gates in different ways across languages and layers. We leave further study of LS gating to future work.
  On detailed results and comparison to other baselines. Table 1 summarizes our results.7 Although LS (Zhang et al., 2020b) improves O2M, it fails to surpass the vanilla Baseline on M2O, indicating that the position of its LS layer, i.e. on top of the encoder outputs, is sub-optimal for
  7We list the number of trainable parameters for each model in Table 6, Appendix D.
  many-to-one translation. Compared to LS , CLSR-S uses no LS modeling while CLSR-L injects LS projection into each sub-layer, both of which delivers inferior performance on O2M (against LS and Baseline) and M2O (against Baseline), echoing our findings from Figure 2. By contrast, CLSR?, which uses an optimized budget p, yields promising results, beating Baseline on O2M and M2O, underlining the importance of seeking balances between sharing and not sharing.
  Can we use these insights to improve multilingual Transformer? We answer this question by transferring our findings from Figure 3 into the following two Transformer variants: one enhances the top and bottom encoder/decoder layers with LS projections (Top-Bottom), while the other makes high-LSScore sub-layers LS (Dedicated).8 Results of these experiments, in Table 1, demonstrate that incorporating our findings into Transformer helps to improve translation performance. The dedicated model in particular recovers and even surpasses the performance of CLSR?.
sections.heading: 5.2 RESULTS ON WMT-14
sections.text: Figure 4 shows the capacity trade-off on WMT-14, reconfirming the ability of CLSR. One noticeable difference is that the relative improvements become smaller. We ascribe this to the smaller number of language pairs in WMT-14, where the effect of introducing LS capacity into the model is smaller compared to the massively multilingual setting (Arivazhagan et al., 2019). Table 2 shows similar results as Table 1, where CLSR? outperforms both fully-shared (CLSR-S) and fully-LS (CLSRL) baselines, and both Top-Bottom and Dedicated improve multilingual translation compared to Baseline (except for M2O with oversampling). More results are given in Appendix C. Our results demonstrate that the CLSR approach generalizes to different datasets, with varying number of lan-
  8Detail about which sub-layers are specialized in Dedicated is given in Table 4, Appendix B.
  guages and resource sizes. We provide additional experiments ablating different elements on CLSR in Table 7, Appendix E.
sections.heading: 6 CONCLUSION AND FUTURE WORK
sections.text: Share or not? This is an open question when developing MNMT models. In this paper, we attempt to answer this question by proposing conditional language-specific routing (CLSR). Our empirical results demonstrate that CLSR learns to balance between shared and LS paths across all NMT sublayers, improving the quality of multilingual translation. Our analysis on OPUS-100 and WMT-14 suggest that both the position and the amount of LS capacity greatly affects MNMT. Scheduling 10%-30% LS layers to the top and/or bottom encoder/decoder layers reaches CLSR’s peak performance. We also demonstrate how our findings can be leveraged to design a multilingual Transformer with an optimal sharing pattern. We believe that our work improves our understanding on the tradeoff between sharing and not sharing, paving the way for better multilingual models.
  In the future, we plan to extend our study to many-to-many translation as well as larger-capacity models. We also plan to adapt CLSR to other multilingual multi-task learning settings to better handle knowledge transfer among different tasks, especially for cross-lingual downstream transfer.
sections.heading: ACKNOWLEDGEMENTS
sections.text: We would like to thank Yuan Cao for his valuable feedback. We would also like to thank the Google Translate team for their constructive discussions and comments. We thank the reviewers for their insightful comments. Rico Sennrich has received funding from the Swiss National Science Foundation (project no. 176727).
sections.heading: A DATASET DETAILS
sections.text: Table 3 lists the statistics for WMT-14. We collect these datasets following Siddhant et al. (2020). Figure 5 shows the training data distribution. Training data in WMT-14 is more imbalanced across different language pairs compared to OPUS-100.
sections.heading: B MORE RESULTS ON OPUS-100
sections.text: Figure 7 shows the translation performance of CLSR on each resource group when varying the budget p. We find that LS modeling greatly improves translation on low-resource settings, although
  this improvement is partially offset by the oversampling strategy. We observe that the success of oversampling is built on top of sacrificing quality on high-resource languages, where CLSR largely narrows or even closes the performance gap against the multilingual baseline trained on the original
  data. In addition, we find no significant difference in term of the trade-off curve across resource groups compared to Figure 2.
  Figure 6 shows the heatmaps for LSScore distribution. Regardless of translation directions (O2M or M2O) and data settings (Original or Oversample), there is no clear linguistic pattern behind these heatmaps. The learning of CLSR is not linguistic driven.
sections.heading: C MORE RESULTS ON WMT-14
sections.text: We show more results on WMT-14 here. Both heatmaps in Figure 9 and fine-grained trade-off curves in Figure 8 show a very similar story to the one on OPUS-100. In terms of LSScore distribution in Figure 10(d), we also have similar observation except that the cross-attention sub-layer in the decoder on M2O shows no clear LS preference.
  These results suggest the high generalization of CLSR to different data settings.
sections.heading: D MODEL PARAMETERS
sections.text: We list model parameters in Table 6. Note that the source/target vocabulary for O2M is the same as the target/source vocabulary for M2O, thus models for O2M and M2O have the same number of trainable parameters except for Dedicated where the network structure differs.
sections.heading: E ABLATION STUDY
sections.text: We provide two variants of CLSR for ablation. The first one examines whether CLSR could induce an optimal budget automatically by eschewing the budget constraint in Eq. 6 (CLSR w/o p); the second one further replaces the hard gating network with a vanilla soft gating network (CLSR-Gate).
  Table 7 summarizes the translation results. Without the regularization term, CLSR w/o p yields an average gating value of 0.37/0.16 and 0.39/0.19 for O2M/M2O on OPUS-100 and WMT-14, respectively9, resonating with our finding that O2M requires more LS capacity compared to M2O. Compared to CLSR?, CLSR w/o p produces more open gates, achieving comparable but overall worse performance in all data settings. This highlights the importance of the added regularization term, where without such regularization the model is likely spending extra learning cycles to automatically optimize for the budget as with CLSR w/o p. This in return hints at the importance of optimization routines (learning) which we held constant in this study.
  By contrast, switching to soft gating network, CLSR-Gate gains translation quality in most settings, although the quality gains are mostly less than 0.1 compared to CLSR?. Note that, CLSR-Gate relies on continuous gates, indicating a much higher flexibility in balancing parameter sharing patterns. However, we also notice that CLSR-Gate performs worse on M2O, even fails to beat the vanilla Baseline (All) on the oversampled WMT-14. Unlike CLSR-Gate, CLSR? uses hard binary gates which eases capacity analysis and achieves consistent improvements across data settings.
  9We also average the gating value over the original and oversampled data settings.