Fine-tuning the entire set of parameters of a large pretrained model has become the mainstream approach for transfer learning. To increase its efficiency and prevent catastrophic forgetting and interference, techniques like adapters and sparse fine-tuning have been developed. Adapters are modular , as they can be combined to adapt a model towards different facets of knowledge (e.g., dedicated language and/or task adapters). Sparse fine-tuning is expressive , as it controls the behavior of all model components. In this work, we introduce a new fine-tuning method with both these desirable properties. In particular, we learn sparse, real-valued masks based on a simple variant of the Lottery Ticket Hypothesis. Task-specific masks are obtained from annotated data in a source language, and language-specific masks from masked language modeling in a target language. Both these masks can then be composed with the pretrained model. Unlike adapter-based fine-tuning, this method neither increases the number of parameters at inference time nor alters the original model architecture. Most importantly, it outperforms adapters in zero-shot cross-lingual transfer by a large margin in a series of multilingual benchmarks, including Universal Dependencies, MasakhaNER, and AmericasNLI. Based on an in-depth analysis, we additionally find that sparsity is crucial to prevent both 1) interference between the fine-tunings to be composed and 2) overfitting. We release the code and models at https://github.com/ cambridgeltl/composable-sft . Fine-tuning of pretrained models (Howard and Ruder, 2018; Devlin et al., 2019, inter alia ) is arguably the dominant paradigm in NLP at present. Originally, fine-tuning involved supervised learning of all the parameters of a model pretrained on unlabeled texts. However, given the size of Transformer-based architectures, this approach is often timeand resourceinefficient, and may result in catastrophic forgetting and interference (Wang et al., 2020) during multiple adaptations. To overcome these limitations, two main alternatives have emerged: 1) through adapters , new parameters can be added to a pretrained model in the form of extra intermediate layers (Rebuffi et al., 2017; Houlsby et al., 2019) and fine-tuned while keeping all the pretrained parameters fixed; 2) sparse fine-tuning (SFT) of a small subset of pretrained model parameters (Guo et al., 2021; Zaken et al., 2021; Xu et al., 2021b, inter alia ). Adapters have proven especially useful in multilingual NLP (Bapna and Firat, 2019; stn et al., 2020; Pfeiffer et al., 2020b; Vidoni et al., 2020; Pfeiffer et al., 2021b; Ansell et al., 2021) because they exhibit a surprising degree of modularity . This ability to disentangle and recombine orthogonal facets of knowledge in original ways (Ponti et al., 2021; Ponti, 2021) allows for separately learning a task adapter from labeled data in a source language and dedicated language adapters from unlabeled data in the source language and target languages. By stacking these components, it is possible to perform zero-shot cross-lingual transfer. Compared to sequentially fine-tuning the full model on both the task and target language, this yields superior performance and efficiency (Pfeiffer et al., 2020b). Notably, achieving coverage over NT tasks in NL target languages with the sequential approach requires NTNL models to be trained, whereas the modularity of adapters reduces this to NT + NL . Meanwhile, the advantage of SFTs over adapters is their expressivity : rather than a non-linear transformation of the output of Transformer layers (e.g., using a shallow MLP as with adapters), they can operate directly on a pretrained model's embedding and attention layers. It therefore seems natural to search for a parameter-efficient fine-tuning method that is both modular and expressive. To this end, we propose Lottery Ticket Sparse Fine-Tuning (LT-SFT), a simple and general-purpose adaptation technique inspired by the Lottery Ticket Hypothesis (LTH; Frankle and Carbin, 2019; Malach et al., 2020), which was originally conceived for pruning large neural networks. In particular, after fine-tuning a pretrained model for a specific task or language, we select the subset of parameters that change the most. Then, we rewind the model to its pretrained initialization (without setting any value to zero, contrary to the original LTH algorithm). By re-tuning again only the selected subset of parameters, we obtain a sparse fine-tuning in the form of a vector of differences with respect to the pretrained model. Multiple SFTs can be composed by simply summing them with the pretrained model. We provide a graphical representation of our method in Figure 1. We benchmark LT-SFT on a series of multilingual datasets, including Universal Dependencies (Zeman et al., 2020) for part-of-speech tagging and dependency parsing, MasakhaNER (Adelani et al., 2021) for named entity recognition, and AmericasNLI (Ebrahimi et al., 2021) for natural language inference. We evaluate it in a zero-shot cross-lingual transfer setting on 35 typologically and geographically diverse languages that include both languages seen and unseen during masked language modeling of the pretrained model. The results in all transfer tasks indicate that LT-SFT consistently achieves substantial gains over the current state-of-the-art adapter-based method for cross-lingual transfer, MAD-X (Pfeiffer et al., 2020b). In addition to its superior performance, modularity, and expressivity, LT-SFT offers a series of additional advantages over adapters: 1) the number of parameters remains constant, which prevents the decrease in inference speed observed when adapter layers are added; 2) the neural architecture remains identical to the pretrained model, which makes code development model-independent rather than requiring special modifications for each possible architecture (Pfeiffer et al., 2020a). Finally, 3) we empirically demonstrate that the peak in performance for LT-SFT is consistently found with the same percentage of tunable parameters, whereas the best reduction factor for MAD-X is task-dependent. This makes our method more robust to the choice of hyper-parameters. In addition, we find that a high level of sparsity in language and task fine-tunings is beneficial to performance, as this makes overlaps less likely and poses a lower risk of creating interference between the knowledge they contain. Moreover, it makes fine-tunings less prone to overfitting due to their constrained capacity. Thus, sparsity is a fundamental ingredient for achieving modularity and composability. These properties in turn allow for systematic generalization to new combinations of tasks and languages in a zero-shot fashion. To establish a broader context for our research, we first provide a succinct overview of current methods for efficient fine-tuning, such as adapters and SFT. We then recapitulate the Lottery Ticket Hypothesis, 1779 upon which our newly proposed method is built. Adapters and Composition. An adapter is a component inserted into a Transformer model with the purpose of specializing it for a particular language, task, domain, or modality (Houlsby et al., 2019). Previous work in multilingual NLP has mainly adopted the lightweight yet effective adapter variant of Pfeiffer et al. (2021a). In this setup, only one adapter module, consisting of a successive down-projection and up-projection, is injected per Transformer layer, after the feed-forward sub-layer. The adapter A b at the b -th Transformer layer performs the following operation: A b ( h b , r b ) = U b a ( D b h b ) + r b . h b and r b are the Transformer hidden state and the residual at layer b , respectively. D b  R m  h and U b  R h  m are the downand up-projections, respectively ( h being the Transformer's hidden layer size, and m the adapter's dimension), and a (  ) is a non-linear activation function. The residual connection r b is the output of the Transformer's feed-forward layer whereas h b is the output of the subsequent layer normalization. During fine-tuning of a pretrained model with adapters, only the adapter parameters U and D are modified while the pretrained model's parameters are kept fixed. In the MAD-X adapter composition framework for cross-lingual transfer (Pfeiffer et al., 2020b), a language adapter (LA) for a massively multilingual Transformer (MMT) is learned for each source and target language through masked language modeling (MLM), and a task adapter (TA) is learned for each target task, where the LA for the source language is inserted during TA training. At inference time, the task adapter and target language adapter are composed by stacking one on top of the other. This adapter composition approach has been shown to be highly effective for cross-lingual transfer (Pfeiffer et al., 2020b, 2021b; Ansell et al., 2021), especially for low-resource languages and target languages unseen during MMT pretraining. Sparse Fine-Tuning. We call F (cid:48) = F (  ;  +  ) a sparse fine-tuning (SFT) of a pretrained neural model F (  ;  ) if  is sparse. We sometimes refer to  itself as an SFT, or as the SFT's difference vector . Previously proposed SFT methods include DiffPruning (Guo et al., 2021), BitFit (Zaken et al., 2021) and ChildTuning (Xu et al., 2021b). DiffPruning simulates sparsity of the difference vector during training by applying a continuous relaxation of a binary mask to it. BitFit on the other hand allows non-zero differences only for bias parameters. ChildTuning selects a subset of fine-tunable parameters by using Fisher information to measure the relevance of each parameter to the task. These methods have been shown to be competitive with full fine-tuning on GLUE (Wang et al., 2019), despite the difference vector  having fewer than 0.5% non-zero values. Lottery Ticket Hypothesis. (LTH; Frankle and Carbin, 2019; Malach et al., 2020) states that each neural model contains a sub-network (a winning ticket) that, if trained again in isolation, can match or even exceed the performance of the original model. To achieve this, after a pruning stage where some parameters are zero-masked and frozen according to some criterion (e.g., weight magnitude), the remaining parameters are restored to their original values and then re-tuned. This process of pruning and re-training can be iterated multiple times. The LTH has so far been used mostly for model compression through network pruning; to our knowledge, we are the first to use it for pretrained model adaptation . Multi-Source Task Training. Ansell et al. (2021) showed that training task adapters using data from multiple source languages can result in sizable improvements in downstream zero-shot transfer performance even when the total number of training examples is held constant. In their training setup, each batch consisted of examples from a single, randomly selected source language, the language adapter for which is activated for the duration of the training step. Training. In this work, we propose Lottery Ticket Sparse Fine-Tuning (LT-SFT). Similar to the Lottery Ticket algorithm of Frankle and Carbin (2019), our LT-SFT method consists of two phases: (Phase 1) Pretrained model parameters  (0) are fully fine-tuned on the target language or task data D , yielding  (1) . Parameters are ranked according to some criterion, in our case greatest absolute difference |  (1) i   (0) i | , and the top K are selected for tuning in the next phase: a binary mask  is set to have 1 in positions corresponding to these parameters, and 0 elsewhere. (Phase 2) After resetting the parameters to their 1780 original values  (0) , the model is again fine-tuned, but this time only the K selected parameters are trainable whereas the others are kept frozen. In practice, we implement this by passing the masked gradient  (cid:12)   L ( F (  ;  ) , D ) (where (cid:12) denotes element-wise multiplication and L a loss function) to the optimizer at each step. From the resulting fine-tuned parameters  (2) we can obtain the sparse vector of differences  =  (2)   (0) . In addition, we experiment with applying a regularization term which discourages parameters from deviating from their pretrained values  (0) . Specifically, we use L1 regularization of the form J (  ) = N (cid:80) i |  i   (0) i | . Composition. Although we often use the term sparse fine-tuning to refer to the difference vector  itself, an SFT is most accurately conceptualized as a functional which takes as its argument a parameterized function and returns a new function, where some sparse difference vector  has been added to the original parameter vector. Suppose we have a language SFTSL and a task SFTST defined by SL ( F (  ;  )) = F (  ;  +  L ) ST ( F (  ;  )) = F (  ;  +  T ) . We adopt a similar cross-lingual transfer setup to MAD-X (Pfeiffer et al., 2020b, see also 2). We start with an MMTF with pretrained parameters  learned through masked language modeling on many languages, such as mBERT (Devlin et al., 2019) or XLM-R (Conneau et al., 2020). For each language of interest l , we learn a language SFT  ( l ) L through LT-SFT (also with an MLM objective) on text from language l . For each task of interest t , we learn a task SFT  ( t ) T through LT-SFT on annotated data from some source language s . When learning the task SFT, we first adapt to the source language by applying the language SFT for s . 1 The language SFT is removed again after training. That is, we perform 1 Adapting to the source language yields substantial improvements in cross-lingual transfer performance with both MAD-X and LT-SFT, with gains of 2-3 points in our preliminary experiments. Paradoxically, our results (see Table 7) and results from previous work (Pfeiffer et al., 2020b; Ansell et al., 2021) suggest that adapting to high-resource target languages at inference time does not give similarly large benefits. We think this phenomenon warrants further investigation. LT-SFT on F (  ;  +  ( s ) L ) to obtain fine-tuned parameter vector  (cid:48) . We then calculate  ( t ) T =  (cid:48)  (  +  ( s ) L ) . Note that during task training, we also learn a classifier head, which is fully fine-tuned during both phases of LT-SFT adaptation, with the same random initialization applied at the beginning of each phase. We perform zero-shot adaptation of F to target language l for task t by composing language and task SFTs to obtain F t,l = F (  ;  +  ( t ) T +  ( l ) L ) . On top of this, we stack the classifier head learned for t . For a formal algorithm of LT-SFT and the transfer procedure, we refer to Appendix A. 4 Experimental Setup To evaluate our new method extensively, we benchmark its zero-shot cross-lingual performance on four distinct tasks: part-of-speech tagging (POS), dependency parsing (DP), named entity recognition (NER), and natural language inference (NLI). Table 1 summarizes our experimental setup, including the datasets and languages considered in our experiments. We put emphasis on low-resource languages and languages unseen during MMT pretraining, although we also evaluate on a few high-resource languages. In total, we cover a set of 35 typologically and geographically diverse languages, which makes them representative of cross-lingual variation (Ponti et al., 2019, 2020). The main baseline is MAD-X, the state-of-the-art adapter-based framework for cross-lingual transfer (Pfeiffer et al., 2020b). We use the MAD-X 2.0 variant, where the last adapter layers are dropped. Pfeiffer et al. (2021b) found that this improved performance, which we could confirm in our preliminary experiments. Since adapters with the configuration used by Pfeiffer et al. (2020b) are unavailable for many languages in our evaluation, we train our own for all languages. In Appendix D we also provide an evaluation with comparable language adapters from AdapterHub (Pfeiffer et al., 2020a) where available. We also perform experiments with BITFIT (Za-ken et al., 2021) to establish a baseline for an existing SFT technique. In addition to the main LT-SFT model variant, on POS and DP we test a RANDSFT variant as an ablation, where the K parameters to be fine-tuned are selected at random rather than based on an informed criterion. MLM Training Data. For all languages in our POS and DP evaluation, we perform MLM language SFT/adapter training on Wikipedia corpora. We also use Wikipedia for all languages in our NER evaluation if available. Where this is not the case, we use the Luo News Dataset (Adelani et al., 2021) for Luo and the JW300 corpus (Agic and Vulic, 2019) for Nigerian Pidgin. The main corpora for the languages in our NLI evaluation are those used by the dataset creators to train their baseline models (Ebrahimi et al., 2021); however, since the sizes of these corpora are restricted due to containing only parallel data, we augment them with data from Wikipedia and the corpora of indigenous Peruvian languages of Bustamante et al. (2020) where available. More details on data sources are provided in Appendix B. Training Setup and Hyper-parameters. For both SFTs and adapters, we train for the lesser of 100 epochs or 100,000 steps of batch size 8 and maximum sequence length 256, subject to an absolute minimum of 30,000 steps since 100 epochs seemed insufficient for some languages with very small corpora. Model checkpoints are evaluated every 1,000 steps (5,000 for high-resource languages) on a held-out set of 5% of the corpus (1% for high-resource languages), and the one with the smallest loss is selected at the end of training. We use the AdamW optimizer (Loshchilov and Hutter, 2019) with an initial learning rate of 5 e -5 which is linearly reduced to 0 over the course of training. Following Pfeiffer et al. (2020b), the reduction factor (i.e., the ratio between model hidden size and adapter size) for the adapter baseline was set to 2 for a total of  7.6M trainable parameters. For comparability, we set the same number of trainable parameters K for our language LT-SFTs. This results in language SFTs with a sparsity of 4.3% for mBERT and 2.8% for XLM-R. Since BITFIT tunes exclusively the bias parameters, its language SFTs have a fixed sparsity of 0.047% for mBERT and 0.030% for XLM-R. Importantly, during language sparse fine-tuning, we decouple the input and output embedding matrices and fix the parameters of the output matrix; otherwise, we find that the vast majority of the K most changed parameters during full fine-tuning belong to the embedding matrix, seemingly due to its proximity to the model output, which damages downstream performance. We also fix the layer normalization parameters; all other parameters are trainable. For language adaptation, we apply L1 regularization as described in 3.1 with  = 0 . 1 . Note that the specified training regime is applied in the same way during both phases of LT-SFT. For language adapter training in the MAD-X baseline, we use the Pfeiffer configuration (Pfeiffer et al., 2021a) with invertible adapters, special additional sub-components designed for adapting to the vocabulary of the target language, which yields consistent gains. For POS tagging, DP, and NER, 2 we train task SFTs/adapters on the datasets indicated in Table 1 for 10 epochs with batch size 8, except during the first phase of LT-SFT training where we train for only 3 epochs. 3 Model checkpoints are evaluated on the validation set every 250 steps, and the best checkpoint is taken at the end of training, with the selection metric being accuracy for POS, labeled attachment score for DP, and F1-score for NER. Similarly to language fine-tuning, we use an initial learning rate of 5 e -5 which is linearly reduced to 0 over the course of training. For POS and NER we use the standard token-level single-layer multi-class model head. For DP, we use the shallow variant (Glava and Vulic, 2021) of the biaffine dependency parser of Dozat and Manning (2017). For NLI, we employ the same fine-tuning hyper-parameters as Ebrahimi et al. (2021): 5 epochs with batch size 32, with checkpoint evaluation on the validation set every 625 steps, and an initial learning rate of 2 e -5. We apply a two-layer multi-class clas-sification head atop the MMT output corresponding to the [CLS] token. We found that the number of trainable parameters during task adaptation (governed by K for SFTs and reduction factor for adapters) has a large effect on performance: we thus experiment with a range of values. Specifically, we test adapter reduction factors of 32, 16, 8, 4, 2, and 1, and equivalent values of K 4 for SFT. To validate that task LT-SFT training, like task adapter training in prior work (Ansell et al., 2021), benefits from the presence of multiple source languages in the training data, and to push the boundaries of zero-shot cross lingual transfer, we perform multi-source training experiments on DP and NLI. 2 MasakhaNER and CoNLL 2003 datasets respectively use the DATE and MISC tags which are not used by the other; we replace these with the O tag at both train and test time. 3 This is because full fine-tuning is more prone to overfitting than sparse/adapter fine-tuning. Early stopping somewhat addresses overfitting, but it is insufficient in a cross-lingual setting because the target language performance generally starts to deteriorate faster than the source language performance. 4 Approximately 442K, 884K, 1.7M, 3.5M, 7.1M, and 14.2M respectively, amounting to sparsity levels of 0.25%, 0.50%, 1.0%, 2.0%, 4.0% and 8.0% for mBERT and 0.16%, 0.32%, 0.63%, 1.3%, 2.6% and 5.1% for XLM-R. We adopt a similar setup to Ansell et al. (2021): we obtain the training set by concatenating the training data for all source languages. We randomly shuffle the training set and train as in the single-source case, except that each batch is composed of examples from a single source language, whose language SFT is applied during the training step. We prioritize maximizing performance rather than providing a fair comparison against the single-source case, so unlike Ansell et al. (2021), we use the entirety of the training sets. In derogation of this principle, we set a maximum of 15K examples per language for DP to better balance our sample. For DP, we train our models on the UD treebanks of 11 diverse high-resource languages. For NLI, we train on MultiNLI (Williams et al., 2018) plus the data for all 14 non-English languages in the XNLI dataset (Conneau et al., 2018). We also evaluate multi-source task SFT training on extractive question answering (QA), as a comparatively generous amount of multilingual data is available for this task. Specifically, we train on English data from SQuAD version 1 (Rajpurkar et al., 2016), all languages from MLQA (Lewis et al., 2020), and those languages from XQuAD (Artetxe et al., 2020) which also appear in MLQA. We evaluate on the languages present in XQuAD but not in MLQA. For QA, we train for 5 epochs with batch size 12 and initial learning rate 3 e -5. Full details of the source languages can be found in Appendix B. We use an equivalent reduction factor of 1 for all tasks, following the strongest setting from our single-source experiments. Except as stated above, the training configuration and hyper-parameters are the same as for single-source training. We report the average test performance of zero-shot cross-lingual transfer for the best reduction factor (or equivalent K ) in Table 2. Some patterns emerge across all four tasks: first, LT-SFT consistently outperforms all the baselines. In particular, it surpasses the state-of-the-art MAD-X across all tasks, with gains of 2.5 accuracy in part-of-speech tagging, 2.5 UAS and 3.7 LAS in dependency parsing, 1.8 F1 score in named entity recognition, and 1.9 accuracy in natural language inference. Compared to RAND-SFT, its superior performance demonstrates the importance of selecting winning tickets rather than a random subset 1783 POS DP NER NLI Accuracy UAS LAS F1 score Accuracy LT-SFT 71.1 (1) 57.1 (1) 37.8 (1) 71.7 (1) 51.4 (1) RAND-SFT 69.2 (1) 54.3 (1) 33.9 (1) -MAD-X 68.6 (16) 54.6 (2) 34.1 (1) 69.9 (8) 49.5 (2) BITFIT 58.1 45.7 23.9 54.9 38.3 LT-SFT TA-ONLY 51.3 (32) 39.1 (1) 19.9 (1) 55.3 (8) 39.9 (4) MAD-X TA-ONLY 52.1 (32) 38.9 (1) 19.5 (1) 52.4 (32) 41.7 (4) Table 2: Results of zero-shot cross-lingual transfer evaluation averaged over all languages when best equivalent reduction factor (shown in parentheses after each result) is chosen. of parameters. Secondly, the results demonstrate the importance of language SFTs/adapters for specializing pretrained models to unseen languages, as they bring about a large increase in performance across the 4 tasks compared to the corresponding settings with task adaptation only (TA-ONLY ). We remark that LT-SFT's zero-shot performance also exceeds translation-based baselines on the AmericasNLI task, achieving an average accuracy of 51.4%, compared with the 48.7% of the translate-train' baseline of Ebrahimi et al. (2021). In Figure 2, we provide a more detailed overview of average cross-lingual model performance across a range of different reduction factors. The results for the LT-SFT and RAND-SFT methods generally improve or stay steady as the number of trainable task parameters increases. On the contrary, there is not such a trend for MAD-X, as lower reduction factors may degrade its results. This makes it easier to choose a good setting for this hyper-parameter when using SFT. Moreover, it is worth stressing again that, contrary to MAD-X, this hyper-parameter does not affect inference time. BITFIT performs much worse than the other methods which perform language adaptation across all tasks. Bearing in mind the strong trend towards increasing performance with increasing K for the other SFT methods, it seems likely that BITFIT , with two orders of magnitude fewer trainable parameters, lacks the capacity to learn effective task 1784 el ro ru th tr XLM-R Base, full FT 71.1/54.3 78.3/63.7 74.1/57.8 67.1/55.7 67.5/51.1 XLM-R Large, full FT (Artetxe et al., 2020) 79.8/61.7 83.6/69.7 80.1/64.3 74.2/62.8 75.9 / 59.3 XLM-R Base MS, LT-SFT 81.9 / 65.5 86.3 / 73.3 81.4 / 64.6 82.4 / 75.2 75.2/58.6 Table 3: Results of zero-shot cross-lingual transfer evaluation on XQuAD (Artetxe et al., 2020), restricted to languages which do not appear in MLQA (Lewis et al., 2020) (see 4.4) in the format F1/exact match score. For additional results at the level of individual languages and an analysis of the efficacy of language adaptation for highversus lowresource target languages, we refer the reader to Appendix C. 5.1 Multi-Source Training As shown in Table 4, multi-source LT-SFT training brings about a large improvement in zero-shot cross-lingual transfer performance on DP, and a modest improvement for NLI. This may be a result of the fact that the training set for NLI contains a relatively small number of non-English examples compared to the DP training set. Also, the AmericasNLI target languages generally have a lower degree of genealogical relatedness to the source languages compared to the DP target languages. Table 3 demonstrates that multi-source training is also beneficial to zero-shot cross-lingual transfer for QA on a series of relatively high-resource languages. In particular, LT-SFT multi-source training of XLM-R Base outperforms single-source full fine-tuning of XLM-R Large (a larger model) comfortably, and outperforms XLM-R Base single-source full fine-tuning by a significant margin. The fact that such an improvement occurs despite each of the 6 non-English source languages having more than an order of magnitude less training data than the English data from SQuAD illustrates the disproportionate advantage of multilingual source data. Finally, we address the following question: is sparsity responsible for preventing the interference of separate fine-tunings when they are composed? To support this hypothesis with empirical evidence, we use LT-SFT to train language 5 and task fine-tunings with different levels of density, i.e. the percentage of non-zero values (from 5% to 100%). We then evaluate all possible combinations of density levels. The results are visualized in the form of a contour plot in Figure 3 for selected combinations of tasks and languages: Buryat, Cantonese, Erzya, Maltese, and Upper Sorbian for DP, and Hausa, Igbo, Luganda, Swahili and Wolof for NER. From Figure 3, it emerges that the performance decreases markedly for SFTs with a density level greater than ~30% of fine-tuned parameters. 6 We speculate that this is due to the fact that sparser fine-tunings have a lower risk of overlapping with each other, thus creating interference between the different facets of knowledge they encapsulate. It must be noted, however, that alternative hypotheses could explain the performance degradation in addition to parameter overlap, such as overfitting as a result of excessive capacity. While we leave the search for conclusive evidence to future work, both of these hypotheses illustrate why enforcing sparsity in adaptation, as we propose in our method, is crucial to achieving modularity. Within the framework of the Lottery Ticket Hypothesis, a series of improvements have been suggested to make the original algorithm to find winning tickets (Frankle and Carbin, 2019) more stable: after fine-tuning, Frankle et al. (2019) rewind the parameters to their values after a few iterations rather than their values before training, whereas Renda et al. (2020) also rewind the learning rate. In addition, Zhou et al. (2019) found that 1) different criteria can be used to select weights as an alternative to the magnitude of their change; 2) different rewinding methods are also effective, such as restoring the original sign, but not the value. In future work, we will investigate whether these variants also benefit our method for cross-lingual transfer, where the LTH is used for adaptation rather than pruning. Whereas the LTH was originally conceived in the vision domain for convolutional architectures, it is also effective for pruning models trained on NLP tasks (Yu et al., 2020), such as neural machine translation, and based on Transformer architectures (Prasanna et al., 2020). Recently, Xu et al. (2021a) adapted the LTH specifically to prune pretrained models after fine-tuning. To the best of our knowledge, Wortsman et al. (2020) is the only instance where winning tickets were composed in previous work. In their experiment, a set of task-specific masks were linearly combined at inference time, in order to generalize to new tasks in a continuous learning setting. 6 Note, furthermore, that levels of task fine-tuning density greater than ~60% do not vary in performance. This is because their subsets of parameters include embeddings of tokens never encountered during task training, which are therefore never updated even if trainable. We have presented a new method to fine-tune pretrained models that is both modular (like adapters) and expressive (like sparse fine-tuning). This method is based on a variant of the algorithm to find winning tickets under the framework of the Lottery Ticket Hypothesis. We infer a sparse vector of differences with respect to the original model for each individual language (by modeling unlabeled text) and each individual task (with supervised learning). The adaptations for a language and a task can then be composed with the pretrained model to enable zero-shot cross-lingual transfer. Comparing our method with the state-of-the-art baseline in several multilingual tasks, the results have indicated substantial gains across the board in both languages seen and unseen during pretraining (which includes many truly low-resource languages). In future work, our method offers several potential extensions. In addition to the variants to the Lottery Ticket algorithm surveyed in 6, given the importance of sparsity for modularity (5.2), we plan to experiment with additional algorithms previously applied to pruning that can identify and fine-tune a subset of the model parameters, such as DiffPruning (Guo et al., 2021) and ChildTuning (Xu et al., 2021b). Finally, given its simplicity and generality, our method is suited for many other applications of transfer learning in addition to cross-lingual transfer, such as multimodal learning, debiasing, and domain adaptation. The code and models are available online at https: //github.com/cambridgeltl/composable-sft . (cid:70)(cid:70)(cid:70)(cid:70)(cid:70)(cid:70)(cid:70)(cid:70)(cid:70)(cid:70)(cid:70)(cid:70) Alan wishes to thank David and Claudia Harding for their generous support via the Harding Distinguished Postgraduate Scholarship Programme. Anna and Ivan are supported by the ERC PoC Grant MultiConvAI (no. 957356) and a Huawei research donation. We would like to thank Chiara Ponti for the graphic illustration. We also thank the anonymous reviewers for their helpful suggestions.
abstain abstain abstain abstain objective abstain abstain abstain abstain abstain result other abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain method abstain result abstain other objective abstain result abstain abstain abstain objective objective other other other other other other other other other other other other other abstain abstain other other other other other other other other other objective other other other abstain abstain abstain abstain abstain abstain method abstain method method abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain method abstain method method method abstain abstain method method method method abstain method abstain method abstain method abstain abstain method method method abstain abstain method method abstain method method method method method method method method method abstain abstain abstain abstain method method abstain method method method method method method abstain abstain abstain abstain abstain other abstain abstain result abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other objective other other abstain other other other objective abstain abstain abstain result abstain method abstain other other other other other