Structured pruning has been extensively studied on monolingual pre-trained language models and is yet to be fully evaluated on their multilingual counterparts. This work investigates three aspects of structured pruning on multilingual pre-trained language models: settings, algorithms, and efficiency. Experiments on nine downstream tasks show several counterintuitive phenomena: for settings, individually pruning for each language does not induce a better result; for algorithms, the simplest method performs the best; for efficiency, a fast model does not imply that it is also small. To facilitate the comparison on all sparsity levels, we present Dynamic Sparsification , a simple approach that allows training the model once and adapting to different model sizes at inference. We hope this work fills the gap in the study of structured pruning on multilingual pre-trained models and sheds light on future research. Large-scale pre-trained monolingual language models like BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) have shown promising results in various NLP tasks while suffering from their large model size and high latency. Structured pruning has proven to be an effective approach to compressing and accelerating these large monolingual language models (Michel et al., 2019; Wang et al., 2020c; Prasanna et al., 2020; Liang et al., 2021), making them practical for real-world applications. Similarly, multilingual pre-trained models (Con-neau and Lample, 2019; Conneau et al., 2020; Xue et al., 2021; Luo et al., 2021) are also powerful and even have more parameters. However, little attention has been paid to evaluating the effectiveness of structured pruning on these multilingual models. Applying pruning to multilingual pre-trained  Collaborated work while doing an Alibaba DAMO Academy internship. models is non-trivial, as it typically involves many languages and needs to carefully design the roles of modules within the network. For example, most attention heads have little impact on the performance of monolingual pre-trained models (Michel et al., 2019; Voita et al., 2019), while it is the opposite for multilingual pre-trained models (See Section 5.3 and also Budhraja et al. (2021)). This work intends to examine how structured pruning reacts to multilingual pre-trained models. We take the most representative multilingual pre-trained model family, XLM-R (Conneau et al., 2020; Goyal et al., 2021) for our case study and evaluate the pruning performance on nine cross-lingual understanding tasks in XTREME (Hu et al., 2020). We investigate three aspects of structured pruning: settings, algorithms, and efficiency. Settings Traditional pruning produces a single small model, which is shared across languages ( shared setting). Recent work on multilingual translation (Li et al., 2020; Lin et al., 2021; Xie et al., 2021; Gong et al., 2021) suggests that tailoring pruning to one language could achieve better results ( non-shared setting). However, our comprehensive experiments show that neither of the two settings can consistently outperform the other one (See Section 5.2). Algorithms There exists a broad spectrum of pruning algorithms (Hoefler et al., 2021), and it is impossible to test all of them considering the cost of pre-training. We focus on two pruning algorithms that have been studied the most in monolingual pre-trained models: the regularization-based pruning (Louizos et al., 2018; Wang et al., 2020c) (and our improved version) and the gradient-based pruning (Michel et al., 2019; Prasanna et al., 2020; Liang et al., 2021) (See Section 4). We experimentally find that the simplest gradient-based pruning is more effective for XLM-R (See Section 5.2). speed of the pruned model vary with the sparsity (Hoefler et al., 2021). However, most pruning algorithms, including those we study in this work, require training the model for each specific sparsity. This limitation makes comparisons against a range of sparsity levels infeasible due to the prohibitive training cost. To solve this issue, we propose the Dynamic Sparsification (DS for short), a simple method that parameterizes subnetworks at any sparsity level and shares their weights afterward (See Section 6.1). DS only trains the model once but can obtain models at any sparsity level during inference. Experiments on XNLI (Conneau et al., 2018) show that DS does not degrade the performance much while dramatically reducing the training cost. Interestingly, we observe that the model size and inference speed are not strongly correlated in XLM-R. This observation suggests that one could not obtain a fast model by simply making the model small by using vanilla pruning algorithms (See Section 6.2). Settings Recent multilingual translation research suggests that adapting subnetworks for each language or language pair rather than for all of them gives better results. Among them, Li et al. (2020) train a shared multilingual model, then select layers for each language pair. Lin et al. (2021) also prune a shared multilingual model for each language pair, though on the level of entries in weight matrices. Instead, Gong et al. (2021) prune attention heads and feedforward networks for each language. Xie et al. (2021) first identify general and language-specific neurons in a shared multilingual network, then tune those neurons using the data of their corresponding language only. These findings inspire us to extend from multilingual translation to see how non-shared pruning settings work on multilingual pre-training. Algorithms There are many structured pruning techniques proposed for monolingual pre-trained language models recently. Michel et al. (2019) propose a simple gradient-based importance score to prune attention heads. Prasanna et al. (2020); Liang et al. (2021) extend to prune other components like the feedforward network of the Transformer (Vaswani et al., 2017). Wang et al. (2020c) decompose the pre-trained model weights and apply L 0 regularization (Louizos et al., 2018) to regulate the ranks of decomposed weights. Sajjad et al. (2020) study layer pruning and show that directly dropping N  Word Embedding Multi-HeadAttention Add & Norm Feed-ForwardNetwork Add & Norm the ana Embedding Rank Head 4 Head 3 Head 2 Head 1 Attention Head Hidden Unit Figure 1: The left is the Transformer encoder, the right is the components that will be pruned at each layer. the top layers performs the best in fine-tuning. Peer et al. (2021) further show that by carefully choosing layers to drop, structured pruning can achieve a performance close to those trained by knowledge distillation (Hinton et al., 2015). Efficiency The pruning algorithms mentioned above need to train one network for each sparsity level used at inference. Hou et al. (2020) propose a dynamic structured pruning method based on Michel et al. (2019), which allows training the model once and making the inference with any size of the model. Compared with our Dynamic Sparsification, Hou et al. (2020)'s method cannot be applied to the non-shared setting as it needs to rearrange the network, i.e., producing a new model, for each language. Cascading methods (Schwartz et al., 2020; Xin et al., 2020) can even adapt the network size for each instance. Since cascading methods cannot perform batch inference and are only available for sentence classification tasks, we do not consider them in this work. In this section, we briefly review the structure of XLM-R (Conneau et al., 2020), a Transformer encoder (Vaswani et al., 2017) pre-trained by masked language modeling task (Devlin et al., 2019). We also revisit how conventional structured pruning algorithms are applied to Transformers by introducing additional gating variables and setting appropriate values to them (See Figure 1 and also Prasanna et al. (2020); Liang et al. (2021)). The XLM-R model consists of N layers. Each layer is made of the multihead attention and feedforward 1853 networks, followed by the residual connection and layer normalization. Attention Following Michel et al. (2019)'s formula, the multihead attention is written as: MHA( X ) = H (cid:88) i =1 G h,i head i (1) where H is the number of heads, head i is the output of i -th head and G h,i is the i -th entry of the gating variables G h  RH . G h,i indicates whether the head i will be pruned. G h,i is set to 1 to retain that head and 0 if to drop it. Different pruning algorithms will have their own ways to determine the values of G h . Feedforward Network The feedforward network contains two linear projections with GeLU activation (Hendrycks and Gimpel, 2016) in between: FFN( X ) = (GeLU( XW 1 + b 1 ) (cid:12) G f ) W 2 + b 2 (2) where W 1  R d  d f , b 1  R d f , W 2  R d f  d and b 2  R d are weights of the feedforward network and d f is the hidden size. (cid:12) denotes the Hadamard product and G f  R d f is a gating vector with a value in the range of [0, 1]. G f functions similar to G h in multihead attention, except that G f controls the activation of hidden units. Embedding To prune the large embedding matrix E (occupying 69% of all parameters), we decompose it via low-rank approximation as in Lan et al. (2020): E =  E diag( G e ) P (3) where  E  R v  d and P  R d  d are the decomposed matrices of E . v is the vocabulary size. G e  R d , governing the rank of E , is a gating vector similar to G h and G f . diag( G e ) converts G e to a diagonal matrix. The right part of Figure 1 is an illustration of the components (such as hidden units, attention heads, and embeddings) that will be pruned. This section will first introduce pruning algorithms that we study and then describe how to adapt them to two pruning settings. The first is the shared setting that shares the pruned network across languages (default setting that all pruning algorithms could run on), and the second is the non-shared setting that prunes one subnetwork for each language (Xie et al., 2021; Gong et al., 2021). Gradient-based pruning (Michel et al., 2019) computes the importance score of each component, e.g., heads in Eq. 1. Then it sets the gating variable of a component, e.g., G h,i in Eq. 1, to 1 if its importance score is larger than a threshold and 0 otherwise. Taking an attention head i as an example, its importance score is defined as: I head i = EX  X (cid:12)(cid:12)(cid:12)(cid:12) head Ti  LMLM ( X )  head i (cid:12)(cid:12)(cid:12)(cid:12) (4) where X is the data distribution and we choose the validation set as X in practice, LMLM is the masked language modeling loss (Devlin et al., 2019). The values of gating variables are set and frozen after pre-training. An additional phase of pre-training is further employed to update network parameters to recover performance loss brought by pruning. Extending gradient-based pruning to the non-shared setting is straightforward: to prune for one language, we use data of that language to compute a unique set of gating variables G = { G h , G f , G e } for it. The L 0 norm has been widely used in many areas, including signal processing (Zhang, 2010; Xu et al., 2011) to induce sparsity. In neural networks, regularization-based pruning, also referred to as L 0 regularization (Louizos et al., 2018), defines a differentiable L 0 norm on the gating variables G = { G h , G f , G e } . It controls the network sparsity by learning the values of G during pre-training. Taking a gating variable g  G as an example, it is modeled as: u  U (0 , 1) (5) s = sigmoid((log u/ (1  u ) +  ) / ) (6)  s = s  ( r  l ) + l (7) g = min(1 , max(0 ,  s )) (8) where U is the uniform distribution, l < 0 and r > 1 are two fixed constants,  is the temperature and  is a learnable parameter of g . During training, u is sampled for each g separately. At inference, 1854 Eq. 6 becomes s = sigmoid(  ) . Compared with gradient-based pruning, the importance score in L 0 regularization is the learnt  and the threshold is fixed to sigmoid  1 (cid:16)  l r  l (cid:17) . The L 0 regularization term of g is: || g || 0 = sigmoid (   log(  l/r )) (9) and the overall L 0 regularization term is 1 : LL 0 = || G || 0 = (cid:88) g  G || g || 0 (10) LL 0 will be multiplied by a hyper-parameter  1 and added to the pre-training loss LMLM . Two issues of the previous native L 0 regularization emerge in practice: 1) The hyper-parameter  1 does not relate to the model sparsity. It requires several expensive try-outs training runs to find an appropriate setup that can reach desired sparsity (Wang et al., 2020c). 2) If we extend L 0 regularization to non-shared setting as done in gradient-based pruning, it easily converges to an optimum where every language shares the network (Gong et al., 2021). This falls back to the shared setting. Thus, we propose two corresponding solutions as below: 1) Sparsity Constraint To address the first issue, we add a sparsity constraint to Eq. where l is the number of languages and G i denotes the set of gating variables for language i . This loss term will keep the subnetwork size of each language close to the targeted size t . 2 2) Diverse Subnetwork To address the second issue, we introduce a diversity loss term to encourage the model to find a distinct subnetwork for each language. It is achieved by diagonalizing the gram matrix of gating variables  G = [ G 1 ;    ; G l ] : L diag = || P (cid:12)  G  GT (cid:12) ( 1  I ) || 1 (12) 1 In practice we weigh the L 0 regularization term of gating variables (See Appendix B). where 1 is a matrix of ones and I is the identity matrix. P  R l  l is used to introduce linguistic prior and is a matrix of ones by default. Eq. 12 will penalize each language pair equally. Intuitively, the subnetworks of two languages that are close, e.g., English and Spanish, should not be penalized. Thus we add linguistic prior P ij = 0 when the i -th and j -th languages belong to the same language family (See Appendix C) and 1 otherwise. To the end, the loss L we used in pre-training is: L = LMLM +  1 LL 0 +  2 L diag (13) Note that the parameter of the gating variable  is randomly initialized. We find that tuning only  in the first few epochs is crucial to obtain better performance. If no further notice, we will use this improved L 0 regularization for experiments with non-shared setting and the native L 0 regularization for shared setting. Pre-training Our pruned models are trained on the CC-100 corpus (Wenzek et al., 2020). We choose 100 languages with a total size of 2.2TB for training, which is consistent with those used in XLM-R (Conneau et al., 2020). The development set we used to induce the importance score for pruning is 3K randomly selected samples from the CC-100 corpus per language. Our model is a 12-layer Transformer with a 768 embedding size and a 3072 hidden size. It is pruned and continually trained based on the publicly available XLM-R model for 150K steps with a batch size of 2048 and a learning rate of 0.0002. Other hyper-parameters remain the same as in the original paper (Conneau et al., 2020). We train our model on 32 Nvidia Tesla V100 32GB GPUs with mixed-precision training. It takes roughly 7-10 days to pre-train one model. For inference, we use 1 Nvidia Tesla V100 32GB GPU and Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz to estimate the GPU and CPU throughput (with a batch size of 128 for GPU and 1 for CPU). Fine-tuning We evaluate the pruned models on 9 downstream tasks from XTREME (Hu et al., 2020). These tasks can be classified into four different categories: (1) sentence-pair classification: XNLI (Conneau et al., 2018), PAWS-X (Yang 1855 Task Sparsity XNLI PAWS-X POS NER XQuAD MLQA TyDiQA BUCC Tatoeba Avg Metrics Acc. et al., 2019); (2) structured prediction: POS (Nivre et al., 2018), Wikiann NER (Pan et al., 2017); (3) question answering: XQuAD (Artetxe et al., 2020), MLQA (Lewis et al., 2020), TyDiQA (Clark et al., 2020); (4) sentence retrieval: BUCC2018 (Zweigenbaum et al., 2017), Tatoeba (Artetxe and Schwenk, 2019). The hyper-parameter setup of fine-tuning could be found in Appendix A. Following previous work (Hu et al., 2020), we study the pruned models in two fine-tuning settings: Cross-lingual Transfer (a.k.a., zero-shot) and Translate-Train-All (a.k.a., multi-task). Note that for the two sequence labelling tasks POS and NER, translation cannot give us the correct training labels. We thus use human-annotated data for translate-train-all training on them. Table 1 shows the fine-tuning results of using different methods to prune XLM-R to 50% sparsity (also the value of t in Eq. 11). We follow the convention of Prasanna et al. (2020) to compute the sparsity of the encoder, which excludes the embeddings in the calculation. For DistilBERT , we remove half of the original layers of XLM-R as done in Sanh et al. (2019). Note that in Table 1 (the rows of  L 0 (shared)), regularization-based pruning with shared setting has a lower sparsity (20%). 3 3 We have tried various hyper-parameters settings to pretrain models toward 50% sparsity (for a fair comparison with Methods Sparsity XNLI POS NER TyDiQA Avg L 0 20% 73.4 87.5 85.1 61.2/45.9 74.9 Impv. regularization-based pruning. Table 1 shows that vanilla L 0 in shared setting has more parameters (20% sparsity) but performs worse than gradient-based pruning with fewer parameters (50% sparsity). Despite that our proposed improved L 0 works better ( non-shared setting), it still underperforms the gradient-based pruning counterpart. This is because regularization-based pruning keeps modifying the subnetwork structure when weights are updating, which might introduce too much noise during training. Gradient-based pruning, on the other hand, keeps the pruned network unchanged and adapts weights only. Despite that some works (Hoefler et al., 2021) suggest that regularization-based pruning should be preferred, it might not be the same conclusion for XLM-R. DistilBERT ) using vanilla L 0 , but the resulting sparsity is either too high (  70%) or too low (  20%). This is in line with the trainability issue of L 0 as indicated in Section 4.2. translation has suggested that non-shared setting provides consistent gains, as this way allows the pruned model to adapt for each language (Li et al., 2020; Lin et al., 2021; Xie et al., 2021; Gong et al., 2021). However, this is not the case for XLM-R. As shown in Table 1, regularization-based pruning ( L 0 ) works the best with the non-shared settings 4 , but for gradient-based pruning it is the shared setting. We analyze that this is because XLM-R covers more low-resource languages (100 languages in XLM-R vs. 24 in most multilingual translation research), which makes sharing the subnetwork for a universal representation more preferable (Aharoni et al., 2019). Simple distillation performs less effective than pruning. For most tasks, distillation is not as effective as pruning. 5 This might be that distillation prunes a whole layer, while more fine-grained components are pruned in structured pruning. But combining distillation with pruning could provide some gain, as shown in Table 2. Our improved L 0 regularization-based pruning can further boost the performance. In Section 4.2.1, we propose an improved L 0 regularization to solve the drawbacks of standard L 0 . Table 2 shows the results. Through the sparsity constraint, we can control the model sparsity to be the desired value t = 50% instead of 20% (the closest we could have using vanilla L 0 ). And along with diverse subnetwork, the improved L 0 can even consistently improve the fine-tuning results. Appendix E visualizes how subnetworks differ between two languages after applying the diversity loss term. 4 Non-shared model with more parameters dropped (50% sparsity) is better than shared model with fewer parameters dropped (20% sparsity). 5 Although adopting advanced distillation techniques might improve the result, the pruning algorithm is also simple here. Moreover, integrating with distillation (the last row of Table 2) can further improve the results. Why does regularization-based pruning perform poorly? Since regularization-based pruning learns the subnetwork from scratch, we believe its poor performance results from the low-resource languages. We choose XNLI with the translate-train-all setting for empirical verification. On the one hand, the translate-train-all setting ensures that each language has the same dataset for fine-tuning (except for NER and POS). This way eliminates the difference in fine-tuning. On the other hand, among all tasks except NER and POS, XNLI covers more languages. Figure 2 supports our hypothesis. It shows the accuracy loss and corpus size of each language in regularization-based and gradient-based pruning. We observe that for regularization-based pruning accuracy loss strongly correlates with pre-training dataset size (a value of 0.83 for Pearson's  ), while it is not for gradient-based pruning. Where does pruning methods behave differently? In Figure 3, we compare in which aspect different pruning algorithms behave differently. Figure 3 shows the sparsity of each component (attention heads and hidden units) at each layer. Interestingly, we see that gradient-based pruning preserves all attention heads and only a tiny number of hidden units, while regularization-based pruning prunes heads and hidden units more evenly. Though previous works (Michel et al., 2019; Voita et al., 2019) have suggested that most attention heads have little impact on the final performance of monolingual models, our results show that this is not the case for XLM-R. Besides, both pruning methods tend to drop more in the middle layers. In practice, we may need models with different sparsities to fit various resource constraints or compare a set of methods. Nevertheless, existing pruning techniques must train the model independently for each sparsity level, which is prohibitive for large models. Here we propose Dynamic Sparsification ( DS for short), a method that trains the model once but allows inference with any level of sparsity. Section 4 shows that both gradient-based and regularization-based pruning follow the same procedure: we first determine a threshold, then get the importance score for each component, and set the gating variable to 1 if its score is larger than that threshold and 0 otherwise. By adjusting the threshold, one can obtain networks with any sparsity. Based on this, we model a gating variable g as: g = f (  + t ) (14) where  is a trainable importance score as in regularization-based pruning, t is the targeted network size (which is one minus the sparsity), t is the threshold with a learnable  , f is a function with output ranging between 0 and 1. We choose f to be Eqs. 6 8 because it enables us to optimize  and  via L 0 regularization. If  and  are set properly, Eq. 14 will automatically determine whether its corresponding component should be activated under the targeted network size t . Then is how to find  and  using pruning algorithms. We know that pruning algorithms could rank different components by their importance scores. Based on this ranking, we identify the boundary network size that a specific component will be activated (denoted as  t ) and will not. These Methods XNLI POS NER TyDiQA Avg Grad (shared) 76.8 88.4 88.0 69.5/54.6 78.8 + DS 74.6 87.6 87.1 64.0/48.3 76.4 L 0 (non-shared) 76.3 87.9 86.8 67.8/52.5 77.8 + DS 76.2 87.9 86.7 67.9/52.4 77.7 Table 3: The results of gradient-based and regularization-based pruning with or without dynamic sparsification (Sparsity=50%). where  is the network size that one component contributes to,  t is the boundary network size where the corresponding gating variable g should be 1 if t >  t and 0 if t <  t   . t equals the ranking divided by the total number of components. Eq. 15 has a closed-form solution for  and  : 6 (cid:40)  = (cid:0) 1   t/ (cid:1) f  1 (1) + ( t/ ) f  1 (0)  = (cid:0) f  1 (1)  f  1 (0) (cid:1) / (16) Before training, we use gradient-based pruning to initialize  and  via Eq. 16. If only gradient-based pruning is adopted,  and  are then clamped and only the retained network parameters will be updated, otherwise they can be jointly optimized via regularization-based pruning. During training, we sample different t s to train different sized subnetworks. At inference, t is set to the targeted network size to prune the model. If one wants to extend DS to non-shared setting, he can prune for each language once and compute a unique set of  and  for each language. Table 3 ( + DS rows) shows the 50% sparsity results after applying DS to the two pruning algorithms under their best performing pruning settings (according to Table 1). Surprisingly, we observe that gradient-based pruning with shared setting suffers from a significant loss, while regularization-based pruning with non-shared setting has almost no loss. This is because DS shares the weights between subnetworks of different sparsities hurts the model capacity, and non-shared setting enlarges the subnetwork capacity by untying weights of different languages. Due to the expensive cost of training models without DS, we only test the impact of DS on 50% sparsity, but we compare it with other systems with a smaller size (See Appendix F). The leftmost part of Figure 4 shows more on how the two pruning methods trade accuracy for efficiency under various sparsities. The second sub-figure from the left of Figure 4 shows a non-linear relationship between the number of parameters and sparsity, as embeddings are not included in sparsity calculation (Prasanna et al., 2020). Since embeddings are more important than most parts of the model and are very large (69% of the overall parameters), the number of parameters remains high even when the encoder is quite sparse (Sparsity  50% ). Pruning algorithms only start to prune these large embeddings when the encoder is very sparse (Sparsity > 50% ) and results in a great drop in the number of parameters, as shown in Figure 5. The two rightmost panels of Figure 4 describe how the CPU and GPU throughput vary as the sparsity changes. We observe a strong correlation between the CPU throughput and sparsity when the sparsity  50% . However, there is no such trend observed when the sparsity < 50% . This might be 90 70 50 30 10 0 50 100 Network Sparsity [%] L a y e r S p a r s it y [ % ] Layer 1 Layer 3 Layer 5 Layer 7 Layer 9 Layer 11 Figure 6: Sparsity of different layers pruned by regularization-based pruning vs. the sparsity. due to the time consumption of irregular memory access out-weights the speed-up brought by the small tensor computation. Interestingly, we see that sparse models show no acceleration on GPU even when the sparsity is high (e.g., 90%). Although pruning algorithms here optimize the model size instead of inference efficiency, it is expected that the resulting sparse models still have speedup as shown in CPU and in other work (Wang et al., 2020c). In Figure 6, we find that the highest sparsity of all layers is close to but not exactly 100%. This implies that pruning tends to produce a deep and narrow model. Previous studies (Sanh et al., 2019; Wang et al., 2020a; Li et al., 2021) show that GPU throughput is more sensitive to the model height instead of its width. This explains why we did not observe any acceleration even for a model with 1 / 10 of the original size. Though not shown in Table 1 and Figure 4, it is still possible to obtain actual speedup in GPU for sparse models. Previous observations on GPU throughput only hold for inference with the same batch size . In practice, the sparse models have a smaller memory footprint and we can use a larger batch size for higher parallelism. For pruned models in Table 1, a nearly 2  speedup is observed when we double the inference batch size. In summary, Figure 4 suggests that the correlation between the model size and throughput is very week for XLM-R : for model size, reducing the embedding size is important, but it has almost no impact on throughput (an O (1) complexity table lookup); for throughput, compressing parts other than embeddings is more effective as shown in Figure 4, but they have much fewer parameters than the embeddings (193M parameters for embeddings vs. 86M for the others). This advocates special 1859 care needed to be taken if one wants to compress and accelerate XLM-R simultaneously. Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics , 7(0):597610. Aakriti Budhraja, Madhura Pande, Pratyush Kumar, and Mitesh M. Khapra. 2021. On the prunability of attention heads in multilingual BERT. CoRR , abs/2109.12683. Hongyu Gong, Xian Li, and Dmitriy Genzel. 2021. Here we study what DS will prune under various sparsities. Figure 5 shows which component (em-beddings, attention heads and hidden units) will be preferred during pruning. In general, gradient-based pruning behaves similar to regularization-based pruning: they first prune hidden units, and only prune attention heads and embeddings when the sparsity is high. The main difference between them is that gradient-based pruning starts to prune embeddings earlier (at 70% sparsity) than regularization-based pruning. This explains why we observe a significant drop in performance for gradient-based pruning with 70% sparsity (See the left of Figure 4): the model already lost much information at the beginning and there is no way to recover. Figure 6 shows how regularization-based pruning prunes each layer with DS. Though we do not plot the curves of gradient-based pruning, its phenomenon is similar to regularization-basd pruning. We find that regularization-based pruning behaves differently at low and high sparsity. It first prunes bottom layers when the sparsity is low, then gradually shift to higher layers as the sparsity increases. In the end, it retains more parameters in the bottom layers instead of the top layers. This provides insight for future model design: a pyramid structure is better when the model size is very small . In this work, we study three aspects of structured pruning on multilingual pre-trained models: settings, algorithms and efficiency. Experiments show interesting phenomena: The best pruning setting depends on the choice of algorithms; The simplest pruning algorithm performs the best; A fast model does not mean it should be small. We hope this work will give insight to future research.
abstain objective abstain method method abstain abstain abstain abstain abstain abstain abstain method method objective abstain abstain result abstain result result abstain method abstain objective abstain abstain result abstain other other other other other objective other other other other other other other other other abstain other method method abstain other other other other other other other other other method other other other other objective other abstain abstain abstain method abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method method method abstain abstain method abstain method method abstain other method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain method abstain abstain abstain result abstain result abstain abstain abstain abstain abstain result abstain abstain abstain result abstain result abstain abstain result abstain abstain abstain result other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain