The growing size of neural language models has led to increased attention in model compression. The two predominant approaches are pruning , which gradually removes weights from a pre-trained model, and distillation , which trains a smaller compact model to match a larger one. Pruning methods can significantly reduce the model size but hardly achieve large speedups as distillation. However, distillation methods require large amounts of unlabeled data and are expensive to train. In this work, we propose a task-specific structured pruning method CoFi 1 ( Co arseand Fi ne-grained Prun-ing), which delivers highly parallelizable subnetworks and matches the distillation methods in both accuracy and latency, without resorting to any unlabeled data. Our key insight is to jointly prune coarse-grained (e.g., layers) and fine-grained (e.g., heads and hidden units) modules, which controls the pruning decision of each parameter with masks of different granularity. We also devise a layerwise distillation strategy to transfer knowledge from unpruned to pruned models during optimization. Our experiments on GLUE and SQuAD datasets show that CoFi yields models with over 10  speedups with a small accuracy drop, showing its effectiveness and efficiency compared to previous pruning and distillation approaches. 2 1 Introduction Pre-trained language models (Devlin et al., 2019; Liu et al., 2019a; Raffel et al., 2020, inter alia ) have become the mainstay in natural language processing. These models have high costs in terms of storage, memory, and computation time and it has motivated a large body of work on model compression to make them smaller and faster to use in real-world applications (Ganesh et al., 2021). The 1 CoFi is pronounced as . Pruning methods search for an accurate subnetwork in a larger pre-trained model. Recent work has investigated how to structurally prune Transformer networks (Vaswani et al., 2017), from removing entire layers (Fan et al., 2020; Sajjad et al., 2020), to pruning heads (Michel et al., 2019; Voita et al., 2019), intermediate dimensions (McCarley et al., 2019; Wang et al., 2020b) and blocks in weight matrices (Lagunas et al., 2021). The trend of structured pruning leans towards removing fine-grained units to allow for flexible final structures. However, thus far, pruned models rarely achieve large speedups (2-3  improvement at most). By contrast, distillation methods usually first 3 Following previous work, we exclude embedding matrices in calculating the number of parameters. specify a fixed model architecture and perform a general distillation step on an unlabeled corpus, before further fine-tuning or distillation on task-specific data (Sanh et al., 2019; Turc et al., 2019; Sun et al., 2019; Jiao et al., 2020). Well-designed student architectures achieve compelling speedup-performance tradeoffs, yet distillation to these randomly-initialized student networks on large unlabeled data is prohibitively slow. 4 For instance, TinyBERT (Jiao et al., 2020) is first trained on 2,500M tokens for 3 epochs, which requires training 3.5 days on 4 GPUs (Figure 1). 5 In this work, we propose a task-specific, structured pruning approach called CoFi ( Co arse and Fi ne-grained Pruning) and show that structured pruning can achieve highly compact subnetworks and obtain large speedups and competitive accuracy as distillation approaches, while requiring much less computation. Our key insight is to jointly prune coarse-grained units (e.g., self-attention or feed-forward layers) and fine-grained units (e.g., heads, hidden dimensions) simultaneously. Different from existing works, our approach controls the pruning decision of every single parameter by multiple masks of different granularity. This is the key to large compression, as it allows the greatest flexibility of pruned structures and eases the optimization compared to only pruning small units. It is known that pruning with a distillation objective can substantially improve performance (Sanh et al., 2020; Lagunas et al., 2021). Unlike a fixed student architecture, pruned structures are unkown prior to training and it is challenging to distill between intermediate layers of the unpruned and pruned models (Jiao et al., 2020). Hence, we propose a layerwise distillation method, which dynamically learns the layer mapping between the two structures. We show that this strategy can better lead to performance gains beyond simple prediction-layer distillation. Our experiments show that CoFi delivers more accurate models at all levels of speedups and model sizes on the GLUE (Wang et al., 2019) and SQuAD v1.1 (Rajpurkar et al., 2016) datasets, compared to strong pruning and distillation baselines. Concretely, it achieves over 10  speedups and a 95% sparsity across all the datasets while preserving 4 There are exceptions like DistillBERT (Sanh et al., 2020), which initializes the student from the teacher by taking one layer out of two, yet it is unclear how to generalize this initialization scheme to other compact structures. 5 See training time measurement details in Appendix J. more than 90% of accuracy. Our results suggest that task-specific structured pruning is an appealing solution in practice, yielding smaller and faster models without requiring additional unlabeled data for general distillation. A Transformer network (Vaswani et al., 2017) is composed of L blocks and each block consists of a multi-head self-attention (MHA) layer, and a feed-forward (FFN) layer. An MHA layer with N h heads takes an input X and outputs: MHA( X ) = (cid:80) N h i =1 Att( W ( i ) Q , W ( i ) K , W ( i ) V , W ( i ) O , X ) , where W ( i ) Q , W ( i ) K , W ( i ) V , W ( i ) O  R d  d h denote the query, key, value and output matrices respectively and Att(  ) is an attention function. Here d denotes the hidden size (e.g., 768) and d h = d/N h denotes the output dimension of each head (e.g., 64). Next comes a feed-forward layer, which consists of an up-projection and a down-projection layer, parameterized by WU  R d  d f and WD  R d f  d : FFN( X ) = gelu( XWU )  WD . Typically, d f = 4 d . There is also a residual connection and a layer normalization operation after each MHA and FFN layer. MHAs, FFNs account for 1 / 3 and 2 / 3 of the model parameters in Transformers (embeddings excluded). According to Ganesh et al. (2021), both MHAs and FFNs take similar time on GPUs while FFNs become the bottleneck on CPUs. Knowledge distillation (Hinton et al., 2015) is a model compression approach that transfers knowledge from a larger teacher model to a smaller student model. General distillation (Sanh et al., 2019; Sun et al., 2020; Wang et al., 2020a) and task-specific distillation (Sun et al., 2019) exploit unlabeled data and task-specific data respectively for knowledge transfer. A combination of the two leads to increased performance (Jiao et al., 2020). General distillation or pre-training the student network on unlabeled corpus is essential for retaining performance while being computationally expensive (Turc et al., 2019; Jiao et al., 2020). Most distillation approaches assume a fixed student structure prior to training. Hou et al. (2020) attempt to distill to a dynamic structure with specified widths and heights. Yin et al. (2021) adopt a one-shot Neural Architecture Search solution to search architectures of student networks. Pruning gradually removes redundant parameters from a teacher model, mostly producing task-specific models. Previous works focus on pruning different components in Transformer models, from coarse-grained to fine-grained units. Layer pruning Fan et al. (2020) and Sajjad et al. (2020) explore strategies to drop entire Transformer blocks (a pair of MHA and FFN layer) from a pre-trained model. Empirical evidence suggests that 50% of layers can be dropped without big accuracy drop, resulting in a 2  speedup. Head pruning Voita et al. (2019); Michel et al. (2019) show that only a small subset of heads are important and the majority can be pruned. We 6 CoFi requires slightly longer training time compared to the task-specific distillation of TinyBERT, as CoFi searches model structures and learns parameters simultaneously. Only removing heads does not lead to large latency improvementLi et al. (2021) demonstrate a 1.4  speedup with only one remaining head per layer. FFN pruning The other major partfeed-forward layers (FFNs)are also known to be overparam-eterized. Strategies to prune an FFN layer for an inference speedup include pruning an entire FFN layer (Prasanna et al., 2020; Chen et al., 2020b) and at a more fine-grained level, pruning intermediate dimensions (McCarley et al., 2019; Hou et al., 2020) by introducing z int  { 0 , 1 } d f : FFN( X ) = gelu( XWU )  diag( z int )  WD . Block and unstructured pruning More recently, pruning on a smaller unit, blocks, from MHAs and FFNs have been explored (Lagunas et al., 2021). However, it is hard to optimize models with blocks pruned thus far: Yao et al. (2021) attempt to optimize block-pruned models with the block sparse MatMul kernel provided by Triton (Tillet et al., 2019), but the reported results are not competitive. Similarly, unstructured pruning aims to remove individual weights and has been extensively studied in the literature (Chen et al., 2020a; Huang et al., 2021). Though the sparsity reaches up to 97% 1515 (Sanh et al., 2020), it is hard to obtain inference speedups on the current hardware. Combination with distillation Pruning is commonly combined with a prediction-layer distillation objective (Sanh et al., 2020; Lagunas et al., 2021). Yet it is not clear how to apply layerwise distillation strategies as the pruned student model's architecture evolves during training. We propose a structured pruning approach CoFi, which jointly prunes Co arse-grained and Fi ne-grained units (3.1) with a layerwise distillation objective transferring knowledge from unpruned to pruned models (3.2). A combination of the two leads to highly compressed models with large inference speedups. Recent trends in structured pruning move towards pruning smaller units for model flexibility. Pruning fine-grained units naturally entails pruning coarse-grained unitsfor example, pruning N h (e.g., 12) heads is equivalent to pruning one entire MHA layer. However, we observe that this rarely happens in practice and poses difficulty to optimization especially at a high sparsity regime. To remedy the problem, we present a simple solution: we allow pruning MHA and FFN layers explicitly along with fine-grained units (as shown in 2.3) by introducing two additional masks z MHA and z FFN for each layer. Now the multi-head self-attention and feed-forward layer become: MHA( X ) = z MHA  N h (cid:88) i =1 ( z ( i )head  Att( W ( i ) Q , W ( i ) K , W ( i ) V , W ( i ) O , X ) , FFN( X ) = z FFN  gelu( XWU )  diag( z int )  WD . With these layer masks, we explicitly prune an entire layer, instead of pruning all the heads in one MHA layer (or all the intermediate dimensions in one FFN layer). Different from the layer dropping strategies in Fan et al. (2020); Sajjad et al. (2020), we drop MHA and FFN layers separately, instead of pruning them as a whole. Furthermore, we also consider pruning the output dimensions of MHA( X ) and FFN( X ) , referred to as hidden dimensions' in this paper, to allow for more flexibility in the final model structure. We define a set of masks z hidn  { 0 , 1 } d , shared across layers because each dimension in a hidden representation is connected to the same dimension in the next layer through a residual connection. These mask variables are applied to all the weight matrices in the model, e.g., diag( z hidn ) WQ . Empirically, we find that only a small number of dimensions are pruned (e.g., 768  760 ), but it still helps improve performance significantly (4.3). CoFi differs from previous pruning approaches in that multiple mask variables jointly control the pruning decision of one single parameter. For example, a weight in an FFN layer is pruned when the entire FFN layer, or its corresponding intermediate dimension, or the hidden dimension is pruned. As a comparison, a recent work Block Pruning (Lagu-nas et al., 2021) adopts a hybrid approach which applies a pruning pruning strategy on MHAs and FFNs separately. To learn these mask variables, we use l 0 regularization modeled with hard concrete distributions following Louizos et al. (2018). We also follow Wang et al. (2020b) to replace the vanilla l 0 objective with a Lagrangian multiplier to better control the desired sparsity of pruned models. 7 We adapt the sparsity function accordingly to accommodate pruning masks of different granularity:  s = 1 M  4  d h  (cid:80) Li (cid:80) N h j (cid:80) dk z ( i ) MHA  z ( i,j ) head  z ( k ) hidden + 1 M  2  (cid:80) Li (cid:80) d f j (cid:80) dk z ( i ) FFN  z ( i,j ) int  z ( k ) hidden , where  s is the expected sparsity and M denotes the full model size. All masking variables are learned as real numbers in [0 , 1] during training and we map the masking variables below a threshold to 0 during inference and get a final pruned structure where the threshold is determined by the expected sparsity of each weight matrix (see Appendix B for more details). Previous work has shown that combining distillation with pruning improves performance, where the distillation objective only involves a cross-entropy loss between the pruned student's and the teacher's output probability distributions p s and p t (Sanh et al., 2020; Lagunas et al., 2021): 7 We also tried a straight-through estimator as proposed in Sanh et al. (2020) and found the performance comparable. We choose l 0 regularization because it is easier to control the sparsity precisely. In addition to prediction-layer distillation, recent works show great benefits in distillation of intermediate layers (Sun et al., 2019; Jiao et al., 2020). In the context of distillation approaches, the architecture of the student model is pre-specified, and it is straightforward to define a layer mapping between the student and teacher model. For example, the 4-layer TinyBERT 4 model distills from the 3 , 6 , 9 and 12 -th layer of a 12-layer teacher model. However, distilling intermediate layers during the pruning process is challenging as the model structure changes throughout training. We propose a layerwise distillation approach for pruning to best utilize the signals from the teacher model. Instead of pre-defining a fixed layer mapping, we dynamically search a layer mapping between the full teacher model and the pruned student model. Specifically, let T denote a set of teacher layers that we use to distill knowledge to the student model. We define a layer mapping function m (  ) , i.e., m ( i ) represents the student layer that distills from the teacher layer i . The hidden layer distillation loss is defined as L layer = (cid:88) i T MSE ( W layer H m ( i ) s , H it ) , where W layer  R d  d is a linear transformation matrix, initialized as an identity matrix. H m ( i ) s , H it are hidden representations from m ( i ) -th student FFN layer and i -th teacher FFN layer. The layer mapping function m (  ) is dynamically determined during the training process to match a teacher layer to its closest layer in the student model: m ( i ) = arg min j : z ( j ) FFN > 0 MSE ( W layer H js , H it ) . Calculating the distance between two sets of layers is highly parallelizable and introduces a minimal training overhead. To address the issue of layer mismatch, which mostly happens for small-sized datasets, e.g., RTE, MRPC, we add a constraint to only allow matching a teacher layer to a lower student layer than the previously matched student layer. When pruning with larger sized datasets, layer mismatch rarely happens, showing the superiority of dynamic matchinglayers between student and teacher models match in a way that benefits the pruning process the most. where  controls the contribution of each loss. Datasets We evaluate our approach on eight GLUE tasks (Wang et al., 2019) and SQuAD v1.1 (Ra-jpurkar et al., 2016). GLUE tasks include SST-2 (Socher et al., 2013), MNLI (Williams et al., 2018), QQP, QNLI, MRPC (Dolan and Brockett, 2005), CoLA (Warstadt et al., 2019), STS-B (Cer et al., 2017) and RTE (see Appendix D for dataset sizes and metrics). Training setup In our experiments, sparsity is computed as the number of pruned parameters divided by the full model size (embeddings ex-cluded). Following Wang et al. (2020b); Lagunas et al. (2021), we first finetune the model with the distillation objective, then we continue training the model with the pruning objective with a scheduler to linearly increase the sparsity to the target value. We finetune the pruned model until convergence (see Appendix A for more training details). We train models with target sparsities of { 60% , 70% , 75% , 80% , 85% , 90% , 95% } on each dataset. For all the experiments, we start from the BERT base model 8 and freeze embedding weights following Sanh et al. (2020). We report results on development sets of all datasets. Baselines We compare CoFi against several baselines: DistillBERT 6 (Sanh et al., 2019), TinyBERT 6 and TinyBERT 4 (Jiao et al., 2020), DynaBERT (Hou et al., 2020), and Block Pruning (Lagunas et al., 2021) (see Appendix C for details). We also compare to other pruning methods such as FLOP (Wang et al., 2020b), Layer-Drop (Fan et al., 2020), Movement Pruning (Sanh et al., 2020) and distillation methods such as MobileBERT (Sun et al., 2020) and AutoTinyBERT (Yin et al., 2021) in Appendix F. 9 For TinyBERT and DynaBERT, the released models are trained with task-specific augmented data. For a fair comparison, we train these two models with the released code without data augmentation. 10 For Block Pruning, we train models 8 We also experiments CoFi on RoBERTa models (Liu et al., 2019a). Please refer to Appendix I for details. 9 We show these results in Appendix F as they are not directly comparable to CoFi. distills the student model on a large unlabeled corpus. Train time is measured in GPU hours (see Appendix J for details). The number of parameters for both models are around 5 M (around 95% sparsity). CoFi closes the gap between distillation and pruning with significantly less computation. Note that we remove data augmentation from TinyBERT for a fair comparison, see Table 3 for experiments with augmented data. actual improvement in inference latency 11 . We use an unpruned BERT base as the baseline and evaluate all the models with the same hardware setup on a single NVIDIA V100 GPU to measure inference speedup. The input size is 128 for GLUE tasks and 384 for SQuAD, and we use a batch size of 128. Note that the results might be different from the original papers as the environment for each platform is different. Overall performance In Figure 2, we compare the accuracy of CoFi models to other methods in terms of both inference speedup and model size. CoFi delivers more accurate models than distillation and pruning baselines at every speedup level 11 Models with the same compression rate could have considerably different speedups. and model size. Block Pruning (Lagunas et al., 2021), a recent work that shows strong performance against TinyBERT 6 , is unable to achieve comparable speedups as TinyBERT 4 . Instead, CoFi has the option to prune both layers and heads & intermediate units and can achieve a model with a comparable or higher performance compared to TinyBERT 4 and all the other models. Additionally, DynaBERT performs much worse speed-wise because it is restricted to remove at most half of the MHA and FFN layers. Comparison with TinyBERT 4 In Table 2, we show that CoFi produces models with over 10  inference speedup and achieves comparable or even better performance than TinyBERT 4 . General distillation (GD), which distills information from a large corpus, is essential for training distillation models, especially for small-sized datasets (e.g., TinyBERT 4 w/o GD performs poorly on CoLA, RTE and STS-B). While general distillation could take up to hundreds of GPU hours for training, CoFi trains for a maximum of 20 hours on a task-specific dataset with a single GPU. We argue that pruning approachestrained with distillation objectives like CoFiare more economical and efficient in achieving compressed models. We further compare CoFi with TinyBERT 4 under the data augmentation setting in Table 3. As the augmented dataset is not publicly released, we follow its GitHub repository to create our own augmented data. We train CoFi with the same set of augmented data and find that it still outperforms TinyBERT 4 on most datasets. 12 4.3 Ablation Study Pruning units We first conduct an ablation study to investigate how additional pruning units such as MHA layers, FFN layers and hidden units in CoFi affect model performance and inference speedup beyond the standard practice of pruning heads and FFN dimensions. We show results in Table 4 for models of similar sizes. Removing the option to prune hidden dimensions ( z hidn ) leads to a slightly faster model with a performance drop across the board and we find that it removes more layers than CoFi and does not lead to optimal performance 12 We only conduct experiments with data augmentation on four datasets because training on augmented data is very expensive. For example, training on the augmented dataset for MNLI takes more than 200 GPU hours in total. See more details in Appendix E. under a specific sparsity constraint. In addition, removing the layer masks ( z MHA , z FFN ) brings a significant drop in speedup on highly compressed models (95%, 5M). This result shows that even with the same amount of parameters, different configurations for a model could lead to drastically different speedups. However, it does not affect the lower sparsity regime (60%, 34M). In short, by placing masking variables at different levels, the optimization procedure is incentivized to prune units accordingly under the sparsity constraint while maximizing the model performance. Distillation objectives We also ablate on distillation objectives to see how each part contributes to the performance of CoFi in Table 5. We first observe that removing distillation entirely leads to a performance drop up to 1.9-6.8 points across various datasets, showing the necessity to combine pruning and distillation for maintaining performance. The proposed hidden layer distillation objective dynamically matches the layers from the teacher model to the student model. We also experiment with a simple alternative i.e., fixed Hidden Distillation, which matches each layers from the teacher model to the corresponding layer in the student model  if a layer is already pruned, the distillation objective will not be added. We find that fixed hidden distillation underperforms the dynamic layer matching objective used for CoFi. Interestingly, the proposed dynamic layer matching objective consistently converges to a specific alignment between the layers of the teacher model and student model. For example, we find that on QNLI the training process dynamically matches the 3, 6, 9, 12 layers in the teacher model to 1, 2, 4, 9 layers in the student model. 13 Moreover, as shown in the table, removing it hurts the performance for all the datasets except SST-2. Finally, we study the pruned structures produced by CoFi. We characterize the pruned models of sparsities { 60% , 70% , 80% , 90% , 95% } on five datasets. For each setting, we run CoFi three times. Figure 3 demonstrates the number of remaining heads and intermediate dimensions of the pruned models for different sparsities. 14 Interestingly, we discover common structural patterns in the pruned models: (1) Feed-forward layers are significantly pruned 13 Please refer to subsection G.1 for more details. across all sparsities. For example, at the 60% sparsity level, the average number of intermediate dimensions in FFN layers after pruning is reduced by 71% ( 3 , 072  884 ), and the average number of heads in MHA is reduced by 39% ( 12  7 . 3 ). This suggests FFN layers are more redundant than MHA layers. (2) CoFi tends to prune submodules more from upper layers than lower layers. For example, upper MHA layers have fewer remaining heads than lower layers on average. Furthermore, we study the number of remaining FFN and MHA layers and visualize the results in Table 6 for highly compressed models (sparsity = 95% ). Although all the models are roughly of the same size, they present different patterns across datasets, which suggests that there exist different optimal sub-networks for each dataset. We find that on SST-2 and QNLI, the first MHA layer is preserved but can be removed on QQP and SQuAD. We also observe that some layers are particularly important across all datasets. For example, the first MHA layer and the second MHA layer are preserved most of the time, while the middle layers are often removed. Generally, the pruned models contain more MHA layers than FFN layers (see 0 500 1000 1500 A v g i n t e r m ed i a t e d i m 1 2 3 4 5 6 7 8 9 10 11 12 FFN layers 60% 70% 80% 90% 95% 0 2 4 6 8 10 12 A v g #head s 1 2 3 4 5 6 7 8 9 10 11 12 MHA layers ChartDirector (unregistered) from www.advsofteng.com Figure 3: The average intermediate dimensions at each FFN layer and the average number of heads at each MHA layer in the pruned models across five datasets (SST-2, MNLI, QQP, QNLI, and SQuAD 1.1). We study different sparsities { 60% , 70% , 80% , 90% , 95% } . Appendix H), which suggests that MHA layers are more important for solving downstream tasks. Similar to Press et al. (2020), we find that although standard Transformer networks have interleaving FFN layers and MHA layers, in our pruned models, adjacent FFN/MHA layers could possibly lead to a better performance. Structured pruning has been widely explored in computer vision, where channel pruning (He et al., 2017; Luo et al., 2017; Liu et al., 2017, 2019c,b; Molchanov et al., 2019; Guo et al., 2020) is a standard approach for convolution neural networks. The techniques can be adapted to Transformer-based models as introduced in subsection 2.3. Unstructured pruning is another major research direction, especially gaining popularity in the theory of Lottery Ticket Hypothesis (Frankle and Carbin, 2019; Zhou et al., 2019; Renda et al., 2020; Frankle 1520 Dataset Pruned Models SST-2 M F M M M F M F F M F M F M F M F M F M M F M F M F M F M F M M MQNLIM F M M F M F M F M M F M M F M M MM F M M F M F M FMNLIF M F M M F FM F M F M M F M M M F M F M F M M MQQPF M M M F M F F M F F M F M F M F M F F M M F M M F M F M M SQuAD F M F M F M M F M F F M M F M F M F M FF M F M M F M F M F Table 6: Remaining layers in the models pruned by CoFi on different datasets. All models are pruned at a sparsity of 95% . For each setting, we run the experiments three times to obtain three different pruned models. M represents a remaining MHA layer and F represents a remaining FFN layer. et al., 2020; Chen et al., 2020a). Unstructured pruning produces models with high sparsities (Sanh et al., 2020; Xu et al., 2021; Huang et al., 2021) yet hardly bring actual inference speedups. Developing computing platform for efficient sparse tensor operations is an active research area. DeepSparse 15 is CPU inference engine that leverages unstructured sparsity for speedup. Huang et al. (2021) measure the real inference speedup induced by unstructured pruning on Moffett AI's latest hardware platform ANTOM. We do not directly compare to these methods because the evaluation environments are different. While all the aforementioned methods produce task-specific models through pruning, several works explore upstream pruning where they prune a large pre-trained model with the masked language modeling task. Chen et al. (2020a) show a 70%-sparsity model retains the MLM accuracy produced by iterative magnitude pruning. Zafrir et al. (2021) show the potential advantage of upstream unstructured pruning against downstream pruning. We consider applying CoFi for upstream pruning as a promising future direction to produce task-agnostic models with flexible structures. Besides pruning, many other techniques have been explored to gain inference speedups for Transformer models, including distillation as introduced 15 https://github.com/neuralmagic/deepsparse in subsection 2.2, quantization (Shen et al., 2020; Fan et al., 2021), dynamic inference acceleration (Xin et al., 2020) and matrix decomposition (Noach and Goldberg, 2020). We refer the readers to Ganesh et al. (2021) for a comprehensive survey. We propose CoFi, a structured pruning approach that incorporates all levels of pruning, including MHA/FFN layers, individual heads, and hidden dimensions for Transformer-based models. Coupled with a distillation objective tailored to structured pruning, we show that CoFi compresses models into a rather different structure from standard distillation models but still achieves competitive results with more than 10  speedup. We conclude that task-specific structured pruning from large-sized models could be an appealing replacement for distillation to achieve extreme model compression, without resorting to expensive pre-training or data augmentation. Though CoFi can be directly applied to structured pruning for task-agnostic models, we frame the scope of this work to task-specific pruning due to the complexity of the design choices for upstream pruning. We hope that future research continues this line of work, given that pruning from a large pre-trained model could possibly incur less computation compared to general distillation and leads to more flexible model structures. The authors thank Tao Lei from Google Research, Ameet Deshpande, Dan Friedman, Sadhika Mal-ladi from Princeton University and the anonymous reviewers for their valuable feedback on our paper. This research is supported by a Hisashi and Masae Kobayashi *67 Fellowship and a Google Research Scholar Award.
abstain abstain abstain abstain objective method abstain result abstain abstain abstain abstain other abstain abstain objective abstain abstain abstain objective abstain method abstain abstain abstain objective result result abstain abstain result other other other other other other other other other other other other other other other other other other other other method other other other other other other other other other method abstain abstain abstain method method other method method method method abstain method abstain abstain abstain method method method method abstain method method abstain abstain abstain abstain method method method method abstain abstain abstain abstain method abstain abstain method abstain method method method method method method method method method method abstain method abstain abstain abstain abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain other abstain abstain result other other other other other other other method other other other method other abstain objective objective result method method other other