Conventional wisdom in pruning Transformer-based language models is that pruning reduces the model expressiveness and thus is more likely to underfit rather than overfit. However, under the trending pretrain-and-finetune paradigm, we postulate a counter-traditional hypothesis, that is: pruning increases the risk of overfitting when performed at the fine-tuning phase. In this paper, we aim to address the overfitting problem and improve pruning performance via progressive knowledge distillation with error-bound properties. We show for the first time that reducing the risk of overfitting can help the effectiveness of pruning under the pretrain-and-finetune paradigm. Ablation studies and experiments on the GLUE benchmark show that our method outperforms the leading competitors across different tasks. Recently, the emergence of Transformer-based language models (using pretrain-and-finetune paradigm) such as BERT (Devlin et al., 2019) and GPT-3 (Brown et al., 2020) have revolutionized and established state-of-the-art (SOTA) records (be-yond human-level) on various natural language (NLP) processing tasks. These models are first pre-trained in a self-supervised fashion on a large corpus and fine-tuned for specific downstream tasks (Wang et al., 2018). While effective and prevalent, they suffer from redundant computation due to the heavy model size, which hinders their popularity on resource-constrained devices, e.g., mobile phones, smart cameras, and autonomous driving (Chen et al., 2021; Qi et al., 2021; Yin et al., 2021a,b; Li et al., 2021; Choi and Baek, 2020). Various weight pruning approaches (zeroing out certain weights and then optimizing the rest) have been proposed to reduce the footprint requirements of Transformers (Zhu and Gupta, 2018; Blalock  These authors contributed equally Dense model D Sparse model Pruning D Task knowledge Discardedknowledge et al., 2020; Gordon et al., 2020; Xu et al., 2021; Huang et al., 2021; Peng et al., 2021). Conventional wisdom in pruning states that pruning reduces the overfitting risk since the compressed model structures are less complex, have fewer parameters and are believed to be less prone to overfit (Ying, 2019; Wang et al., 2021; Tian et al., 2020; Gerum et al., 2020). However, under the pretrain-and-finetune paradigm, most pruning methods understate the overfitting problem. In this paper, we postulate a counter-traditional hypothesis, that is: model pruning increases the risk of overfitting if pruning is performed at the fine-tuning phase. As shown in Figure 1b, the pretrain-and-finetune paradigm contains two types of knowledge, the general-purpose language knowledge learned during pre-training ( L ) and the task-specific knowledge from the downstream task data ( D ). Compared to conventional pruning that only discards task-specific knowledge (Figure 1a), pruning under pretrain-and-finetune (Figure 1b) discards extra knowledge (red area) learned in pretraining phase. Thus, to recover both the extra discarded general-purpose knowledge and the discarded task-specific knowledge, pruning under 190 0 1000 2000 3000 Training steps 0.4 0.6 0.8 1.0 A cc u r a c y dev set training set pretrain-and-finetune increases the amount of information a model needs, which results in relative data deficiency, leading to a higher risk of overfitting. To empirically verify the overfitting problem, we visualize the training and evaluation performance on a real-world task data of MRPC (Devlin et al., 2019) in Figure 2. From Figure 2 (b), it is observed that the evaluation accuracy on the training dataset remains improved while it keeps the same for the validation set through the training process. From Figure 2 (c), the difference in performance becomes more significant when the pruning rate becomes higher and the performance on the validation set even becomes worse after 2,000 training steps. All these observations verify our hypothesis. The main question this paper attempts to answer is: how to reduce the risk of overfitting of pre-trained language models caused by pruning? However, answering this question is challenging. First, under the pretrain-and-finetune paradigm, both the general-purpose language knowledge and the task-specific knowledge are learned. It is nontrivial to keep the model parameters related to both knowledge when pruning. Second, the amount of data for downstream tasks can be small, such as the data with privacy. Thus, the overfitting problem can easily arise, especially in the face of high pruning rate requirements. A little recent progress has been made on addressing overfitting associated with model compression. However, their results are not remarkable and most of them focus on the vision domain (Bai et al., 2020; Shen et al., 2021). To address these challenges, we propose SPD, a sparse progressive distillation method, for pruning pre-trained language models. We prune and optimize the weight duplicates of the backbone of the teacher model (a.k.a., student modules). Each student module shares the same architecture (e.g., the number of weights, the dimension of each weight) as the duplicate. We replace the corresponding layer(s) of the duplicated teacher model with the pruned sparse student module(s) in a progressive way and name the new model as a grafted model. We validate our proposed method through the ablation studies and the GLUE benchmark. Experimental results show that our method outperforms the existing approaches. We summarize our contributions as follows:  We postulate, analyze, and empirically verify a counter-traditional hypothesis: pruning increases the risk of overfitting under the pretrain-and-finetune paradigm. We propose a sparse progressive pruning method and show for the first time that reducing the risk of overfitting can help the effectiveness of pruning. Moreover, we theoretically analyze that our pruning method can obtain a sub-network from the student model that has similar accuracy as the teacher. Last but not least, we study and minimize the interference between different hyperparameter strategies, including pruning rate, learning rate, and grafting probability, to further improve performance. To summarize, our contribution is determining the overfitting problem of pruning under the pretrain-and-finetune paradigm and proposing the sparse progressive distillation method to address it. We demonstrate the benefits of the proposed framework through the ablation studies. We validate our method on eight datasets from the GLUE benchmark. To test if our method is applicable across 191 Teacher   Input Output Pruning (  " ) Grafted model   Pruning (  $ ) zero non-zero Input Output update Pruning(  % )  Pruning(  & ) Input Output Input Output Input Output  Input Output Input Output Input Output p i =0.25 p i =0.50 p i =0.25 p i =1 p i =1 p i =0.25 p i =0.50 Grafted model   Student modules Grafted model   Grafted model   Grafted model   Grafted model   Grafted model   Final grafted model   ! tasks, we include the tasks of both single sentence and sentence-pair classification. Experimental results show that our method outperforms the leading competitors by a large margin. Network Pruning. Common wisdom has shown that weight parameters of deep learning models can be reduced without sacrificing accuracy loss, such as magnitude-based pruning and lottery ticket hypothesis (Frankle and Carbin, 2019). (Zhu and Gupta, 2018) compared small-dense models and large-sparse models with the same parameters and showed that the latter outperforms the former, showing the large-sparse models have better expressive power than their small-dense counterparts. However, under the pretrain-and-finetune paradigm, pruning leads to overfitting as discussed. Knowledge Distillation (KD). As a common method in reducing the number of parameters, the main idea of KD is that the small student model mimics the behaviour of the large teacher model and achieves a comparable performance (Hinton et al., 2015; Mirzadeh et al., 2020). (Sanh et al., 2019; Jiao et al., 2020; Sun et al., 2020) utilized KD to learn universal language representations from large corpus. However, current SOTA knowledge distillation methods are not able to achieve a high model compression rate (less than 10% remaining weights) while achieving an insignificant performance decrease. Progressive Learning. The key idea of progressive learning is that student learns to update module by module with the teacher. (Shen et al., 2021) utilized a dual-stage distillation scheme where student modules are progressively grafted onto the teacher network, it targets the few-shot scenario and uses only a few unlabeled samples to achieve comparable results on CIFAR-10 and CIFAR-100. (Xu et al., 2020) gradually increased the probability of replacing each teacher module with their corresponding student module and trained the student to reproduce the behavior of the teacher. However, the performance on Transformer-based models of the aforementioned first method is unknown while the second method has an obvious performance drop with a low sparsity (50%). The teacher model and the grafted model (shown in Figure 3) are denoted as f S and f G , respectively. Both models have N + 1 layers (i.e., the first N layers are encoder layers, and the ( N + 1) -th layer is the output layer). Denote f T i (  ) , f G i (  ) as the behaviour function induced from the i -th encoder of the teacher model, and the grafted model, respectively. As shown in Figure 4, we utilize layer-wise knowledge distillation (KD), where we aim to bridge the gap between f T i (  ) and f G i (  ) . The grafted model is trained to mimic the behavior of the teacher model. where X denotes the training dataset,  i is coefficient of i -th layer loss, LD is the distillation loss of the layer pair, x i is the input of the i -th layer. During KD, each student module mimics the behavior of the corresponding teacher layer. Similar to (Jiao et al., 2020), we take the advantage 192 Teacher encoder 1 Teacher encoder 2 Teacher encoder 3 Teacher encoder N  Input Input Embedding Output Layer Output Teacher model  ! of abundant knowledge in self-attention distribution, hidden states of each Transformer layer, and the final output layer's soft logits of teacher model to help train the student model. Specifically, we design the KD loss as follows LKD = (cid:40) L hidn + L attn 1  i  N L pred i = N + 1 (2) where L hidn = MSE( H Ti , H Si ) ( 1  i  N ) indicates the difference between hidden states, L attn = MSE( A Ti , A Si ) indicates the difference between attention matrices. MSE(  ) is the mean square error loss function and i is the index of Transformer layer. L pred = -softmax( z T )  log _softmax( z S / temp ) indicates the difference of soft cross-entropy loss, where z T and z S are the soft logits of teacher and student model, respectively. T is the temperature hyper-parameter. We further reduce the number of non-zero parameters in the weight matrix while maintaining accuracy. We denote { W j } j = i j =1 as the collection of weights in the first i layers,  j as the sparsity of the j -th layer. Then, the loss function of sparse knowledge distillation becomes L = (cid:88) x X N +1 (cid:88) i =1  i LKD ( f Ti ( x , { W j } j = i j =1 ) , f Gi ( x , { W j } j = i j =1 )) s.t. sparsity ( W j )   j for j = 1 , ..., N (3) After training, we find the sparse weight matrix W  j using W  j =  S j ( W j ) for j = 1 , ..., N, (4) where  S j (  ) denotes the Euclidean projection onto the set S j = { W j | sparsity ( W j )   j } . Our pruning method is similar to finding matching subnetworks using the lottery ticket hypothesis (Frankle and Carbin, 2019; Pensia et al., 2020) methodology. We analyze the self-attention (ex-cluding activation). Some non-linear activation functions has been analyzed in (Pensia et al., 2020). Feed-forward layer. Consider a feed-forward network f ( x ) = w  x , and g ( x ) = ( (cid:80) ni =1 w i ) x . Lueker et al. (Lueker, 1998) and Pensia et al. (Pen-sia et al., 2020) show that existing a subset of w i , such that the corresponding value of g ( x ) is very close to f ( x ) . Corollary: When w  1 , ..., w  n belongs to i.i.d. uniform distribution over [-1,1], where n  C log 2  ,   min { 1 ,  } . Then, with probability at least 1 , we have  G spd  { 1 , 2 , ..., n } ,  w  [  0 . 5 , 0 . 5] , s.t (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) w  (cid:88) i  G spd w  i (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)   (5) Analysis on self-attention. Consider a model f ( x ) with only one self-attention, when the token size of input x is 1, softmax ( Q  KT  d k ) = 1 , we have Z = V , where V = w V x. Consider f G ( x ) = (cid:16)(cid:80) d i =1 w G i (cid:17) x and a pruning sparsity  , base on Corollary , when d  C log 4 / , there exists a pattern of w Gi , such that, with probability 1   ,  w  [  1 , 1] ,   i  { 0 , 1 } , s.t. (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) w  ( (cid:88) i  [1 ,d ] w Gi I (  i )) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) <  (7) where I (  i ) is the indicator to determine whether w Gi will be remained. with a self-attention, then k Base on Corollary , when d  C log 4 / , there exists a pattern of w Gi , such that, with probability 1   ,  w c i. To avoid overfitting in the training process for the sparse Transformer model, we further graft student modules (scion) onto the teacher model duplicates (rootstock). For the i -th student module, we use an independent Bernoulli random variable I (  i ) to indicate whether it will be grafted on the rootstock. To be more specific, I (  i ) has a probability of p (grafting probability) to be set as 1 (i.e., student module substitutes the corresponding teacher layer). Otherwise, the latter will keep weight matrices unchanged. Once the target pruning rate is achieved, we apply linear increasing probability to graft student modules which enable the student modules to orchestrate with each other. Different from the model compression methods that update all model parameters at once, such as TinyBERT (Jiao et al., 2020) and DistilBERT (Sanh et al., 2019), SPD only updates the student modules on the grafted model. It reduces the complexity of network optimization, which mitigates the overfitting problem and enables the student modules to learn deeper knowledge from the teacher model. The overview is described in Algorithm 1. We will further demonstrate the effectiveness of progressive student module grafting in 4.2. Datasets. We evaluate SPD on the General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2018) and report the metrics, i.e., accuracy scores for SST-2, QNLI, RTE, and WNLI, Matthews Correlation Coefficient (MCC) for CoLA, F1 scores for QQP and MRPC, Spearman correlations for STS-B. Baselines. We first use 50% sparsity (a widely adopted sparsity ratio among SOTA), and compare SPD against two types of baselines  nonprogressive and progressive. For the former, we select BERT-PKD (Sun et al., 2019), DistilBERT (Sanh et al., 2019), MiniLM (Wang et al., 2020), TinyBERT (Jiao et al., 2020), SparseBERT (Xu et al., 2021) and E.T. (Chen et al., 2021), while for the latter, we choose Theseus (Xu et al., 2020). We further compare SPD against other existing works under higher sparsity, e.g., TinyBERT (Jiao et al., 2020), SparseBERT (Xu et al., 2021) and RPP (Guo et al., 2019). SPD Settings. We use official BERTBASE , uncased model as the pre-train model and the fine-tuned pre-train model as our teacher. Both BERTBASE and teacher model have the same architecture (i.e., 12 encoder layers (L = 12; embedding dimension d model = 768; self-attention heads H = 12)). We finetune BERTBASE using best performance from { 2 e  5 , 3 e  5 , 4 e  5 , 5 e  5 } as the learning rate. For SPD model training, the number of pruning epochs, linear increasing module grafting epochs, finetuning epochs vary from [10, 30], [5, 194 20], [5, 10], respectively. For pruning, we use AdamW (Loshchilov and Hutter, 2018) as the optimizer and run the experiments with an initial grafting probability from {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9}. The probability with the best performance will be adopted. After pruning, we adjust the slope of the grafting probability curve so that the grafting probability equals 1 at the end of module grafting. For module grafting and finetuning, an AdamW optimizer is used with learning rate chosen from { 3 e  5 , 1 e  4 , 3 . 2 e  4 , 5 e  4 , 6 . 4 e  4 }. The model training and evaluation are performed with CUDA 11.1 on Quadro RTX6000 GPU and Intel(R) Xeon(R) Gold 6244 @ 3.60GHz CPU. Accuracy vs. Sparsity. We do experiments on eight GLUE benchmark tasks (Table 1). For non-progressive baselines, SPD exceeds all of them on QNLI, SST-2, CoLA, STS-B, and MRPC. For RTE, TinyBERT 6 has a 1.6% higher accuracy than SPD. However, TinyBERT 6 used augmented data while SPD does not use data augmentation to generate the results in Table 1. On average, SPD has 6.3%, 5.6%, 1.2%, 1.7%, 3.7% improvement in performance than BERT 6 -PKD, DistilBERT, TinyBERT 6 , SparseBERT, E.T. respectively. Furthermore, on CoLA, SPA achieves up to 25.9% higher performance compared to all nonprogressive baselines. For the progressive baseline, we compare SPD with BERT-of-Theseus. Experimental results show that SPD exceeds the latter on all tasks. SPD has a 3.9% increase on average. Among all the tasks, CoLA and RTE have 20.2% and 5.9% gain respectively. For the comparison with sparse and non-progressive baseline, SPD has an improvement of 16.8%, 5.5%, 3.2%, 2.7%, 2.0%, 1.9%, 1.6%, 1.6% on CoLA, RTE, MNLI, QNLI, QQP, MRPC, STS-B, SST-2, respectively. On all listed tasks, SPD even outperforms the teacher model except for RTE. On RTE, SPD retains exactly the full accuracy of the teacher model. On average, the proposed SPD achieves a 1.1% higher accuracy/score than the teacher model. We conclude the reason for the outstanding performance from three respects: 1) There is redundancy in the original dense BERT model. Thus, pruning the model with a low pruning rate (e.g., 50%) will not lead to a significant performance drop. 2) SPD decreases the overfitting risk which helps the student model learn better. 3) The interference between different hyperparameter strategies is mitigated, which enables SPD to obtain a better student model. We also compare SPD with other baselines (i.e., 4-layer TinyBERT (Jiao et al., 2020), RPP (Guo et al., 2019), and SparseBERT (Xu et al., 2021)) under higher pruning rates. Results are summarized in Table 2. For the fairness of comparison, we remove data augmentation from the above methods. We mainly compare the aforementioned baselines with very high sparsity (e.g., 90%, 95%) SPD. For the comparison with TinyBERT 4 , both SPD ( 90% sparsity) and SPD ( 95% sparsity) win. SPD ( 90% sparsity) has 63.4% and 9% higher evaluation score than TinyBERT 4 on CoLA and MRPC, respectively. For the setting of 95% sparsity, SPD outperforms TinyBERT 4 with 41 . 3% and 7 . 6% higher performance, respectively. Compared to RPP, both SPD ( 90% sparsity) and SPD ( 95% sparsity) show higher performance on MRPC, with 9 . 8% and 8 . 3% higher F1 score, respectively. For SparseBERT, SPD exceeds it on all tasks in Table 2. Especially on CoLA, SPD ( 90% sparsity) and SPD ( 95% sparsity) have 2.69  and 2.33  higher Mcc score on CoLA, respectively. SparseBERT has competitive performance with SOTA when using data augmentation. The reason for the performance drop for SparseBERT may because its deficiency of ability in mitigating overfitting problems. Overfitting Mitigation. We explore the effectiveness of SPD to mitigate the overfitting problem. Depending on whether progressive, grafting, or KD is used, we compare 4 strategies: (a) no progressive, no KD; (b) progressive, no KD; (c) no progressive, KD; (d) progressive, KD (ours). We evaluate these strategies on both training and validation sets of MRPC. The results are summarized in Figure 5. From (a) to (d), the gap between the evaluation results of the training set and the dev set is reduced, which strongly suggests that the strategy adopted by SPD, i.e., progressive + KD, outperforms other strategies in mitigating the overfitting problem. Figure 5 (a), (b), and (c) indicate that compared to progressive only, KD has a bigger im-pact on mitigating overfitting, as the performance gap between the training set and the dev set decreases more from (a) to (c) than from (a) to (b). From Figure 5 (a), (b) and (c), we also observe that compared to no progressive, no KD, either using progressive (Figure 5 (b)) or KD (Figure 5 (c)) is very obvious to help mitigate the overfitting prob-195 Model #Param MNLI QQP QNLI SST-2 CoLA STS-B MRPC RTE Avg. lem. Figures 5 (b), (c) and (d) indicate that the combination of progressive and KD brings more benefits than only using progressive or KD as Figure 5 (d) has the smallest performance gap between the training set and the dev set. Combined with Table 1 and Table 2, Figure 5 shows that SPD mitigates overfitting and leads to higher performance. In this section, we justify the three schedulers used in our method (i.e., grafting probability, pruning rate, and learning rate), and study the sensitivity of our method with respect to each of them. Study on Components of SPD. The proposed SPD consists of three components (i.e., sparse, knowledge distillation, and progressive module grafting). We conduct experiments to study the importance of each component on GLUE benchmark tasks with the sparsity of 50% and results are shown in Table 3. Compared to both sparse + KD and sparse + progressive, SPD achieves gains on performance among all tasks. Effects of Grafting Probability Strategy. In our method, we set the grafting probability greater than 0 during pruning, to allow student modules to learn deeper knowledge from the teacher model. To verify the benefit of this design, we change the grafting probability to zero and compare it with our 196 Model #Param MNLI QQP QNLI SST-2 CoLA STS-B MRPC RTE Avg. method. The result on RTE is shown in Figure 6. Pruning with grafting (the red curve) shows better performance than pruning without grafting, which justifies the existence of grafting during pruning. In addition, we study the sensitivity of our method to grafting probability (Figure 7). It is observed that p 0 = 0.6 achieves the best performance, and the progressive design is better than the non-progressive. rate scheduler, we compare the strategies with different pruning ending steps. The results are shown in Figure 8. It is observed that the pruning during when grafting probability p = p 0 has a higher F1 score than other strategies on MRPC. Effects of Optimizer Strategy. We also compare our strategy with the strategy that only has one learning rate scheduler. The results (Figure 9) indicate that our strategy (i.e., two independent optimizers) is better. We also evaluate different learning rates with the pruning rate of 0.9 and the grafting probability of 0.8. In this paper, we postulate a counter-traditional hypothesis that pruning increases the risk of overfitting under the pretrain-and-finetune paradigm. We analyze and empirically verify this hypothesis, and propose a sparse progressive pruning method 197 to address the overfitting problem. We theoretically analyze that our pruning method can obtain a subnetwork from the student model that has a similar accuracy as the teacher. We study and minimize the interference between different hyperparameter strategies, including pruning rate, learning rate, and grafting probability. A number of ablation studies and experimental results on eight tasks from the GLUE benchmark demonstrate the superiority of our method over the leading competitors. This research was supported in part by National Science Foundation (NSF) CRII Award No. 2000722 and NSF CAREER Award No. 2046102. Sanguthevar Rajasekaran has been supported in part by the NSF RAISE Award No. 1743418 and NSF EAGER Award No. 1843025. In addition, it used the Extreme Science and Engineering Discovery Environment (XSEDE) through allocations TG-CCR200004.
abstain method objective objective result abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain result abstain abstain abstain result method abstain abstain abstain abstain abstain abstain abstain objective method abstain objective objective result objective objective result result objective objective abstain other abstain abstain other other other other other other other other other other other other other abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain method abstain other abstain method abstain abstain abstain abstain abstain abstain method other abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain method method method abstain method abstain method abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain objective result method objective other other other