Behavior of deep neural networks can be inconsistent between different versions. Regressions 1 during model update are a common cause of concern that often over-weigh the benefits in accuracy or efficiency gain. This work focuses on quantifying, reducing and analyzing regression errors in the NLP model updates. Using negative flip rate as regression measure, we show that regression has a prevalent presence across tasks in the GLUE benchmark. We formulate the regression-free model updates into a constrained optimization problem, and further reduce it into a relaxed form which can be approximately optimized through knowledge distillation training method. We empirically analyze how model ensemble reduces regression. Finally, we conduct CHECKLIST behavioral testing to understand the distribution of regressions across linguistic phenomena, and the efficacy of ensemble and distillation methods. Regression-free model update is a desirable system property which guarantees interoperability of a new system with a legacy version, also known as backward compatibility. Regression occurs when the newly updated system stops functioning as intended. As advances in deep learning spark industrial applications in AI areas such as natural language processing, the long-term maintenance of such systems is becoming ever more challenging. While models with complex neural architectures and huge parameter space continue to reach higher accuracy, the lack of interpretability and functional decomposibility in these models make it infeasible to apply traditional software regression testing  Work done while at Amazon AWS AI. methods such as unit tests. As result, validating and mitigating regressions during model update is often a long and painful engineering process, which often over-shadows the benefits of a new model. The model regression issue in deep learning first comes into sight in Shen et al. (2020), where they inspect compatible representation learning for image retrieval. Yan et al. (2020) proposed the positive-congruent training (PCT) for image classification that minimizes prediction errors and model regression at the same time. To our best knowledge, the model update regression has not been studied on NLP tasks. Following Yan et al. (2020), in this work we measure the model update regression in NLP by negative flips . In Figure 1, we demonstrate prediction flip scenarios. Negative flips are shown in the upper-right quadrant where the old model makes correct predictions and the new model predictions are wrong. As we will show in Section 2, regression are prevalent in NLU model updates even with the slightest changes in the new model training process. To develop a model with minimum regression, we first formulate the learning task into a constrained optimization problem by taking the regression-free conditions as constraints. We apply the Lagrangian relaxation to bring the regression-free constraint into the optimization objective as an additional penalty loss, and provide approximate solution via knowledge distillation. Yan et al. (2020) also observed that model ensemble can also reduce negative flips without explicit input from the old model. We evaluate both distillation and ensemble based methods on a diverse set of NLP tasks. To further understand how the above methods contribute to reducing it, we utilize CHECKLIST (Ribeiro et al., 2020) to quantify linguistic behavioral changes before and after applying proposed methods. We find that regressions are prevalent in NLP tasks, and their distribution correlates with different linguistic phenomena. Our main contributions are as follows:  We provide empirical evidence to show that the model update regression occurs across text classification tasks in NLP;  We formulate the regression-free model updates into a constrained optimization problem, and further reduce into a relaxed form which can be approximately optimized through knowledge distillation training method;  We also explore the model ensemble as another method to reduce regression, and analyzed its efficacy;  We analyze the source of the regressions in NLP tasks through linguistic behavioural testing, compare reduction in both distillation and ensemble methods. In this section, we first formulate the measure of model update regression on classification tasks. Then we benchmark on GLUE tasks (Wang et al., 2018) and show that there is a prevalent presence of regression when updating models in NLP. Similar to software regression testing, we need to collect a group of test cases when measuring regression. We start from a regression set: D reg = { x i , y i } Ni =1 , y i  { l 1 , l 2 , ..., l C } , where l i is the i -th label and C is the number of classes. In practice, we can use the development set or compile a collection of critical use cases as D reg . In a classification task, given a input x i , a neural network model f , parameterized by  , approximates the posterior probabilistic distribution p ( y i | x i ) over all possible labels: (cid:126)f  ( x i ) = ( p  ( y = l 1 | x i ) , ..., p  ( y = l C | x i )) (cid:62) . To simplify, we denote the final prediction of a model to be f  ( x ) = arg max l j p  ( l j | x ) . The regression RNF between two models f  old and f  new on D reg can be defined as the portion of negative flip cases: RNF ( D reg , (cid:126)f  old , (cid:126)f  new ) = |{ x | f  old = y, f  new (cid:54) = y }| |D reg | . We use negative flip RNF as our regression measurement for classification tasks. Lower RNF for a new models means better compatibility with the old model. The success of Transformer (Vaswani et al., 2017) and BERT (Devlin et al., 2019) have made pretraining then fine-tuning a standard paradigm in NLP systems. When updating these systems, differences can come from various aspects:  Changes in the fine-tuning hyperparameters (e.g. random seed, learning rate schedule, epoch, etc.)  Changes in model size or architecture (e.g. from BERT base to BERT large )  Changes in pre-training procedure or objective (e.g. BERT to ROBERTA (Liu et al., 2020), to BERT whole-word-masking or to ELECTRA (Clark et al., 2020))  Changes in pre-trained model architecture (e.g. BERT to ALBERT (Lan et al., 2020)) While accuracy or efficient improvements are strong motivations for these model updates, they could also introduce behavioral incongruence when compared to the previous model. To benchmark the severity of regression, we apply a general setup: Fine-tune various pre-trained language models (LM) on GLUE and calculate RNF when updating from BERT base to other LMs . We use dev sets as D reg . Results in Table 1 show that: 1. Model update regression is prevalent on NLU tasks. A minimum of 1 . 98% RNF is observed across diverse classification tasks and model update scenarios, while the average accuracy gain is only 1 . 4% . 2. Minor changes such as random seeds can introduce significant regression. Shown in CoLA MRPC QQP MNLI-m Train size 8.6k 3.7k 360k 390k Dev size 1k 0.4k 40k 9.8k Old: BERT base Acc 82.84% 86.03% 90.76% 83.82%  BERT base Acc 83.80%(+0.96%) 86.03%(-0.00%) 90.56%(-0.20%) 83.55%(-0.27%) RNF  (cid:45) 3.36%  (cid:45) 4.17%  (cid:45) 2.32%  (cid:45) 3.56%  BERT large Acc 85.43%(+2.59%) 87.75%(+1.72%) 91.11%(+0.35%) 86.10%(+1.97%) RNF  (cid:45) 3.16%  (cid:45) 5.88%  (cid:45) 2.82%  (cid:45) 3.95%  ROBERTA base Acc 84.85%(+2.01%) 89.22%(+3.19%) 91.25%(+0.52%) 87.58% (+4.42%) RNF  (cid:45) 4.67%  (cid:45) 4.66%  (cid:45) 1.98%  (cid:45) 2.64%  ELECTRA base Acc 85.81%(+2.97%) 86.03%(+0.00%) 91.35%(+0.59%) 88.87%(5.05%) RNF  (cid:45) 5.18%  (cid:45) 5.39%  (cid:45) 3.20%  (cid:45) 3.50%  ALBERT base Acc 76.51%(-6.33%) 86.27%(+0.24%) 90.73%(-0.03%) 85.26%(+1.44%) RNF  (cid:45) 10.74%  (cid:45) 6.86%  (cid:45) 3.78%  (cid:45) 5.22% Table 1: Regression measurement when performing different model updates on GLUE benchmark. 56% negative flip. 3. Negative flip rates are often much higher than the accuracy gains. When updating to BERT large on QQP, RNF is about 8 X the accuracy gain. This implies reducing error rate alone does not ensure the decrease in regression. 4. Pre-training objective or architecture updates often lead to higher regressions than those caused by model size or random seeds. The regressions are higher when updating to ALBERTA, compared with updating to a larger model BERT large or a different random seed. This implies systematic regression could be introduced if the backbone models are different. In this section, we first formulate regression-free model update as a constrained optimization problem, then further reducing it to a joint optimization objective combining the training loss on the original task and a distillation loss with respect to the old model's behavior. Unlike typical optimizations in neural model training where we minimizes a loss function on a training set, the regression-free model update requires the model to learn the target task as well as comply with conditions posed by the old model. 2 Full results on GLUE can be found in Appendix A We can cast the regression-free model update as a constrained optimization problem by writing down the classification loss as the optimization objective and the regression-free conditions as constraints: min  new (cid:88) x D train LCE ( x,  new ) s.t. RNF ( f  old , f  new , D reg ) = 0 . where D train , D reg represent the training and regression sets, respectively. The constraint in Equation 1 asks for zero regression on D reg . It would be difficult to ensure the constraint is satisfied along the model training. We instead relax the hard constraint into a soft inequality condition that allows the regression measure to be less than a constant C : min  new (cid:88) x D train LCE ( x,  new ) s.t. C  RNF ( f  old , f  new , D reg )  0 . Training a model directly with the regression-free constraint still remains difficult in that signals from old predictions are sparse and RNF is nondifferentiable. Here, we propose two proxies of RNF to measure regression in continuous space. Proxy from Prediction Probabilities. We use the KL divergence between the predicted probabilities of both models as one soft regression measure: R KL-div ( f  old , f  new , D reg ) = (cid:88) x D reg DKL ( p  old ( y | x ) || p  new ( y | x )) . (3) Proxy from Deep Representations. We can also use the l 2 distance between models' sentence representations, e.g. [CLS] embedding in BERT as another soft regression measure: R l 2 ( f  old , f  new , D reg ) = (cid:88) x D reg l 2 ( (cid:126)f  old ( x ) , (cid:126)f  new ( x )) . (4) A linear projection is used to align the representations if they initially lie in different spaces. Reduce to Knowledge Distillation. Finally, we apply the Lagrangian relaxation to bring the regression-free constraint into the optimization objective as an additional penalty loss: min  new (cid:88) x D train LCE ( x,  new )    ( C  R soft ( f  old , f  new , D reg )) , (5) where  is a positive penalty scaling parameter and R soft can be chosen from R KL-div or R l 2 . Then, the above optimization problem can be cast into a joint learning of the original target task and knowledge distillation from the old model. The distillation loss acts as a surrogate of the model update regression measure. The joint learning process minimizes this term as an approximation of minimizing the overall model update regression. Since we usually update models from elementary ones to improved ones, in the experiments we take origin BERT base (12-layer, 768-hidden, 12-heads, 110M parameters) (Devlin et al., 2019) as the old model's backbone and update it to a homogeneous model, e.g. BERT base with different fine-tuning random seeds or parameters, or a heterogeneous models with improvements such as BERT large (24-layer, 1024-hidden, 16-heads, 340M parameters). We fine-tune the pre-trained LMs without any constraint as our baselines. We use the GLUE datasets to benchmark the effectiveness of proposed techniques. Details of each GLUE task can be found in Appendix D. For investigative experiments, we use the Microsoft Research Paraphrase Corpus (MRPC) (Dolan and Brockett, 2005), a paraphrase identification dataset that aims to classify whether two sentences are the paraphrase of each other. Pre-trained model artifacts and the GLUE dataset processing procedures are brought from Hugging Face 3 and experiments are done in PyTorch (Paszke et al., 2019) with Tesla V100 GPUs. Cross-entropy is used for fine-tuning on target tasks with batch size 16 for 4 to 6 epochs. The learning rate is searched among 2 e  5 , 3 e  5 and 5 e  5 . During joint training of classification and knowledge distillation, we take the fine-tuned old models as the teacher, and distill with batch size 16 for 6 to 8 epochs. We set D reg = D train when training models with the constraint and use D reg = D dev for reporting results. To encourage constraint satisfaction and reduce regression, we only include the distillation penalty into our loss on the examples where the current model makes negative flips. Yan et al. (2020) reported an intriguing finding on image classification tasks that model ensemble can reduce model update regressions without explicit regularization from the old model. This was attributed to the reduction of variance in ensemble model predictions, making it less prone to over-fitting and indirectly reducing regressions. Here we include model ensemble as an alternative approach to reduce regression, with further analysis on how ensemble reduces regression in Section 5.1. Table 2 shows the efficacy of distillation method and model ensemble on reducing NLP classification task model update regressions. On average, the distillation method reduces RNF by 30 . 6% and 36 . 3% while the ensemble method reduces RNF by 55 . 9% and 20 . 6% when updating to BERT base and to BERT large , respectively. Both distillation and ensemble methods can significantly bring down negative flips across GLUE tasks compared with the baselines. The ensemble seems to work better when the old and new models share the same underlying pre-trained LM. In the update BERT base  BERT base , the ensemble method outperforms the distillation on reducing the regression. On the other hand, the distillation method seems to be more effective on reducing regression under the heterogeneous model update setting. In the update BERT base  BERT large , distillation reduce more regression, with especially 3 https://huggingface.co CoLA SST-2 MRPC QQP MNLI-m QNLI RTE Average Train size 8.6k 67k 3.7k 360k 390k 100k 2.5k Dev size 1k 0.9k 0.4k 40k 9.8k 5.5k 0.3k Old: BERT base Acc 82.26% 91.17% 86.03% 90.76% 83.82% 91.07% 67.15% 84.61%  BERT base Baseline Acc 82.93% 91.63% 86.03% 90.56% 83.55% 90.65% 63.18% 84.08% RNF 4.41% 1.95% 4.17% 2.32% 3.56% 2.35% 11.43% 4.31%  BERT base Distillation Acc 84.47 % 92.09 % 87.01 % 91.14 % 83.77% 91.16% 68.95% 85.81 % RNF 1.92 % 0.80 % 1.72% 1.69% 4.32% 2.47% 8.30% 2.99%  BERT base Ensemble Acc 82.17% 91.63% 86.03% 91.06% 84.35 % 91.62 % 70.76 % 85.37% RNF 2.59% 0.92% 1.23 % 1.18 % 1.66 % 1.06 % 4.69 % 1.90 %  BERT large Baseline Acc 85.62% 92.89% 87.75% 91.11% 86.10% 92.53% 66.43% 86.06% RNF 2.68% 1.72% 5.88% 2.82% 3.95% 2.64% 12.27% 4.57%  BERT large Distillation Acc 85.62 % 92.89 % 88.73% 91.50% 86.73 % 92.15% 73.65 % 87.33 % RNF 2.49 % 1.26 % 2.45% 2.46% 3.76% 2.54 % 5.42 % 2.91 %  BERT large Ensemble Acc 84.95% 93.12 % 89.46 % 91.66 % 87.05 % 93.08 % 67.87% 86.74% RNF 2.78% 1.61% 2.45% 2.20 % 3.24 % 2.27 % 10.83% 3.63% Table 2: Results of fine-tuning with distillation and ensemble on GLUE benchmark. large reductions on small datasets such as CoLA and SST-2. We hypothesize that it's because the ensemble focuses on reducing the variance in model predictions, while distillation enables the explicit alignment in either probability distribution or representation space between the old and the new model. When the new model is very different from the old one, it can implicitly align new model's behavior with the old one. As introduced in Section 3, we can have several variants of distillation loss to be used to constrain new model training on the old model. We explore and benchmark the following variants on the MRPC task:  Distillation -R KL-div , Logits calculates the distillation loss as the KL divergence between the two Bernoulli distributions set by the old and new model prediction probabilities;  Distillation -R l 2 , [CLS] uses the [CLS] token embedding from the final layer as sentence representations and calculates the distillation loss as the Euclidean distance between the two vectors;  Distillation R l 2 , All [CLS] also calculates the Euclidean distance between the old and new sentence representation vectors, but with concatenated [CLS] token embeddings from all layers instead of the final layer. Pre-trained models could have different layers. For BERT base  BERT large in the All [CLS] setup, we align representations from BERT large 's even layers with the corresponding BERT base layers, e.g. 14 -th layer in BERT large is aligned with 7 -th in BERT base . Table 3 shows the results. In the homogeneous setup, the most effective variant is to align the prediction probabilities via R KL-div , where it achieves up to 58% RNF reduction, i.e. from 4 . 17% to 1 . 72% . For R l 2 setup, aligning at all layers can further reduce RNF compared with only aligning at the final layer. This implies a deeper alignment can help the new model more effectively learn to behave similarly as the old one when fine-tuning the same architecture with a different random seed. In the heterogeneous setup, R l 2 , [CLS] works the best for BERT large that achieves 62% RNF reduction, with R KL-div having a comparable performance. Overall, R KL-div produces consistent regression reductions across different setups, which we pick it as our default setting in the distillation method. From Table 3, we can also observe that the deeper alignment seems to hurt RNF in the heterogeneous update setup. The reason might be that differences between pre-trained models are too significant. The distillation with simple all-layer alignment could mess up pre-trained representations rather than effectively encourage Old: BERT base Acc : 86.03% New:  BERT base  BERT large Acc RNF Acc RNF Baseline 86.03% 4.17% 87.75% 5.88% Distillation R KL-div , Logits 87.01 % 1.72% 88.73% 2.45% Distillation R l 2 , [CLS] 85.54% 3.19% 88.73% 2.21 % Distillation R l 2 , All [CLS] 85.54% 2.45% 87.99% 4.90% Ensemble 86.0% 1.23 % 89.46 % 2.45% Table 3: Accuracy and regression results on MRPC with BERT base  BERT base and BERT base  BERT large updates using variants of distillation and ensemble methods. well. Another interesting finding is that the simple model ensemble is a competitive solution comparing to the distillation. In the BERT base  BERT base setup, the ensemble even outperforms all the other distillation variants. This is indeed a bit counter-intuitive as the distillation explicitly encourages the new model to pick up old models' correct predictions while the ensemble does not involve the old model in the process. We conduct deeper analysis trying to understand on which aspects that these methods work to reduce the regression in the next section. In this section, we first analyze the model ensemble and present our hypothesis on how it reduces regression. Next, we conduct behavioral testing across diverse linguistic phenomena to see where the reduced and remaining regressions reside. Similar to the findings of Yan et al. (2020), we observe in Table 2 and 3 that a simple ensemble of models trained with different random initialization before finetuning can reduce regression in some cases. We fine-tune BERT base on MRPC with 20 random seeds as our old base models, and another 20 seeds as our new single models, and another 100 seeds for building 20 ensemble models. Next, we calculate RNF on the dev set in each model update setup, i.e. 400 update pairs. Figure 2 plots their model update regression RNF distributions. We observe that the ensemble can not only bring down RNF but also reduce its variance. From Figure 2, we conjecture that each single 1 2 3 4 5 6 7 8 0 10 20 30 40 50 60 RNF (%) C oun t s ( T o t a l 400 ) Single,Average RNF = 4 . model could learn a subset of all possible patterns in the data to achieve comparable accuracy on the task. Models fine-tuned with different seeds could rely on different sets of patterns for predictions, leading to behavioral difference and regression. On the other hand, ensemble aggregates distinct and complementary behaviors from individual models, leading to less eccentric behavior and increased compatible with individual models on average. In a parallel work, Zeyuan and Li (2020) provides a theoretical framework of how ensemble works from the multi-view perspective. They show that single models can pick up multiple but different views of the data, and the ensemble naturally collects more view features, leading to a higher accuracy. Our hypothesis concurs with their findings. However, ensemble is not required to achieve moderate model behavior. To verify this, we designed the following simple model selection Acc RNF Old: BERT base Single 85.81(  1.07)% Single 85.39(  1.43)% 4.30(  1.53)%  BERT base Ensemble 86.18(  1.12)% 3.08(  1.14)% Centric 87.75% 2.79(  0.69)% Single 86.32(  2.50)% 5.37(  2.69)%  BERT large Ensemble 87.65(  1.34)% 3.64(  1.21)% Centric 87.25% 4.24(  0.78)% Table 4: The selected single model centric can achieve similar accuracy and negative flip as ensemble. procedure. We first train 20 new single models, among which we compute for each model the average RNF on the first half of dev set when updating from the other 19 models. We then select the single model with the lowest average RNF as the centric . Results in Table 4 show the accuracy and RNF on the second half of the dev set. Indeed the single centric model achieves substantial reduction in RNF comparable to model ensemble. We further plot all the BERT base models based on their class predictions down-projected by PCA (Hotelling, 1933). Figure 3 shows that single models tend to spread while ensembles are more concentrated and close together. We can also see that the centric indeed sits near the center of single model cluster. In essence, the centric model is a single model that requires much less compute resource than the ensemble model during inference, yet can achieve comparable performance and reductions in regression. To further understand where the regression happens and how the above methods contribute to reducing regression, we conduct qualitative analysis across diverse linguistic phenomena. More precisely, we leverage the CHECKLIST (Ribeiro et al., 2020) behavioral testing and construct regression sets for relevant linguistic capabilities and tests based on perturbations and provided templates. For example, to test the capability of dealing with lexical taxonomy in the paraphrase detection task, we replace adjectives in one sentence with their synonyms with the label unchanged and expect the model can still predict correctly. We manually set the templates, apply CHECKLIST to automatically generate testing sentence pairs, and calculate RNF for each linguistic test. Detailed linguistic behavioral testing setups with examples can be found in Appendix C. Table 5 shows the linguistic behavioral testing results when updating from BERT base 1 Seed to BERT large . Each row denotes one specific behavioral test and 500 cases are sampled in each test. We focus on negative flips where the new model fails the test while the old model passes. We can observe that the vanilla fine-tuned BERT large has significant regressions on switching with synonyms, asymmetric ordering, and active-passive swap related to people names (see Appendix C). Also, we observe that models tend to either fail or pass almost all cases in a test, which leads to high variances in RNF . This implies that models fine-tuned with different seeds can have different behavioral patterns, which could be one source of regression. Furthermore, Table 5 shows that the distillation can effectively reduce regressions across almost all types of behavioral tests. This demonstrates that minimizing the surrogate regression measure, formulated as a knowledge distillation objective, reduces the regression through actually aligning new model's behavior with the old model. For the ensemble, although it can reduce significant regressions in the benchmark, we observe that it can only improve the model update compatibility on a handful of capabilities. We hypothesize that the ensemble mostly improves the compatibility with the underlying constituent models. Without an explicit alignment, it cannot proactively reduce the regression on certain behavior tests when updating Old: BERT base (Error %) New: BERT large ( RNF ) 1 Seed 1 Seed KD 1 Seed 5 Seeds KD 5 Seeds Centric Ensemble Coref He/She 0.0% 13.4% 22.4% 12.3(  16.1)% 41.2(  47.0)% 0.0 0.0(  0.0)% Vocab People 0.0% 13.8% 1.9 % 59.5(  34.8)% 34.1 (  47.1)% 89.0% 55.2(  50.8)% Vocab More/Less 100.0% 0.0% 0.0% 0.0(  0.0)% 0.0(  0.0)% 0.0% 0.0(  0.0)% Taxonomy Synonym 0.0% 42.3% 0.0 % 77.0(  26.2)% 20.3 (  44.5)% 73.1% 61.9(  54.1)% SRL Pharaphrase 0.0% 12.5% 99.9% 63.2(  43.4)% 42.0 (  48.3)% 26.9% 61.9(  54.1))% SRL Asymmetric Order 0.0% 47.1% 0.0 % 70.7(  20.0)% 22.3 (  43.7)% 92.0% 58.6(  52.2)% SRL Active/Passive 1 0.1% 9.3% 65.2% 58.0(  43.2)% 52.7 (  50.1)% 98.5% 42.4(  51.5)% SRL Active/Passive 2 0.1% 90.0% 0.7 % 95.5(  4.8)% 23.8 (  43.3)% 99.6% 65.2(  56.5)% SRL Active/Passive 3 99.9% 0.1% 0.0% 0.1(  0.0)% 0.0 (  0.0)% 0.1% 0.1(  0.1)% Temporal Before/After 100.0% 0.0% 0.0% 0.0(  0.0)% 0.0(  0.0)% 0.0% 0.0% Average 38.5% 19.9% 14.8 % 38.2% 20.3 % 43.8% 33.3% Table 5: Behavioral tests with CHECKLIST . The backward compatibility representation learning first comes into sight in Shen et al. (2020) on learning inter-operabile visual embeddings for image retrieval tasks. Later, Yan et al. (2020) formalize the model update regression problem in machine learning and explore solutions on image classification tasks. They suggest negative flip (NF) as the empirical measurement of regression and propose a specialized knowledge distillation loss (Hinton et al., 2015) as a surrogate of regression for joint optimizations. Our work investigates the model update regression in NLP classification tasks, which involve discrete signals and rich linguistic structures. We formulate our solutions from the perspective of constraint satisfaction and verify their efficacy on scenarios including update to distinct architectures. Pre-training a model on large corpora and finetuning on downstream tasks has emerged as a standard paradigm in NLP (Devlin et al., 2019; Lan et al., 2020; CONNEAU and Lample, 2019; Raffel et al., 2020; Brown et al., 2020; Clark et al., 2020). Our work follows this transfer learning paradigm but our main focus is to investigate the regression phenomenon when updating backbone pre-trained models. Another related stream of research is lifelong learning (Lopez-Paz and Ranzato, 2017; Yoon et al., 2018; Delange et al., 2021; Sun et al., 2019; Chuang et al., 2020), incremental learning (Rebuffi et al., 2017; Chaudhry et al., 2018; Prabhu et al., 2020), or concept drifting (Schlimmer and Granger, 1986; Tsymbal, 2004; Klinkenberg, 2005; liobaite I., 2016) which aims to accumulate knowledge learned either in previous tasks or from data with changing distribution. The model update regression problem differs in that models are trained on the same task and dataset, but we update from one model to another. To analyze whether a fine-tuned model can handle linguistic phenomena for a specific end task, perturbation techniques are often used (Belinkov and Bisk, 2018; Ribeiro et al., 2018; Prabhakaran et al., 2019; Wu et al., 2019; Talmor et al., 2020). In particular, CHECKLIST (Ribeiro et al., 2020) leverages and expands those techniques to efficiently evaluate a wide range of linguistic behavioral capabilities of NLP models. Our work applies CHECKLIST to inspect where the model update regressions come from and on which linguistic phenomena our proposed solutions help to reduce regressions. In this work, we investigated the regression in NLP model updates on classification tasks and show that it has a prevalent presence across tasks and models. We formulated the regression-free model update problem as a constrained optimization problem and reduce it into a joint learning objective on target task while distilling from the old model. Together with the ensemble, these methods can cut down the regression by 60% at best. Experiments on the GLUE benchmark showed that ensemble can be effective in reducing the regression when updating to homogeneous models. On the other hand, knowledge distillation produced more significant regression reductions under the heterogeneous setting. Through linguistic behavioral testing we showed that distillation can reduce the regression across a wider range of linguistic phenomena than ensemble method. While the regression reduction achieved by the discussed methods are promising, they are far from reaching regression-free. We leave the design of more advanced regression-reduction methods as future works. The authors would like to acknowledge the AWS AI team for inspiring discussions, honest feedback, and full support. We are also very grateful to the reviewers for insightful comments and helpful suggestions.
abstain abstain method result method method method abstain abstain abstain abstain abstain abstain abstain abstain method method objective abstain objective objective objective abstain method abstain result other objective result method abstain method abstain abstain abstain method abstain abstain other method method abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain other abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other objective abstain abstain abstain other abstain other other objective objective objective abstain abstain abstain result abstain abstain other other