Real-world natural language processing (NLP) models need to be continually updated to fix the prediction errors in out-of-distribution (OOD) data streams while overcoming catastrophic forgetting. However, existing continual learning (CL) problem setups cannot cover such a realistic and complex scenario. In response to this, we propose a new CL problem formulation dubbed continual model refinement (CMR). Compared to prior CL settings, CMR is more practical and introduces unique challenges (boundary-agnostic and non-stationary distribution shift, diverse mixtures of multiple OOD data clusters, error-centric streams, etc.). We extend several existing CL approaches to the CMR setting and evaluate them extensively. For benchmarking and analysis, we propose a general sampling algorithm to obtain dynamic OOD data streams with controllable non-stationarity, as well as a suite of metrics measuring various aspects of online performance. Our experiments and detailed analysis reveal the promise and challenges of the CMR problem, supporting that studying CMR in dynamic OOD streams can benefit the longevity of deployed NLP models in production. 1 1 Introduction Fine-tuning large pre-trained language models (LMs) has become the de facto standard for training models of a variety of tasks in natural language processing (NLP). These success stories are usually in places where the training and testing data are drawn from the same distribution. However, in real-world scenarios, a deployed model (e.g., a question answering service) often encounters examples that are out of the training distribution (i.e., out-of-distribution, OOD ). Such distribution shift often leads to a high error rate. In practice, it is highly  The work was done when Bill was an intern at FAIR. preferred to continually refine deployed models whenever new errors are reported and annotated, in order to reduce their further negative impacts. In spite of its importance, the challenge of continually refining a model over OOD data streams has been underexplored. Prior work in continual learning (CL) has primarily focused on task-incremental settings with boundary-aware data streams. These CL methods are usually evaluated on simple models and data (e.g., image classification with MNIST) (Aljundi et al., 2019). It is not clear to what extent they can efficiently refine a model in boundary-agnostic streams for a complex language task (e.g., reading comprehension) with modern LMs. In addition, there is no existing evaluation protocol for comprehensively comparing the collection of applicable methods for such a practical and complex problem. Traditional CL paradigms mainly focus on incrementally learning a model from a data stream with a sequence of distinct tasks with explicit delineation, which is rather unrealistic in real-world NLP applications. To address these research questions, we propose a novel CL formulation named continual model refinement ( CMR ), which aims to efficiently update a model for error correction in an out-of-distribution data stream without catastrophically forgetting its acquired knowledge over time. In contrast to prior CL setups, CMR targets learning a model of a particular task (e.g., question answering) from its prediction errors in dynamic OOD data streams. Instead of assuming that the streams are drawn from a fixed unseen distribution, we study CMR under a more general and realistic scenario, where the underlying distribution of OOD data streams is non-stationary across time steps without clear boundaries while being diverse at every time step. In this paper, we focus on studying whether existing methods can address CMR and how we should benchmark and analyze their performance. We first formulate the CMR problem with several ba-3128 sic metrics covering multiple desiderata for a CMR method: the ability to instantly fix known errors, the retention of previously acquire knowledge from upstream/online data, and the generalization to unseen OOD data (Sec. 2). Then, we propose a general method to create the dynamic data streams of the aforementioned characteristics and evaluation metrics to benchmark CMR methods, yielding a comprehensive evaluation protocol for CMR (Sec. 3). We employ and extend several suitable methods from the CL literature to study the CMR problem, which is based on parameter regularization or memory replay (Sec. 4). We have conducted a comprehensive analysis with extensive experimental results, which reveal many interesting, non-trivial findings (Section 5). For example, we find that even though replay methods are generally better than regularization-based methods, EWC (Kirkpatrick et al., 2017), a typical regularization method, achieves the best score in generalizing to unseen OOD data. We also find that a simple variant of ranking criteria in conditional replay methods achieves more stable results. Moreover, we find that different CMR methods have orthogonal improvements and our positive initial results suggest that integrating regularization terms for replay methods is a promising future direction to develop advanced CL methods to address CMR. In this section, we formally introduce the proposed continual learning setup, continual model refinement (CMR). We first define the notations and describe the learning objectives that are also illustrated in Fig. 1, then we design a few basic evaluation metrics for assessing CMR methods, and finally, we briefly discuss the unique challenges compared to other CL formulations. Upstream learning. Suppose that we want to build a question answering (QA) model. To do this, we usually need to offline fine-tune a large pre-trained LM with the existing QA data we have now. Formally, we denote a dataset with D = { ( x i , y i ) } , consisting of the examples are drawn from an upstream distribution U , i.e., D  U . The fine-tuned LM is named upstream model f 0 . Query streams. After the model f 0 is deployed in production, it is common to see ever-changing distribution shifts in real-world data. We use  ! { Q 1 , . . . , QT } to denote the arriving examples grouped in T episodes and call this sequence of datasets as a query stream . We discuss our method of creating such challenging query streams for evaluating CMR in Sec. 3.2 and Alg. 1. Error streams. In real-world scenarios, the size of Q t can be very large even in a short period of time, and it is unrealistic to assume that we can annotate all of them to refine the model f t  1 . A common practice is to only annotate the ones that are reported as prediction errors or bugs. Motivated by this, we use E t to denote the examples in Q t that are predicted incorrectly by f t  1 . This thus forms an evolving, dynamic stream of prediction errors { E 1 , . . . , ET } , where E t = { ( x, y )  Q t | f t  1 ( x )  = y } . Learning objectives. To improve the user satisfaction over time, we need a continual model refinement (CMR) method g that can efficiently take the model f t  1 and E t as input and then output a refined model f t for processing future examples. We expect f t to output correct answers for the known errors E t immediately while maintaining its correct predictions on previous questions that are answered correctly. We also want the refined models to keep their generalization ability to unseen future data in the stream. Sec. 2.2 shows the metrics to assess a CMR method g toward these goals. We use five metrics to describe the desiderata for CMR methods and assess them quantitatively. We show how to use these metrics for benchmarking in a comprehensive yet concise way in Sec. 3.3. Error-fixing rates (EFR). To assess the responsiveness of the error-fixing methods, we look at how many errors can be fixed right away. We define the instant error-fixing rate at time step t as: EFR ( t ) =: Acc ( f t , E t ) =: |{ ( x, y )  E t | f t ( x ) = y }| | E t | . Knowledge retention (UKR&OKR). We define two metrics below to assess how much knowledge acquired from upstream or online data streams that the model maintains over time: UKR ( t ) =: Acc( f t , D ) and OKR ( t ) =: Acc( f t , Q <t ) , where Q <t = (cid:83) t  1 i =1 Q i . We down-sample D and Q <t and compute periodically for efficiency. Cumulative success rates (CSR). To monitor the model performance on incoming query examples, we compute a running average of success rates at past time steps: CSR ( t ) =: 1  | E <t | / | Q <t | . Knowledge generalization (KG). As we only have a finite number of episodes for experiments, to assess the model performance in the future episodes, we test the models with a held-out set of test examples, H , that are drawn from the same underlying distributions which are used to create the query stream. That is, KG ( t ) =: Acc( f t , H ) . Without loss of generality, we suppose that Q t  O t , where {O t } denotes an ever-changing series of unseen distributions. Typical task-incremental CL problem setups such as LAMOL (Sun et al., 2020) and CLIF (Jin et al., 2021) consider Q t and Q t +1 are sampled from two distinct tasks. Therefore, the distribution shifts are sudden (i.e., O t and O t +1 does not share any overlapping components). Also, in conventional CL formulations, the past distribution will never be revisited, which is rather unrealistic in real-world applications. They do not have the concept of error stream either. Instead, the proposed CMR formulation is essentially a boundary-agnostic CL problem in non-stationary data streams, where the distribution shifts are more dynamic, unpredictable, and diverse, yielding a more realistic yet challenging CL setup. We provide a comprehensive evaluation protocol for studying continual model refinement in OOD streams. This section first briefly describes our selected task and datasets (Sec. 3.1), then focuses on our proposed method to sample non-stationary OOD data streams (Sec. 3.2), and finally, illustrate how we use the basic metrics to benchmark various CMR methods in a comprehensive yet concise way. In this paper, we mainly use extractive question answering (i.e., machine reading comprehension) to evaluate and analyze CMR methods, while one could also study the CMR problem in any NLP tasks with the proposed protocol. We use the MRQA-19 benchmark (Fisch et al., 2019) which consists of 6 datasets sharing the same formats. We use the SQuAD (Rajpurkar et al., 2016) as the upstream data for offline training the base LM, and use the other five parts as the OOD data for continual learning: NQ (Kwiatkowski et al., 2019), HotpotQA (Yang et al., 2018), SearchQA (Dunn et al., 2017) and TriviaQA (Trischler et al., 2017). This is because SQuAD is more commonly used for deploying models in production and the real-world QA examples from online users can be more similar to the distribution of NQ and SearchQA. Here we discuss how to create a realistic ever-changing series of distributions (i.e., {O t } in Sec. 2.3) for creating query streams { Q t } . Background. A common practice in CL to create a controllable non-stationary data stream is to control the context-switching probability. For example, OSAKA (Caccia et al., 2020), as a representative method, uses a Markov chain to sample a sequence of tasks with a constant transition probability and then sample the examples from the selected task at each time step. Despite its simplicity, this method is nevertheless limited to the cases where query stream Q t can only be drawn from a single distribution, which can be unrealistic. Instead, it is common that the online data at a time step are from multiple underlying OOD data clusters, each of which has a different feature distribution, thus yielding a more diverse and challenging environment for continual model refinement. Also, it is often that in the early stage of the model deployment, the query streams still contain examples of the upstream distribution U , and the ratio of such in-distribution examples will decay over time. Our proposed method. Motivated by these practical considerations, we propose a novel sampling algorithm to control the dynamics of query streams, aiming to encourage diversity and model the decaying upstream distribution. We consider that there are N underlying data clusters, { V 1 , . . . , VN } , each of which corresponds to an unseen distribution, and we have V 0  U which is a data set sampled from the upstream distribution. Our key motivation is to sample the target Q t from three sources: the in-distribution data cluster V 0 , the data of a major OOD cluster V c t , and the mix of other remaining OOD data clusters V  = c t . As shown in Alg. 1 , we have three key configuration arguments ( , ,  ) for controlling the dynamics of the query stream: 1)  is the decaying factor for the ratio of in-distribution data, 2)  is the transition probability of the Markov chain for deciding the index of the major OOD cluster c t , and 3)  is to control the diversity by adding data from remaining OOD clusters; T is the number of episodes and b is size of Q t . Fig. 2 shows examples of query streams and associated error streams. Overall measurement. Recall that there are five basic metrics in Section 2.2, namely EFR (instnat error-fixing rate), UKR (upstream knowledge re-tention), OKR (online knowledge retention), CSR (cumulative success rate) and KG (knowledge gen-eralization). To have a comprehensive yet concise analysis of CMR methods, we report the average and final values of these metrics. Specifically, we use X to denote the average scores in the metric X (e.g., X=UKR) over all time steps, and X (T) to Algorithm 1: Sampling query streams with controllable non-stationarity from multiple data clusters. denote the score at the final time step. Reporting both can help us quickly assess the trend of performance of f t in addition to its final performance. Besides these fine-grained scores, we also provide an overall evaluation criterion (OEC) by taking the average of the four scores except for the EFRs 2 , i.e., OEC = average( UKR, OKR, CSR, KG ) . Validation/testing streams. To evaluate CMR methods (introduced later in Sec. 4), we use the method in Alg. 1 to sample multiple streams under the same configurations (i.e., T, b, , ,  and { V i } ) and then split them as validation streams and testing streams. The validation streams are used to pick the best hyper-parameters of each CMR method (e.g., the  of regularization-based methods 2 Note that we report EFR scores separately because it computes on the method-specific errors unlike other metrics that test on same examples for all CMR methods. 3131 and the size of R t in replay methods) and then they are evaluated on the same set of testing streams. We first introduce our base LM and then illustrate several typical continual learning methods with our extensions to make them applicable to the CMR problem. We discuss other relevant yet not suitable methods in Related Work (Sec. 6). Base model. Pretrained text-to-text language models, such as BART (Lewis et al., 2020) and T5 (Raffel et al., 2020), are commonly used for studying a wide range of NLP tasks. This is because they are generally applicable to tasks that can be formulated as a text-to-text problem, and that they show better generalization potential (Ye et al., 2021; Wei et al., 2021; Sanh et al., 2021). We thus employ the text-to-text formats to pre-process all data in our experiments and use BART-base as the base model. We find the BART-base model is a great fit to support our extensive experiments for its relatively smaller size and comparable upstream performance versus its alternatives. Thus, we use it for our experiments to ensure the scalabil-ity of our analysis and the generality of our findings. Note that we do not aim to offline train a perfect upstream model f 0 with the upstream dataset D . Instead, we focus on the CMR methods that can continually refine a given upstream model. Continual fine-tuning. The most straightforward method is to always use a vanilla optimizer (e.g., Adam (Kingma and Ba, 2015)) to fine-tune f t  1 with a small learning rate on E t for a few epochs, aiming to minimize the loss L Error ( t ) of fine-tuned model f t on E t . Such refined models f t should be able to output correct outputs for these known errors. This method may overfit these errors and thus forget previously acquired knowledge. We introduce a few regularization methods next. A common solution to preventing forgetting is to add a temporal regularization term to the loss for continual fine-tuning: L total ( t ) = L Error ( t ) +  L Reg ( t ) , so that the parameter changes from f t  1 to f t are restricted to avoid over-fitting. where  t is the parameters of f t . This regularization term mitigates the forgetting issue by applying a penalty for every parameter change. Online EWC. Elastic weight consolidation (Kirkpatrick et al., 2017) is a typical regularization method for CL. Unlike L2Reg which gives an equal penalty to every parameter change, EWC produces a weighted penalty such that the parameters that are more important to the previous tasks will have larger penalty weights, leading the parameter changes to find an overlapping space where both previous knowledge and new knowledge can be stored in the parameters. In particular, it efficiently estimates the Fisher Information Matrices F ( t ) ii and use them for consolidating the weighted penalty: LEWC ( t ) = t  1 (cid:88) j =1 (cid:32) 1 2 (cid:88) i F ( j ) ii (cid:0)  it   it  1 (cid:1) 2 (cid:33) . We here employ an extension of EWC by keeping a running sum of F ii to avoid the growth of computation cost in the online setting. Experience replay. ER (Rolnick et al., 2019) is a simple yet effective replay method that stores the previous examples into a growing memory module M . Then, we periodically (every k time steps) sample a small subset of the memory R t as additional training examples for model refinement. It uses a two-stage process: fine-tune f t  1 on R t to get f  t  1 and then fine-tune f  t  1 on E t to get f t . Maximally interfered replay (MIR). Instead of randomly selecting R t from M , MIR (Aljundi et al., 2019) aims to replay the most forgettable examples, conditioning on the current information: f t  1 and E t . It samples a small candidate pool C  M and then ranks the examples in C by their  interference scores . Finally, the R t of MIR is the subset of C with the largest scores. To compute interference scores, we first fine-tune f t  1 on E t to get a virtual model  f t . Then, we compute the 3132 loss of f t  1 and  f t on each example in C to get the interference scores (i.e., the loss delta): score( x i , y i ) =: loss(  f t ( x i ) , y i )  loss( f t  1 ( x i ) , y i ) . MaxLoss replay. Inspired by Jiang et al. (2019) and Kawaguchi and Lu (2020) that show learning with the examples with largest losses can enhance the learning efficiency, we propose a variant of the MIR by redefining the scoring function to score  ( x i , y i ) =: loss(  f t ( x i ) , y i ) and call it MaxLoss, which takes the examples that have largest losses on the virtual model  f t (instead of the largest delta in MIR). Extension for CMR. (1) Bi-Memory: There are two types of knowledge that we want to maintain in CMR: the knowledge acquired in upstream and online learning respectively. Considering that the upstream data is much larger than the incoming errors, it is thus not reasonable to use a single memory module as in other CL problems. We thus use two separate memory modules M u and M o where the upstream memory is M u = D and the online memory M o grows by adding E t . (2) Mixed-Tuning : Instead of following the two-stage method of using R t , we choose to mix R t and E t for fine-tuning f t  1 . Both modifications are supported by their better empirical results. We first present the setup in Sec. 5.1, and report our main results in Table 1 and Figure 3, which we use to discuss our key findings in Sec. 5.2 to 5.5. Please note that there are other additional results in Appendix, and we will release our codebase and full experimental logs to support reproducibility. Reference range. To get a reference range of the performance, we set up two reference methods. 1) FrozenUpstream : We always use the upstream model (i.e., f t  f 0 ) for inference at every time step. 2) OfflineRefining : We combine all the errors of f 0 as E  T and then offline fine-tune the model f 0 with D  + E  T , where D  is a subset of D , to directly get the final refined model f T . Hyper-parameters. We here use a normal configuration of the streams (i.e., T =100, b =64,  =0.9,  =0.5,  =0.8) for studying the CMR methods and discuss other extreme configurations briefly in Sec. 5.5 and more in Appendix. To select the optimal hyper-parameters of each method (e.g., the learning rate, training epochs, method-specific arguments, etc.), we use grid search and pick the ones with the best overall score on validation streams. We report the results in Table 1 & Figure 3, and organize our findings by answering a coherent list of analysis questions: (Q1-Q7) . (Q1) Can we fix errors without forgetting? From the EFR column, we can see that all methods can achieve a 95+% instant error-fixing rate, meaning that they can indeed quickly fix most of the known errors. However, they tend to forget the previously fixed errors and even examples that are correctly predicted before in the query stream. An oracle method that does not forget the previously acquired knowledge would have an OKR (T) of nearly 100%, while the OKR (T) of the continual fine-tuning method is only 77 . 7% . The issue of forgetting both online and upstream knowledge in the continual fine-tuning baseline is quite serious. Notably, its OKR (T) is much lower than its OKR (83.87  77.73), and similarly for UKR (T) and UKR (72.05  66.21). The curves in Figure 3 also suggest that the forgetting issue can be increasingly more serious over time, and it does not show any trend to diminish after T . This confirms that studying the CMR problem is of great importance for enhancing deployed NLP models. (Q2) How well do CMR methods mitigate the forgetting issue? All tested CMR methods can indeed mitigate forgetting without lowering down the EFRs, but they behave quite differently. The regularization methods (i.e., Online L2Reg and Online EWC) are better at improving OKRs rather than UKRs, while replay methods enhance both OKRs and UKRs quite well. For example, MaxLoss can achieve the best OKR (T) ( 91 . 0% ) while having a UKR (T) that is even slightly better than the FrozenUpstream model (80.6 vs 80.3). Moreover, we find that MaxLoss and MIR have great potential to continually improve knowledge retention in the future. From both curves in Fig. 3 and Table 1 (i.e., the comparisons between UKR/OKR and UKR (T) /OKR (T) ), we can see they tend to have better scores in the later stages, but the retention scores of regularization-based methods are decreasing over time. We have a detailed discussion on replay-based methods in Q4 . (Q3) Can refined models generalize to unseen OOD data? Recall that CSRs evaluate the incoming yet not touched examples over time in the stream and the KGs evaluate the held-out examples that are not in the stream. Both metrics thus test on OOD examples that are unseen to the refined model at that time. Compared to the FrozenUpstream baseline, we see all methods have large performance gains (from 30% to 50+% in CSR (T) and KG (T) ). The MIR w/ Online L2Reg even achieves the best CSR (T) and it is significantly better than others, showing that learning with replay effectively improves the generalization ability. From the KG and KG (T) columns of these CMR methods (and Fig. 3), we can see that refined models are increasingly more generalizable to held-out unseen data over time as well. However, the differences among these methods in these two metrics are not obvious, although they are all better than the continual fine-tuning baseline. Interestingly, the regularization method OnlineEWC gets the best score of KG (T) , even though its CSR (T) is worse than others. This suggests that learning with replay might hurt the held-out knowledge generalization, but regularization could maintain a better generalization ability in the long run. (Q4) How should we replay the memory? We find that increasing the replay frequency (i.e., setting a smaller replay interval k ) can largely improve the overall performance for ER, MaxLoss, and MIR. This is expected as there are more fine-tuning steps over the retrieved data. However, the reason for such improvement varies among them. Increasing the replay frequency primarily benefits ER's UKR (T) , but not for other metrics, and it even causes a lower OKR (T) . Instead, MaxLoss and MIR also benefit from larger OKR (T) (MaxLoss: 84.77  89.26; MIR: 87.50  90.43). This suggests that conditional replay methods can get more important stored memory to replay than ER's random selections. Thus, it is promising to develop more advanced conditional replay methods for CMR. (Q5) Are larger buffer sizes always better for conditional replay methods? Larger buffer sizes (i.e., c=256  512  1024) can increase MaxLoss's UKR (T) and OKR (T) with a large margin and thus produce better overall scores. However, MIR with larger buffer sizes suffers from decreasing UKR (T) and OKR (T) . This indicates that that delta of loss as the ranking criteria is less stable than using the virtual loss itself (i.e., MaxLoss). This finding conflicts with the MIR experiments on MNIST-based task-aware streams (Aljundi et al., 2019). We thus conjecture it is because our streams are more complex and the loss landscapes of the task are significantly different from the toy datasets used for evaluation in many prior CL works (e.g., image classification over shuffled MNIST). (Q6) Do different CMR methods produce similar refined models? We use Figure 4 to visualize the differences among the refined models produced by selected CMR methods in two different periods. We can see the refined models by continual fine-tuning (CFT) and regularization methods are more similar to each other, and all replay methods are Stream Dynamics CFT EWC ER MxLs MIR  =0.9,  =0.5,  =0.8 15.81 19.54 20.86 20.78 21.63  =0.9,  = 0.1 ,  =0.8 23.40 24.14 26.32 26.05 26.04  =0.9,  = 0.9 ,  =0.8 18.61 19.38 20.78 19.51 20.60  =0.9,  =0.5,  = 0.5 19.97 20.10 21.97 23.01 22.04  =0.9,  =0.5,  = 0.2 17.37 16.22 19.15 20.60 19.45 Table 2: The gain of OEC (T) over the Frozen Upstream baseline for each method under different stream dynamics. quite distinct from other methods. Also, the divergence among different methods rapidly increases from t = [10 , 20] to t = [30 , 40] . Therefore, we believe that the improvement of these CMR methods is orthogonal to each other, especially between regularization and replay methods. replay methods? Inspired by Fig. 4 and findings in (Q3) , we add an initial experiment by combining the MIR and OnlineL2Reg and show its performance in Table 1. Interestingly, we indeed observe this combination produces a noticeable improvement over both MIR and OnlineL2Reg , yielding the state-of-the-art performance in OEC (T) scores. To the best of our knowledge, there is little prior work that has studied the effect of integrating regularization in (conditional) replay methods, and our initial results suggest that this is a very promising direction for future research . Our above analysis is based on the results of a normal stream configuration (i.e.,  =0.9,  =0.5,  =0.8), but can such tuned hyper-parameters of CMR methods directly apply to streams of extreme configurations? In Table 2, we briefly compare the gain of the previous CMR methods in terms of their OCE (T) improvement over the vanilla FrozenUpstream baseline under a few extreme settings of We find that, in general, all replay methods are still better than continual fine-tuning and Online EWC. ER shows more stable results in extreme settings (e.g.,  = 0.1 or 0.9) but MIR and MaxLoss (MxLs) are more sensitive to the non-stationarity yet less sensitive to the diversity. Continual Learning for NLP. Recently, continual learning (or lifelong learning) has drawn attention in the NLP field (Biesialska et al., 2020; Sun et al., 2020; Wang et al., 2019; Huang et al., 2021; Jin et al., 2021). However, most of these works 3135 follow the traditional task-incremental , boundary-aware , never-revisiting CL setup, which is not directly beneficial to most of the real-world scenarios of deployed NLP models. For example, the CLIF formulation (Jin et al., 2021) focuses on learning over a sequence of different NLP tasks with few-shot data so that the trained model can generalize better to unseen tasks. In contrast, the proposed CMR in this work is a particularly novel CL setup where we focus on continually refining a model with its prediction errors in OOD data streams, thus yielding a boundary-agnostic, dynamically non-stationary environment for CL methods to work. Such fundamental differences between CMR and traditional CL setups make it difficult to directly apply many CL methods that are based on boundary-aware streams, especially for those who require learning task representations. CMR vs. OSAKA The OSAKA (Caccia et al., 2020) problem is similar to the CMR in that we both focus on CL in non-stationary boundary-agnostic data streams. However, it does not consider the distribution diversity inside each time step or the decay of upstream distribution in the online setting. Our sampling method (Alg. 1) fills the gap and yields a more realistic CL setup. In addition, the data streams of CMR are always the prediction errors of the latest model, thus producing a naturally evolving and adversarial environment for CL methods to explore. Moreover, the experiments of OSAKA are limited to simple networks and tasks such as MNIST, but our work uses pretrained Transformer LMs and the QA task, and thus we believe our analysis and findings are more useful for the NLP community and beyond. Model Refinement. Model refinement has recently become an emerging topic in NLP, but existing works have mainly been limited to offline editing time-sensitive factual knowledge in pretrained LMs (Zhu et al., 2020; De Cao et al., 2021; Mitchell et al., 2021). In contrast, our work studies the model refinement in an online continual learning setting and for downstream NLP tasks such as reading comprehension and natural language inference. Jang et al. (2021) attempt to study the knowledge editing problem at a larger scale, but its problem formulation only contains two time-steps, thus being significantly different from CMR. Dhingra et al. (2021) propose a simple method to jointly model text with its timestamp so that the trained language models can be calibrated when new knowledge arrives, while CMR focuses on the error cases from OOD data streams where the timestamps have little correlation with the skills we want the deployed model to learn. Besides, Yao et al. (2021) propose a method of learning from explanations to fix prediction errors, which shares similar high-level motivation but has few direct connections to our focus in this work. In this paper, we propose a novel continual learning formulation named continual model refinement (CMR). The CMR problem aims to efficiently fix prediction errors when learning in out-of-distribution data streams without catastrophically forgetting the acquired knowledge. For studying such a realistic and complex problem, we presented a dedicated evaluation protocol with a general method to create non-stationary, diverse OOD data streams for analysis. Also, we design multiple evaluation metrics to deliver a comprehensive yet concise measurement of CMR methods. The proposed CMR problem with our comprehensive analysis opens up a range of new opportunities for studying continual learning problems that are closer to real-world applications for the NLP community and beyond. For example, based on our results and analysis about (Q3) and (Q6) , we find that it is promising to study how we can integrate both regularization methods and replay methods for mitigating the forgetting issue while improving the generalization ability. The analysis about (Q5) suggests that developing more stable ranking criteria is also important to conditional replay methods (e.g., our simple extension MaxLoss can outperform MIR under specific settings). Developing CMR methods of which the configurations can generalize to diverse types of streams is also an important challenge. We release our codebase and processed datasets for supporting the reproducibility of our experiments and future research.
abstain abstain objective abstain objective objective result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain method method objective objective objective result result result objective abstain abstain abstain method method abstain abstain abstain abstain method abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain method method abstain method abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain method method abstain method method abstain method abstain method method method abstain abstain abstain method method method method method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain method method abstain method abstain method abstain method method abstain abstain abstain abstain method method method abstain method method result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method other abstain abstain method abstain abstain abstain abstain abstain abstain abstain other other other other objective other method other abstain other other method other other objective other abstain abstain objective abstain method method objective result objective abstain abstain