In conversational question answering (CQA), the task of question rewriting (QR) in context aims to rewrite a context-dependent question into an equivalent self-contained question that gives the same answer. In this paper, we are interested in the robustness of a QR system to questions varying in rewriting hardness or difficulty. Since there is a lack of questions classified based on their rewriting hardness, we first propose a heuristic method to automatically classify questions into subsets of varying hardness, by measuring the discrepancy between a question and its rewrite. To find out what makes questions hard or easy for rewriting, we then conduct a human evaluation to annotate the rewriting hardness of questions. Finally, to enhance the robustness of QR systems to questions of varying hardness, we propose a novel learning framework for QR that first trains a QR model independently on each subset of questions of a certain level of hardness, then combines these QR models as one joint model for inference. Experimental results on two datasets show that our framework improves the overall performance compared to the baselines 1 . In conversational question answering (CQA) (Choi et al., 2018; Reddy et al., 2019), several sequential questions need to be answered one by one given a relevant article. To answer a question in CQA, we need to understand the historical context of the question. For example, to answer the question  When did he begin writing these pieces? , we need to know what he refers to in the conversation context. In our work, we address question-in-context rewriting (QR), which aims to rewrite a context-dependent question into an equivalent self-contained question in CQA, e.g., replacing he in the 1 Our source code is available at https://github. com/nusnlp/DiffQRe . This work was done while Wenjuan Han was a research fellow at the National University of Singapore. Topic words : Benigno Aquino III; Senate (2007 10) q 1 : What changes did he make while in the Senate? a 1 : I don't know. q 2 : When was he elected? q  2 : When was Benigno Aquino III elected to Senate? a 2 : May 15, 2007 q 3 : Was he a republican or democrat? a 3 : Genuine Opposition (GO), a coalition comprising a number of parties, including Aquino's own Liberal Party, ... q 4 : Are there any other interesting aspects about this article? q  4 : Are there any other interesting aspects about Benigno Aquino III article aside from political affiliation or when Benigno was elected? a 4 : Aquino was endorsed by the pentecostal Jesus Is Lord Church. above example with its referent from the context. The task is formulated as a text generation task that generates the rewrite of a question given the original question and its conversation context (Elgohary et al., 2019). We are interested in how robust a QR system is to questions with different rewriting hardness (or difficulty). As we can see from the examples in Table 1, rewriting the question q 2 requires only replacing the pronoun he by its referent, which usually appears in the conversation context, and the model can identify the referent by attention (Luong et al., 2015). However, for the question q 4 , to find the missing aside from clause, the model needs to understand the entire conversation since the question asks about other interesting aspects about the article related to the topic of the entire conversation. Understanding the whole context will be challenging for the model. Can a QR model still work well when rewriting the hard questions? In section 6.3, our first study is on evaluating the performance of a QR model under questions varying in hardness. One issue in this process is 2100 that there is a lack of classified questions in different rewriting hardness. Though we can rely on human labor to annotate the questions, it is expensive and not scalable. Instead, we propose a simple yet effective heuristic method to classify the questions automatically. We measure the discrepancy between a question and its rewrite, where the larger the discrepancy, the more difficult to rewrite the question. The intuition is that if a question is very dissimilar to its rewrite, more information has to be filled into the rewrite, which means the question is harder to rewrite. We specifically use the BLEU score to measure the discrepancy, and lower scores mean larger discrepancies. Using this method, we then split the questions into three subsets: hard, medium, and easy, and evaluate the baseline systems using these subsets. In order to verify the classified subsets and find out what makes questions different in rewriting difficulty, in section 6.3.2, we further evaluate the question characteristics in hard, medium, and easy subsets through human evaluation. We first manually summarize the commonly used rules for rewriting questions from the training set, and then annotate the questions using the labels of summarized rewriting rules, followed by counting the number of these rewriting rules used in these subsets. Finally, to enhance the robustness of a QR model to questions varying in difficulties, we propose a novel learning framework in section 5, where we first separately train a QR model on each hard, medium, and easy subset, and then combine these models into a joint model for inference. Training one sole model on each subset is to let the model better learn domain-specific information to deal with one specific type of questions (hard/medium/easy). By combining the models together, we have a joint model capable of rewriting questions differing in rewriting hardness. Specially, we introduce adapters (Houlsby et al., 2019) to reduce parameters when building private models and we present sequence-level adapter fusion and distillation (SLAF and SLAD) to effectively combine the private models into a joint model. Our contributions in this paper include:  We are the first to study the robustness of a QR system to questions with varying levels of rewriting hardness;  We propose an effective method to identify questions of different rewriting hardness;  We manually annotate questions sampled from the subsets with summarized rewriting rules for validity and address what makes questions hard or easy for rewriting;  We propose a novel QR framework by taking into account the rewriting hardness. We have the following observations in our paper:  The baseline systems perform much worse on the hard subset but perform well on the easy subset;  We find that easy questions usually only require replacing pronouns but hard questions involve more complex operations like expanding special Wh* questions ;  Experiments show that our QR learning framework enhances the rewriting performance compared to the baselines. Elgohary et al. (2019) created the QR dataset which rewrites a subset of the questions from QuAC (Choi et al., 2018). Based on this dataset, some recent work has studied this task and formulates QR as a text generation task with an encoder-decoder architecture (Elgohary et al., 2019; Kumar and Joshi, 2016; Vakulenko et al., 2020; Li et al., 2019; Lin et al., 2020a). The difficulty of answering a question given a relevant document has been studied in the question answering community (Dua et al., 2019; Wolfson et al., 2020). Sugawara et al. (2018) examine 12 reading comprehension datasets and determine what makes a question more easily answered. Perez et al. (2020); Min et al. (2019); Talmor and Be-rant (2018); Dong et al. (2017) study how to make a hard question more easily answered. However, there is no work to date that studies whether rewriting difficulties exist in QR and how to measure the difficulties. Some other work is similar to QR but focuses on other tasks such as dialogue tracking (Rastogi et al., 2019; Su et al., 2019; Liu et al., 2020) and information retrieval (Voskarides et al., 2020; Lin et al., 2020b; Liu et al., 2019). Varying rewriting difficulties can result in multiple underlying data distributions in the QR training data. The shared-private framework has been studied to learn from training data with multiple distributions (Zhang et al., 2018; Liu et al., 2017). One issue of the shared-private framework is parameter inefficiency when building private models We use adapter tuning (Rebuffi et al., 2018, 2017) 2101 to build the private models. Adapter tuning was recently proposed for adapting a pre-trained language model, e.g., BERT (Devlin et al., 2019), to downstream tasks (Pfeiffer et al., 2020a,c; Houlsby et al., 2019), and its effectiveness has been verified by previous work (Bapna and Firat, 2019; Pfeiffer et al., 2020b; Wang et al., 2020; He et al., 2021). We are the first to apply it to reduce model parameters in the shared-private framework. How to combine the knowledge stored in multiple adapters is also important. Pfeiffer et al. (2020a) propose adapter fusion to build an ensemble of adapters in multi-task learning. We propose sequence-level adapter fusion in our work. Question-in-context rewriting (QR) aims to generate a self-contained rewrite from a context-dependent question in CQA. Given a conversational dialogue H with sequential question and answer pairs { q 1 , a 1 ,    , q n , a n } , for a question q i from H with its history h i = { q 1 , a 1 ,    , q i  1 , a i  1 } , we generate its rewrite q  i . We define the labeled dataset D = { q i , h i , q  i } |D| i =1 which is a set of tuples of question q , history h , and rewrite q  . Following previous work (Elgohary et al., 2019), we model QR in an encoder-decoder framework, by estimating the parameterized conditional distribution for the output q  given the input question q and history h . For ( q , h , q  )  D , we minimize the following loss function parameterized by  : L NLL =  log P ( q  | q , h ;  ) =  T q  (cid:88) t =1 | V | (cid:88) k =1 1 { q  t = k } log P ( q  t = k | q  <t , q , h ;  ) (1) in which T q  is the length of q  and | V | is the vocabulary size. Following Elgohary et al. (2019), q and h are concatenated into one sequence as the input. All previous turns of the history information are combined for learning. The choice of the encoder-decoder framework can be LSTM (Elgo-hary et al., 2019), transformer (Vakulenko et al., 2020), or pre-trained language models (Lin et al., 2020a). In our work, we build our model based on the pre-trained language model BART (Lewis et al., 2020). The difficulty of rewriting a question varies across questions. We propose a simple yet effective heuristic to formulate rewriting difficulty as the discrepancy between a question and its rewrite. To generate a self-contained rewrite, we need to identify relevant information from the conversation context to incorporate it into the original question. We observe that if the discrepancy is large, we need to identify more missing information from the conversation context which makes the rewriting task more difficult. In this work, we use BLEU score to measure the discrepancy. BLEU has been widely used to measure how similar two sentences are (Papineni et al., 2002). Given a question q and its rewrite q  , we define the difficulty score z for rewriting q as: z = BLEU ( q , q  ) (2) where the rewrite q  is the reference and z  [0 , 1] . A low z score indicates a larger discrepancy between q and q  , making it more difficult to rewrite q into q  . Besides BLEU, we also study the effectiveness of ROUGE, lengths of q and q  , and | q | / | q  | in  6.5 to measure rewriting difficulty. Previous work on QR learns to rewrite questions with only one shared model (Elgohary et al., 2019), which cannot adequately model all questions with different rewriting difficulties. Instead of using only one shared model, we propose a novel method to classify a question into several classes by measuring its rewriting difficulty (  5.1), learn a private model for each class (  5.2), and finally combine the private models for inference (  5.3). Different questions with varying rewriting difficulties result in multiple data distributions in the training set. By dividing the training data into several classes with varying rewriting difficulties, we can better learn the data distributions with the help of private models (Zhang et al., 2018). We compute the difficulty score z of each question in the dataset. We set score intervals and group the questions with difficulty scores within the same interval together. Specifically, we divide the original dataset D into m classes: {D 1 , D 2 ,    , D m } . Setting m to a large number (e.g., the number of training samples) can more accurately model the 2102 + self-attention hard easy med feed forward hard easy med Encoder self-attention hard easy med cross attention hard easy med Decoder feed forward hard easy med Adapter Figure 1: Illustration of our model architecture. data distribution of the training data, but at the expense of data sparsity in each class such that a private model cannot be adequately trained. After dividing the questions into m classes, we learn a private model for each class. By training on each class of data, the private model can better learn the domain-specific information. The common way to use a pre-trained language model (PLM) such as BART is to fine-tune the model on the downstream task. However, doing so will require m times the number of PLM parameters to build all private models, where m is the number of classes. This results in a large number of parameters, leading to inefficiency. To reduce the number of model parameters in learning the private models, we introduce adapters into the PLM. Adapters are light-weight neural networks and are plugged into the PLM. When adapting the PLM to downstream tasks, we only need to update the parameters of the adapters but keep the original parameters of the PLM frozen and shared among all private models. Where to place the adapters in the neural architecture will affect the efficacy of adapters. As shown in Figure 1, for each transformer layer in the encoder, we add the adapters after the self-attention layer and feed-forward layer. We further add the adapters after the cross-attention layer in the decoder. Though our model is built on BART, our proposed placement of adapters can also be used in other PLMs, such as T5 (Raffel et al., 2020). In Figure 1, the adapter is a module with a stack of two linear layers following Houlsby et al. (2019). Formally, given an input hidden vector x from the hard med easy feed forward logits hard med easy Encoder Decoder classifier class distribution hard med easy feed forward distillation Adapter Fusion 1 2 Adapter Distillation Decoder Figure 2: Illustration of our sequence-level adapter fusion and distillation. where f 1 (  ) is the down-scale linear layer and f 2 (  ) is the up-scale linear layer. The hidden vector size is smaller than the dimension of the input vector. Learning a private model for one class only introduces 5  N adapters, where N is the number of layers in the encoder and decoder. The original parameters of the PLM are shared by all adapters, so the number of parameters required when building the private models can be much reduced. After learning the private models for all classes, at test time, we present the question to the corresponding private model to generate its rewrite if we know which class this question belongs to. However, it is not possible to determine the difficulty score by calculating the BLEU score between the question and its rewrite since there is no gold-standard rewrite for the question at test time. As such, we need to combine the private models into one model for inference. In this work, we propose two methods to combine the private models, as explained below. Sequence-level Adapter Fusion (SLAF). After dividing the training set into m classes based on the difficulty scores, we assign a difficulty label to each class to obtain a set of class labels { l 1 , l 2 ,    , l m } . We introduce a classifier to learn to predict the difficulty label l , given a question q and its conversation history h . As shown in Figure 2, during inference, we obtain the logistic output from each private model. The classifier generates the class distribution to combine the logistic outputs for sequence generation. By assigning a difficulty label to each question, we obtain the dataset D  = { q i , h i , q  i , l i } |D  | i =1 . For each training sample ( q , h , q  , l )  D  , we mini-2103 mize the following loss function: L  c NLL =  log softmax (cid:0) m (cid:88) i =1  i f i ( q , h ;  i ) (cid:1)  log P ( l | q , h ;  c ) (4) where f i is the i th private model,  i is the class weight of the i th private model, and  c is the parameter of the classifier. We jointly estimate the conditional distribution for sequence generation and the distribution for classification. In this process, the private models are frozen and not updated. We combine the vectors out of the private models to calculate the vector f c as the input for the classifier: f c = 1 m m (cid:88) i =1 f iencoder ( q , h ;  i ) (5) where f iencoder is the encoder of the i th private model. For each training sample ( q , h , q  , l )  D  , we define the knowledge distillation loss function as follows: L  SKD =  T q  (cid:88) t =1 | V | (cid:88) k =1 P ( l ) { q  t = k | q  <t , q , h ;  ( l ) }  log P ( q  t = k | q  <t , q , h ;  S ) (6) in which we approximate the output distribution of the teacher private model l parameterized by  ( l ) with the student model parameterized by  S . For each private model, we average the token embeddings from the last layer of the encoder. Sequence-level Adapter Distillation (SLAD). SLAF provides a way to combine the private models, but it is time-consuming during inference since it requires each private model to compute its logistic output before combination. Another drawback is that the domain classifier in SLAF cannot generate the best class distributions at test time, causing non-optimal rewriting results by SLAF. As shown in Figure 2, to speed up inference and better combine the private models, we distill the private models into one shared model. We expect the student model S (modeled by adapters) to be able to generate questions with different rewriting difficulties. We learn the student model with the following function: L  S distill = (1   )  L  SKD +   L  SNLL (7) where L  SNLL is the same loss function in Eq. 1, and  is a hyper-parameter. The private models are fixed in the distillation process. Since we directly distill the knowledge of the private models into a Train Valid Test All CANARD 31,526 3,430 5,571 40,527 QRECC 57,150 6,351 16,451 79,952 Table 2: Data splits of CANARD and QRECC. shared model without the soft weights generated by the domain classifier from SLAF, SLAD can better combine the private models and achieve better rewriting performance. We conduct our experiments on CANARD (Elgo-hary et al., 2019) and QRECC (Anantha et al., 2021), which are designed for the task of question rewriting in CQA. CANARD was created from QuAC (Choi et al., 2018), by rewriting a subset of the questions by humans. The dataset consists of tuples of question, conversation history, and rewrite. QRECC answers conversational questions within large-scale web pages. Detailed data splits for the two datasets are shown in Table 2. We divide the questions into hard, medium, and easy classes, and the statistics are presented in Table 3. 6.2 Setup Model Settings. We build our models on the pretrained language model of BART (Lewis et al., 2020). Specifically, we use BART-base to initialize our models. There are 6 transformer layers for the encoder and decoder in BART-base. For our Model  D Hard Medium Easy Mean LSTM-S 26.29 50.79 79.41 49.81 Fine-tune-S 39.38 53.70 66.33 53.14 Adapter-S 39.20 53.14 65.97 52.77 Table 4: BLEU scores (in %) on hard, medium, and easy classes from CANARD , based on the shared model. adapter, we map the dimension of the input hidden vector from 768 to 384 which is re-mapped to 768 for the output vector. The hidden vector size for adapter tuning is the default value of 384. Based on BART-base, we need a total of 6  2 + 6  3 = 30 adapters for each private model. We set  to 0.5 in Eq. 7 for CANARD and 0.9 for QRECC. from Eq. 4 is set to 2 for both CANARD and QRECC. When fine-tuning BART, we set the learning rate to 1e-5, and for adapter tuning, the learning rate is 1e-4 (both values are tuned from {1e-4, 1e-5}). We use the validation set to keep the best model based on the BLEU score. We implement our models with HuggingFace (Wolf et al., 2019) and keep the other default training settings. In CANARD , about 20% of the questions can be rewritten by replacing pronouns with their referents, so we carry out pronoun replacement first for the questions (if any) before using BLEU scores to measure rewriting difficulties. More details are given in Appendix A. Baselines. We compare to the following baselines. S denotes training only one shared model with all the training data, which is commonly used in previous work (Elgohary et al., 2019; Lin et al., 2020a). By adapting BART, P-hard , P-medium , and Peasy are the baselines that train private models on the hard, medium, and easy classes respectively, using fine-tuning or adapter-tuning. Assuming that rewriting difficulty labels are accessible for questions at test time (i.e., the oracle setting), Mix-gold processes a question by the corresponding private model using the difficulty label. SLAF and SLAD denote sequence-level adapter fusion and adapter distillation respectively for combining the private models of P-hard, P-medium, and P-easy. SLAF-uni. combines the private models with uniform distributions. SLAF-pred predicts the class label for the input and then chooses the corresponding private model for generation. LSTM-S trains one model using an LSTM-based Seq2Seq model with copy mechanism (See et al., 2017) which was used in Elgohary et al. (2019). Evaluation Metric. Following Elgohary et al. (2019), we use BLEU 2 to obtain the results on hard, medium, and easy classes, and the three results are 2 https://github.com/mjpost/sacrebleu 2105 averaged to obtain the mean result. We first study rewriting difficulties across different questions. Table 4 shows the results on hard, medium, and easy classes on CANARD . Each class vs. Overall : Comparing to the overall results, the rewriting performances of hard questions drop substantially, but are much higher on the easy class. LSTM-S vs. BART-S : By comparing LSTM-S to tuning on BART, LSTM-S achieves higher performance on the easy class but much worse performance on hard and medium classes. This is probably because for easy questions, the model only needs to copy some words from the context and LSTM-S has an explicit copy mechanism to achieve this goal but not BART. Since BART learns a more complex model than LSTM-S, it can better deal with harder questions. We further divide the test set into ten classes in Figure 3, where the interval [0 , 1] is equally divided into ten sub-intervals of size 0.1. We find that when z gets smaller, rewriting performance degrades, indicating an increase in rewriting difficulty. The above evaluation results show that our method can effectively divide the questions into subsets with different rewriting difficulties. Here, we conduct a human evaluation to evaluate the question characteristics on these subsets for validity and see what makes the questions hard or easy to rewrite. Question Annotation. To find out what makes the questions different, we first summarize the commonly used rewriting rules, which describe the operations of translating a question into its rewrite. 6 rules are summarized from the training set of CANARD and presented in Table 5. Different rules account for different rewriting hardness for QR systems. For example, the rule of replace pronoun is very simple since it only requires the model to determine the pronoun to replace. However, rules 5 and 6 shown in the table will be much harder because the model needs to understand the conversational history well, and the information to be filled in is substantial. Then we randomly select 50 examples from each subset (hard, medium, and easy) from the test set and annotate what rules in Table 5 are used for each example. One question may have multiple rewriting rules. More details are in Appendix B. Model  D Hard Medium Easy Mean LSTM based S 26 . Results. We sum the number of each rewriting rule in each subset and show the distributions of rewriting rules for each subset in Figure 4. The three distributions are quite different. We find that:  the easy subset mainly uses rule 1 for rewriting questions;  for medium and hard subsets, other rules are used, such as rules 2, 3, and 4 which are more complex than rule 1;  the hard class uses more rules 2, 3, 5, and 6 compared to the medium class, which demonstrates that the hard class is more difficult than the medium class. Discussion. By knowing the characteristics of each class of questions, we can optimize the model architecture of private models accordingly. For hard questions, we can add some rules to deal with Wh* questions. For easy questions, LSTM-based models seem to be good enough as Table 4 indicates. In this work, we have shown that the questions vary in rewriting difficulties and to improve the overall rewriting performance, we focus on the ensemble method to combine the private models. We leave optimizing the model architecture to future work. We report our results on question rewriting based on CANARD and QRECC. From the results in Tables 6 and 7, we first show the results of each class, then the mean performances are displayed. Mix-gold, SLAF, SLAD vs. S : ( a ) Mix-gold, SLAF, 2106 Model  D Hard Medium Easy Mean Tuning BART-base with adapters S 45 . and SLAD are consistently better than S, which demonstrates the effectiveness of learning private models to model multiple underlying distributions. ( b ) From the results on each class, SLAF and SLAD can substantially enhance the performance on medium and easy classes compared to S. ( c ) SLAD is more effective than SLAF and SLAD is more efficient during inference. ( d ) We find Mix-gold to be better than SLAF and SLAD, since Mix-gold is an oracle model that uses the correct difficulty label to select the private model for inference. We find that by learning a private model for each class, the performance on the corresponding class can be consistently improved, which explains why Mix-gold, SLAF, and SLAD can outperform S. We also find that the sole private model cannot improve the overall rewriting performance of the three classes, but SLAF and SLAD can outperform S after model ensemble, which demonstrates the necessity of combining the private models. Model Ensemble. One question is whether the improvements of SLAF and SLAD simply come from combing multiple models and whether applying only one private model selected by the predicted class label is better. As shown in Tables 6 and 7, we find SLAF-uni. performs worse than SLAF and SLAD, which demonstrates that the benefits of SLAF and SLAD are not simply because of the model ensemble, but class estimation also helps (In SLAD, class estimation lies in using gold class labels of questions for knowledge distillation during training). SLAF-pred can be regarded as an ensemble method since it uses multiple private models during inference. Compared to SLAF, SLAF-pred uses one-hot class weights to combine the private models. However, SLAF-pred performs worse than Method  D D 1 D 2 D 3 Trend Std. SLAF, and the reason could be that classifying the question into the corresponding class is nontrivial, wrong predictions will have much worse rewriting results as the results of P-hard, -medium, -easy on other classes indicate. Analysis of Rewriting Difficulty Measures. In our work, we use BLEU to measure the discrepancy between a question and its rewrite. We further experiment with other methods to assess their effectiveness for difficulty measurement. CANARD is evaluated here. As shown in Table 8, we first use the length of a question ( | q | ), its rewrite ( | q  | ), and their ratio ( | q | / | q  | ) to calculate a difficulty score. After re-ranking the questions with a difficulty score, we divide the ranked questions equally into three classes. Interestingly, we find that | q | works well. After analysis, we find that rewriting short questions requires finding much missing information, which makes short questions hard questions. The | q | / | q  | metric is not very useful, since | q | / | q  | can only measure the discrepancy in question lengths, but does not necessarily measure their semantic difference. | q  | does not work for difficulty measurement. Not surprisingly, the ROUGE score is also useful in measuring discrepancy just like BLEU. Analysis of Learning Data Distribution. Tables 6 and 7 show that learning private models can enhance performance on each class. We further divide the data into eleven classes ( z  [0 , 0 . 1] , (0 . 1 , 0 . 2] ,    , (0 . 9 , 1) , 1 ) and learn a private model for each class. We build the private models using LSTM-S, in which we first train a shared model on the full training data, then fine-tune the shared model on each class to obtain the private models. Table 9 shows the BLEU scores where the score in the ( i, j ) entry is obtained by training on class 2107 0 1 2 3 4 5 6 7 8 9 10 0 19.2 28.3 34.7 39.9 44.2 50.3 57.9 64.6 71.6 80.3 71.9 1 17.7 28.1 36.1 43.3 48.5 53.6 61.4 66.5 74.5 75.1 74.5 2 16.0 28.6 36.2 44.0 49.3 55.9 64.7 70.3 79.6 86.2 78.6 3 15.0 26.8 35.7 45.3 51.3 57.5 66.9 70.8 80.0 88.4 81.2 4 12.8 26.0 35.9 44.8 52.1 60.1 68.9 73.5 78.5 95.7 81.8 5 12.5 25.3 35.1 44.9 50.3 61.1 70.4 75.9 79.9 94.0 84.4 6 11.8 25.0 34.9 44.4 51.7 61.7 71.0 77.4 81.9 89.4 86.7 7 11.9 24.4 34.5 44.2 51.5 61.8 71.7 80.2 84.9 91.1 87.9 8 9.4 20.8 31.3 41.7 49.4 58.6 68.1 76.0 85.6 97.6 92.0 9 15.8 27.3 35.3 44.7 50.9 60.2 69.5 75.6 83.7 89.4 85.9 10 13.5 24.7 34.8 44.4 51.9 60.2 69.7 75.4 82.0 98.4 92.2 Table 9: BLEU scores for different classes on CANARD . i and testing on class j . On the whole, learning private models can enhance the performance of the corresponding class. With these private models, we can better model the data distributions, but how to combine a large number of private models is a challenge, since it is hard to train a classifier to correctly predict so many class labels, which will have some negative effects on the model ensemble. Analysis of SLAF & SLAD. We plot the class distributions of hard, medium, and easy classes in Figure 5. We find that in the hard class, the class weights are almost equally distributed among the private models, which means that the hard questions are difficult for classification. This result explains why SLAF performs worse than S for hard questions in Tables 6 and 7. We further study the contribution of distillation in SLAD. In Figure 6, on the whole, when  increases, the contribution of distillation decreases, and the performance drops, indicating that distillation is important for SLAD. Case Study. We further show generated rewriting samples of various methods on CANARD in Appendix C. 7 Conclusion In this work, we study the robustness of a QR system to questions varying in rewriting hardness. We use a simple yet effective heuristic to measure the rewriting difficulty. We further propose a novel method to deal with varying rewriting difficulties. Tested on CANARD and QRECC, we show the ef-0 0.1 0.3 0.5 0.9 53 54 BLEU ( % ) =1 54.27 54.08 54.14 54.36 53.55 Figure 6: BLEU socres for different  values on CANARD . fectiveness of our methods. This research is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG-RP-2018-007 and AISG2-PhD-2021-08-016[T]). The computational work for this article was partially performed on resources of the National Supercomputing Centre, Singapore (https://www.nscc.sg).
abstain abstain objective result objective result abstain abstain abstain other abstain abstain objective objective objective objective objective abstain objective objective abstain abstain abstain result abstain abstain abstain objective abstain abstain objective method abstain method method result objective objective abstain objective method other result other other other other other other other other other method other objective other other objective abstain method method method method abstain abstain abstain method abstain abstain method abstain method abstain method abstain method abstain abstain abstain method method abstain method abstain abstain method abstain abstain abstain abstain method abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain method abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain result result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain other abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain result abstain other other