Machine Reading Comprehension (MRC) reveals the ability to understand a given text passage and answer questions based on it. Existing research works in MRC rely heavily on large-size models and corpus to improve the performance evaluated by metrics such as Exact Match ( EM ) and F 1 . However, such a paradigm lacks sufficient interpretation to model capability and can not efficiently train a model with a large corpus. In this paper, we argue that a deep understanding of model capabilities and data properties can help us feed a model with appropriate training data based on its learning status. Specifically, we design an MRC capability assessment framework that assesses model capabilities in an explainable and multi-dimensional manner. Based on it, we further uncover and disentangle the connections between various data properties and model performance. Finally, to verify the effectiveness of the proposed MRC capability assessment framework, we incorporate it into a curriculum learning pipeline and devise a Capability Boundary Breakthrough Curriculum (CBBC) strategy, which performs a model capability-based training to maximize the data value and improve training efficiency. Extensive experiments demonstrate that our approach significantly improves performance, achieving up to an 11.22% / 8.71% improvement of EM / F 1 on MRC tasks. A competency assessment is used to measure some-one's capabilities against the requirements of their job (Cheryl Lasse, 2020). In other words, it measures how (behaviors) someone does the what (task or skill). By showing what it looks like to be good  Equal contribution. Canada CIFAR AI Chair. Corresponding authors. in a job, a competency assessment can effectively empower and engage people who want to understand and improve their unique skill profile and tell them what action to take to close any gaps so they can own their development. A natural question that arises here is: can we develop competency assessments for machine learning models to help better understand their capabilities and improve their performance on a given task? In this paper, we focus on competency assessments for machine reading comprehension (MRC). MRC is a core task in natural language processing (NLP) that aims to teach machines to understand human languages and answer questions (Zeng et al., 2020; Chen et al., 2019). Recently, pre-trained language models (LMs) (Mikolov et al., 2013; Peters et al., 2018; Pennington et al., 2014; Devlin et al., 2018) have demonstrated superior performance on MRC tasks by pre-training on large amounts of unlabeled corpus and fine-tuning on MRC datasets. The performance is usually evaluated by metrics such as Exact Match ( EM ) and F 1 score, lacking interpretability to the capabilities of a model. That is to say, such metrics only tell how good a model performs overall on a specific dataset, but uncovers little about what specific skills a model has gained and the level of each skill. We argue that the value of each data sample varies during the training process of a model, depending on the model's current capabilities. A deep understanding of the model's intrinsic capabilities can help us estimate each data sample's learning value and better manage the training process to improve the training efficiency. Take student learning as an example. There is no doubt that a college student can do well in solving primary school level exercises, but such exercises do not help improve his/her ability. On the contrary, a primary school student can not acquire knowledge efficiently from college-level exercises due to the big gap between his/her current knowledge or skills and the require-5858 Capability c i Subclasses Metrics m ji Reading words Recognize vocabulary Intra-n (Gu et al., 2018) and Ent-n (Serban et al., 2017). ment to solve the exercises. We need to measure the ability of a student and then choose the appropriate exercises accordingly. Existing works on interpreting MRC model capabilities concentrate on analyzing a model's behavior with adversarial data (Jia and Liang, 2017), or defining the prerequisite skills to solve a specific dataset (Sugawara et al., 2017). However, these works require costly human annotation efforts or ignore the fact that model capabilities change during the training progresses. In this paper, we design a competency assessment framework for MRC model capabilities. Specifically, we define four major capability dimensions for understanding text and solving MRC tasks: reading words , reading sentences , understanding words and understanding sentences , which are inspired by the computational models of human text comprehension in psychology (Kintsch, 1988). Based on the proposed framework, we can obtain a more appropriate assessment of model capabilities than the regular EM or F 1 metrics. Furthermore, we analyze a variety of data properties to estimate how good a model has to be to solve a specific data sample and identify the relationships between data properties and model performance. This greatly helps us estimate the learning value of each training sample. Based on this analysis, we discover a very common situation: if a sample is scored as a high value in one capability dimension, the other dimensions have the same tendency as well, and vice versa. To alleviate these inevitable correlations, we utilize data whitening to quantify each sample as four capability-specific scores in a decorrelated fashion. evaluate its efficiency, we employ it in a curriculum learning pipeline and design a Capability Boundary Breakthrough Curriculum (CBBC) strategy. This strategy gradually enlarges the model capability boundary by picking samples around the boundary and breaking through it. Based on the analysis of model capabilities and data properties, we feed the model with training samples that are neither too simple nor too hard for it to solve. Extensive experiments on four benchmark datasets demonstrate that our approach significantly improves the performance of existing MRC models, achieving up to an 11.22% / 8.71% improvement of EM / F 1 on MRC tasks. These results show the reasonability and effectiveness of our proposed assessment framework and provide a widely applicable measurement for dealing with the relationship between the model capability and data quality. In this section, we first formulate our competency assessment framework of 4-dimensional MRC capabilities. Based on this framework, the data properties related to each capability dimension are described as corresponding heuristic metrics. We then uncover the relationship between various data properties and model performance in a decorrelated manner, quantifying each sample as 4-dimensional capability-specific scores with little correlation. Human text comprehension has been studied in psychology for a long time. Constructionist, landscape model, and computational architectures have been proposed for such comprehension (McNamara and Magliano, 2009). Among them, the construction-5859 Context: James is a trouble making Turtle . One day, James went to the grocery store and pulled all the pudding off the shelves and ate two jars. Then he walked to the fast food restaurant and ordedred 15 bags of fries. Q1: Who is the trouble making turtle? A1: James Q2: Where did James go after he went to the grocery store? A2: A fast food restaurant Requidred capabilities: syntactic matching, temporal relation, semantic overlap Figure 1: Two example questions Q 1 and Q 2 with different difficulties require different capabilities. integration (CI) model (Kintsch, 1988) is one of the most basic and influential theories. The CI model assumes three different representation levels (sur-face structure, textbase, and situation model) and a two-step process (construction and integration) to understand text comprehensively. It first constructs the propositions ( i . e . textbase) from the raw textual input ( i . e . surface structure), then integrates the local connections into a globally coherent representation ( i . e . situation model). Based on this situation model, a given text is understood comprehensively and can even be grounded to other modalities. Inspired by the two-step process of the CI model, we formulate our assessment framework by 4-dimensional capabilities as summarized in Table 1. We sketch out the meaning of each MRC capability { c i } 4 i =1 and highlight some heuristic metrics { m ji } n ( i ) j =1 (where n ( i ) means the number of metrics to measure a sample's learning value to capability c i ) as follows. Reading words. To formulate the surface structure of the CI model in our framework, we first highlight the text representation at the verbal or linguistic level. Theoretically, the units at the linguistic level are the words that make up the text and the hierarchical sentence constituents to which these words belong. Empirically, Sugawara et al. (2018) has shown that some questions are answered correctly by just reading the first k tokens. Similarly, the perturbation-based experiments of Nema and Khapra (2018) have demonstrated the significant influence of four types of words ( i . e . content words, named entities, question types, and function words) on an MRC question. Therefore, the dimension of reading words is defined as recognizing the observed vocabulary and the special words' appearance ( i . e . function words). In this study, The former is implemented as Intra-n (Gu et al., 2018) and Ent-n (Serban et al., 2017) to measure vocabulary distribution, while the latter is computed as the frequency of corresponding words. Reading sentences. The rules that are used to form a sentence using the aforementioned linguistic units are conventional phrase-structure grammars. Consequently, before understanding the information contained in a text, an MRC system inevitably requires capturing the sentence structure and handling the possible obscure words. We define the dimension of reading sentences as recognizing grammaticality and readability, and they are implemented by constituency parsing tree statistics and readability metrics 1 , respectively. Understanding words. The semantic representation of text is usually established by local and global links according to the linguistic units at word-level and sentence-level, respectively. To reflect the local semantic structure, we design the dimension of understanding words to assess how well an MRC model understands the relationships between words. In this work, we exemplify two relations ( i . e . the arithmetic operations and logical items) that usually have salient patterns in the text. The former directly focuses on statistical and operational reasoning from the text, while the latter deals with the reasoning of predicate logic, e . g . conditionals and qualifiers. Inspired by the human annotation process (Boratko et al., 2018; Schlegel et al., 2020), where the annotators are asked to label as many reasoning skills as possible by paying more attention to corresponding indicative words, the sub-capabilities of this dimension is quantified as the frequency of those words. Understanding sentences. Integrating the local structures into a global representation requires not only the text itself but also specific knowledge. To simplify the forms of knowledge, we divide the dimension of understanding sentences into two subclasses, linguistic and factual reasoning. They respectively mean understanding the relationship between sentences based on linguistics and the events ( i . e . five dimensions including time, space, causation, intentionality, and objects). Among metrics of this dimension, BERTScore (Zhang* et al., 2020), MoverScore (Zhao et al., 2019) and LS_score (Wu et al., 2020) are used to measure semantic overlap between the context and question and multihop reasoning is an extra particular subclass on the HotpotQA (Yang et al., 2018; Cheng et al., 2021) 1 https://py-readability-metrics.readthedocs.io/ 5860 Value SQuADv1 SQuADv2 HotpotQA RACE r p r p r p r p F 1 v 1 -0.131 0.000 -0.135 0.000 -0.146 0.007 -0.129 0.010 -0.120 0.018 -0.124 0.026 -0.135 0.009 -0.118 0.017 v 2 -0.162 0.002 -0.152 0.027 -0.174 0.022 -0.173 0.003 -0.154 0.000 -0.144 0.025 -0.166 0.000 -0.165 0.005 v 3 -0.134 0.000 -0.130 0.000 -0.141 0.029 -0.135 0.011 -0.163 0.016 -0.159 0.026 -0.170 0.006 -0.164 0.017 v 4 -0.166 0.001 -0.155 0.018 -0.182 0.000 -0.181 0.019 -0.198 0.020 -0.187 0.026 -0.214 0.018 -0.213 0.006 -0.208 0.015 -0.197 0.020 -0.224 0.013 -0.223 0.002 -0.206 0.000 -0.195 0.002 -0.222 0.010 -0.221 0.001 -0.168 0.023 -0.157 0.022 -0.184 0.023 -0.183 0.006 -0.168 0.010 -0.157 0.004 -0.184 0.002 -0.183 0.000 F l o g i t s v 1 -0.144 0.012 -0.151 0.000 -0.165 0.007 -0.142 0.010 -0.133 0.018 -0.140 0.026 -0.154 0.009 -0.131 0.017 v 2 -0.188 0.002 -0.175 0.000 -0.205 0.022 -0.203 0.003 -0.180 0.000 -0.167 0.025 -0.197 0.028 -0.195 0.005 v 3 -0.163 0.000 -0.157 0.000 -0.172 0.000 -0.163 0.011 -0.192 0.016 -0.186 0.026 -0.201 0.006 -0.192 0.017 v 4 -0.206 0.001 -0.192 0.018 -0.228 0.023 -0.226 0.019 -0.238 0.020 -0.224 0.026 -0.260 0.018 -0.258 0.006 -0.248 0.015 -0.234 0.020 -0.270 0.013 -0.268 0.002 -0.246 0.023 -0.232 0.002 -0.268 0.010 -0.266 0.001 -0.208 0.023 -0.194 0.022 -0.230 0.023 -0.228 0.006 -0.208 0.010 -0.194 0.004 -0.230 0.002 -0.228 0.000 Table 2: The Pearson's correlation ( r ) between capability-specific value v i and model performance. dataset. For the other sub-capabilities of this dimension, we consider lessons of the ablation operations performed by Sugawara et al. (2020) to observe the performance change of the MRC model and quantify them using the corresponding indicative structures. Consider the two examples questions shown in Figure 1. To solve Q 1 , an MRC system just needs to match the words between the question and context. However, Q 2 requires understanding temporal relations among the events (went to the grocery store  walked to the fast-food restaurant) and the verb semantics (walk to means go to). Therefore, Q 2 is more challenging to the MRC system than Q 1 . Please refer to Appendix B for more detailed examples and descriptions of our employed metrics. Based on our assessment framework, the learning value of each sample is also decomposed into four dimensions, namely capability-specific values. In this section, we first uncover the connection between the capability-specific values and model performance from four dimensions and then recalibrate the connection by removing the inter-dimension correlations. Capability-specific value. Given a sample x , we represent it by four capability-specific value (de-noted as { v i ( x ) } 4 i =1 ) to reflect its learning value for 0 . each capability dimension. According to our assessment framework, v i ( x ) can be computed by merging the corresponding metrics { m ji ( x ) } n ( i ) j =1 . Specifically, considering the sensitivity of capability-specific value to different ranges of the metric score, we normalize each raw metric m ji ( x ) from its original scale to range [0 , 1] by the cumulative density function (CDF) as Platanios et al. (2019), which is denoted as (cid:103) m ji ( x ) . In this work, the normalization computes the cumulative density from a higher model performance to ensure that the normalized metric and model performance are negatively correlated. The capability-specific score v i ( x ) is formulated as: v i ( x ) = 1 n ( i ) (cid:80) n ( i ) j =1 (cid:103) m ji ( x ) . Analysis between capability-specific values and model performance. For each sample x , we obtain a 4-dimensional score { v i ( x ) } 4 i =1 . It is necessary to explore the relationship between samples' v i ( x ) and model performance for knowing about what specific capabilities a model has gained and the level of each capability. In this work, we employ BERT-base (Devlin et al., 2018) as the MRC model and train it respectively on training split of datasets SQuADv1 (Rajpurkar et al., 2016), SQuADv2 (Rajpurkar et al., 2018), HotpotQA (Yang et al., 2018) and RACE (Lai et al., 2017). We then analyze the correlations between four capability-specific scores and the model's overall performance on the corresponding dev split. In addition to F 1 , we also report the results of scaled F 1 (denoted as F logits ) by taking the model's confidence to an answer span or candidate into account. F logits is computed as: F logits = (cid:40) F 1  ln ( slog )  ln ( elog ) or F 1  ln ( candlog ) (1) where slog and elog mean the model output logits for start and end token in answer extraction style questions, and candlog represents the largest logits among all candidate answers. Table 2 quantitatively shows the Pearson's correlation coefficients ( r ) between capability-specific values and model performance. From the results, we have the following observations: First, each capability-specific score has a relatively strong correlation with the model performance under a statistically significant guarantee, showing the reasonability of our capability-based assessment framework. Second, F logits shows better relevancy than F 1 , which indicates that F logits is a more appropriate performance measurement in our framework. We further explore the distribution of model performance over different ranges of v i . The distribution diagrams of v 1 and v 4 are shown in Figure 2. There are two inspiring characteristics in this diagram: First, among all the bins of v i , the frequency of prediction results within the intermediate range ( 0 . 4  0 . 6 ) are similar (  50% ). Second, as the v i increases, the frequency of prediction results within a low range ( 0 . 0  0 . 2 ) also increases, while the one of a high range ( 0 . 8  1 . 0 ) decreases. These observations reveal that the samples with high v i can be used in indicative measurements to the corresponding model capability c i . Please refer to Appendix C for more diagrams illustrating this relationship. Inter-dimension decorrelation. Let V = { v i | i = 1 ,    , 4 } . Pairwise correlations of V are illustrated in Figure 3a in a heatmap fashion. The results show a common situation where if a sample is difficult (scored as high capability-specific value) in a dimension, the other dimensions have the same tendency and vice versa. To alleviate the inevitable  ( )  ( )  ( )  ( )   ( )  ( )  ( )  ( )           Figure 4: Illustration of our capability boundary breakthrough curriculum learning (CBBC). correlations and construct a clear value representation for our following specific application scenario ( i . e . CBBC), we eliminate the 4-dimensional capabilities by decorrelation. Specifically, we employ zero-phase component analysis (ZCA) whitening (Bell and Sejnowski, 1996) to diagonalize the covariance matrix while keeping the local information of the samples as much as possible. As shown in Figure 3b, the 4-dimensional capabilities are not highly correlated after inter-dimension decorrelation, which can be in favor of constructing clear indicators for our following data sampling in CBBC. In this section, our universal assessment framework of the model capability is adapted into a specific MRC training scenario to evaluate its usefulness and efficiency. Specifically, we embed our proposed assessment framework into a curriculum learning pipeline and make a capability boundary breakthrough curriculum learning (CBBC) strategy. Based on the assessment framework, our CBBC can guide a model to learn according to its capability boundary by understanding what the model has learned from data ( i . e . capability-specific value v i ) and choosing appropriate samples with comparable learning values from four dimensions. It is worth noting that our competency assessment framework is also applicable to other training pipelines that bal-ance the relationship between the model capabilities and data properties, such as active learning (Set-tles, 2009) and self-training (Mihalcea, 2004) (Ap-pendix E). Figure 4 shows an illustration of the pipeline of our CBBC. Following the original formulation of curriculum learning (Bengio et al., 2009), our CBBC organizes all samples by a sequence of ordered training stages { s } Ss =1 and corresponding training sets { D s } Ss =1 with an easy-to-difficult fashion. The classic curriculum learning works (So-viany et al., 2021) usually consist of two essential components: the performance measurer and the curriculum scheduler. In general, the measurer is used to determine the learning status of a model by evaluating performance, while the scheduler is responsible for deciding when and how to update the curriculum by selecting the input samples. In our work, the measurer and scheduler are implemented by analyzing the multi-dimensional capability levels of the model interpretably and measuring the capability-specific values of the data in a decorrelated way, respectively. That is to say, the only difference between our CBBC and the original curriculum learning design is incorporating MRC capability assessment into the curriculum learning. Without significantly increasing the complexity of the pipeline, our proposed assessment framework can generally empower the MRC training pipeline in a plug-and-play manner. Performance measurer. Recall what we have discussed in Section 2.2 that the samples with high v i can be used in indicative measurements to the corresponding model capability c i . In this work, we use samples scored in the topk of each capability-specific value to assess the corresponding model capability. More precisely, we first evaluate the model on the dev set and obtain an average F logits for each capability on the corresponding topk subset. Then partial correlation (Baba et al., 2004) (denoted as  i ) between dimension v i and F logits is computed to mask the contributions of the other dimensions V\{ v i } . After that, each model capability on stage s is quantified as: c si =  i (cid:80) 4 j =1  j F logits . Empirically, we set k in topk as 32 . Curriculum scheduler. Following the most works (Xu et al., 2020; Platanios et al., 2019), we schedule the curriculum at a linear pace (every 1,000 training iterations). During each curriculum schedule, we enlarge the training set two times until it includes all the samples. The capability upper bound c s +1 i for s + 1 stage by exponential growth: c s +1 i = max { c si , 1 . 0 } . After that, we use criterion v i ( x ) < c s +1 i to construct candidate set D s +1 i for the i -th capability on the state s + 1 , and use absolute contribution of v i to F logits as sampling ratio ( i . e .  1 :  2 :  3 :  4 ) to construct D s +1 . Datasets. We employ two question styles to evaluate our CBBC: answer span extraction and multiple choice. The former consists of SQuADv1 (Ra-jpurkar et al., 2016), SQuADv2 (Rajpurkar et al., 2018) and HotpotQA (Yang et al., 2018), while the latter adopts RACE (Lai et al., 2017). For each dataset, we train and evaluate the model on official training and dev split, respectively. Implementation details. The source code and hyperparameters are included in the supplementary material. We use BERT-base (Devlin et al., 2018) as our backbone model, which is initialized by pre-trained parameters from cased BERT. AdamW (Loshchilov and Hutter, 2017) optimizer with weight decay 5 e  4 and epsilon 8 is used to finetune the model with max sequence length 384 , document stride 128 . The learning rate warms up over the first 10% steps and then decays linearly to 0 for all experiments with training batch size 16 and maximum iteration 40 , 000 . Baseline models. In addition to the BERT-base model, we also consider the following ten baselines. The first two baselines are trained through a pre-defined curriculum learning strategy, which sorts the samples, then feeds them to the model stage-by-stage. B+CL+ V ( M 2 ) sorts the samples by four capability-specific scores in an easy-to-difficult order. B+antiCL+ V ( M 3 ) does like  M 2 , but in a reverse difficult-to-easy order. The following five baselines are trained using our CBBC strategy to maximize the data value in each dimension, respectively. B+C+ v 1 ( M 4 ), B+C+ v 2 ( M 5 ), B+C+ v 3 ( M 6 ) and B+C+ v 4 ( M 7 ) use the corresponding v 1 , v 2 , v 3 and v 4 respectively to perform the competency test and filter samples. B+C+ V corr ( M 8 ) is trained using four correlated scores through CBBC. The following three baselines are devised by embedding other instance scoring methods into our CBBC pipeline. B+C+DatasetMap ( M 9 ), B+C+Forgetting ( M 10 ) and B+C+Predictability ( M 11 ) substitute the capability-specific scores with the confidence score of true answer span (Swayamdipta et al., 2020), number of forgotten events (Toneva et al., 2018) and predictability score (Le Bras et al., 2020), respectively. The last two baselines (denoted as M 12 and M 13 ) are 5863 Name Method SQuADv1 SQuADv2 HotpotQA RACE EM F 1 EM F 1 EM F 1 Acc. start-of-the-art curriculum learning pipelines consisting of DRCA (Xu et al., 2020) and CBCL (Pla-tanios et al., 2019). Finally, our full model is trained using four decorrelated scores through CBBC instead. The critical difference between the full model and M 8 is the decorrelation operation. Quantitative Results. We present a summary of our quantitative results in Table 3. As shown in the table, we have the following key observations. On the one hand, our proposed competency framework does benefit the MRC learning efficiency in either a single or multiple dimensions. For example, when using a pre-defined curriculum strategy, M 2 achieves EM and F 1 far beyond M 1 , highlighting that our quantification to data properties properly estimates the learning value contained in the data. M 3 degrades performance w.r.t. M 1 , demonstrating that the learning strategy from easy to difficult samples is more reasonable than the reverse. When equipped with our CBBC, all models of M 4 , M 5 , M 6 and M 7 achieve improvements w.r.t. M 1 on four datasets, which indicates the significant contribution of each capability dimension on gradually increasing the model capability. In particular, among the four different dimensions, M 7 has the best result, indicating that understanding sentences is a relatively more important capability for MRC. M 8 outperforms all the models except for ours. This demonstrates that our CBBC can maximize the learning value of the data sample to increase an MRC model's capability. On the other hand, our framework wins other scoring methods and curriculum learning pipelines by a considerable margin. Although M 9 , M 10 , M 11 , M 12 and M 13 achieve substantial improvements on four datasets w.r.t. M 1 , their perfor-0 5k 10k 15k 20k Steps 0.2 0.3 0.4 0.5 0.6 F l o g i t s OursB+CL+B+antiCL+B 0 5k 10k 15k 20k Steps 0.2 0.3 0.4 0.5 0.6 F l o g i t s OursB+C+ v 4 B+C+ v 3 B+C+ v 2 B+C+ v 1 Figure 5: Illustration of performance (smoothed by averaging F logits every 32 steps) of various baseline models on HotpotQA dev split as training progresses. during (b) A 5-dimensional map of MRC model capabilities on step 50, 1k, 2k, 20k. mances are still worse than our full model. These results verify that our proposed framework can assess the model capability more correctly and make better use of the learning value within data. Finally, our full model achieves significantly higher EM , F 1 and Acc. compared to all other baselines, demonstrating the necessity of the decorrelation between capability-specific scores. Its superior performance roots from constructing a decorrelated value representation of each dimension for our CBBC learning strategy. Overall, compared to M 1 , our full model achieves tremendous improvement of EM / F 1 up to 11.22% / 8.71% on the average of three answer extraction style datasets. Qualitative Results. Figure 5 shows the performance of baselines on the HotpotQA dev set. There are two observations worth noting here. First, the performance of our full model lies consistently on top of the other baseline models during the whole training stage. This result shows that CBBC can make the model more prepared for complex samples by enlarging its capability boundary step by step. Second, the performance plot of the baseline model with v 4 sits on top of other baselines with v 1 , v 2 , and v 3 from the beginning of training to the end. This result highlights the main contribution of v 4 (understanding sentences) to the final performance. ure 6. First, among 4-dimensional capability, the c 3 ( i . e . understanding words) has the largest initial value. A possible explanation is that pre-trained BERT has a fair amount of prior knowledge obtained from unlabeled corpus, which concentrates more on semantic understanding of words. Second, the capability c 1 increases at the fastest speed as the training progresses. Interestingly, the model M 4 based on v 1 does not seem improving accordingly as the capability c 1 increases. The possible reason could be that the superficial structure is easy to learn from samples but makes a limited contribution to the final performance. Please refer to Appendix D for the results of other MRC models. Annotation specification. we ask three annotators to answer ( 100  4 = 400 ) questions randomly sampled from four datasets, consisting of SQuADv1, SQuADv2, HotpotQA, and RACE. Using only our proposed four capabilities, they first read the context, question, and gold standard answer (the correct candidate answer under multiple-choice situation), and then choose the evidence sentences in context. After that, they respectively label the subclasses of four major capabilities as 1 (required) or 0 (not required). Please refer to Appendix A for more details about human annotation. Annotation results. In the annotation of required capabilities, the inter-annotator agreement is 75.33% for all 400 samples. We use the average of three corresponding annotator labels as the final human judgments for a specific sub-capability required by the question. Finally, a sample will be annotated ( 2 + 2 + 2 + 6 = 12 ) human ratings. Table 4 summarizes the correlations between human judgments and capability-specific scores of samples. The relatively strong correlations on all four dimensions indicate that our employed heuristic metrics can reasonably approximate the learning value contained in the samples. Analytic approaches to MRC capability. Some works performed skill-based analyses for the MRC model. In the scientific question domain, Clark et al. (2018) constituted the ARC benchmark, which requires far more powerful knowledge and reasoning than previous benchmarks. In a generalizable definition, Sugawara et al. (2017) proposed a set of 10 skills for MCTest (Richardson et al., 2013). The others focused more on the analysis of the MRC dataset itself. For example, Sugawara et al. (2020) proposed a semi-automated, ablation-based methodology to assess the capacities of datasets. Rajpurkar et al. (2016) analyzed their proposed datasets using several types of reasoning, e . g . lexical and syntactic variation, and multiple sentence reasoning. Nevertheless, they require too costly human efforts and ignore that the model capability changes as training progresses. Data selection for debiased representations. Some works proposed different criteria to score instances according to the model response to input. Swayamdipta et al. (2020) built data maps using training dynamics measures for scoring data samples. Toneva et al. (2018) also employed the number of forgotten events to measure a sample, which was misclassified during a later epoch of training, despite being classified correctly earlier. The others (Le Bras et al., 2020) used adversarial filtering algorithms to rank instances based on their predictability. However, these approaches require training a model once in advance on the dataset to obtain the corresponding training dynamics, which is computationally expensive, especially when using a large model. We design a competency assessment framework for MRC capabilities, which describes model skills in an explainable and multi-dimensional manner. By leveraging the framework, we further uncover and disentangle the connections between various data properties and model performance on a specific task, as well as propose a capability boundary breakthrough curriculum (CBBC) strategy to maxi-5865 mize the data value and improve training efficiency. The experiments performed on four benchmark datasets verified that our approach can significantly improve the performance of existing MRC models. Our work shows a deep understanding of model capabilities and data properties helps monitor the model skills during training and improves learning efficiency. Our framework and learning strategy are also generally applicable to other NLP tasks. This work has been supported in part by the National Key Research and Development Program of China (2018AAA0101900), Zhejiang NSF (LR21F020004), Key Research and Development Program of Zhejiang Province, China (No. 2021C01013), Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies, Chinese Knowledge Center of Engineering Science and Technology (CKCEST).
abstain abstain abstain abstain method objective objective objective abstain abstain abstain objective objective abstain objective method abstain abstain abstain abstain abstain result abstain abstain abstain abstain method abstain abstain method objective objective objective abstain abstain method method abstain objective objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain other abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method method abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method abstain method abstain abstain method abstain abstain method abstain abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other other other other other other other other other other other other other method objective result result abstain other