Knowledge of difficulty level of questions helps a teacher in several ways, such as estimating students' potential quickly by asking carefully selected questions and improving quality of examination by modifying trivial and hard questions. Can we extract such benefits of instance difficulty in Natural Language Processing? To this end, we conduct I nstanceL evel D ifficulty A nalysis of E valuation data (ILDAE) in a large-scale setup of 23 datasets and demonstrate its five novel applications: 1) conducting efficient-yet-accurate evaluations with fewer instances saving computational cost and time, 2) improving quality of existing evaluation datasets by repairing erroneous and trivial instances, 3) selecting the best model based on application requirements, 4) analyzing dataset characteristics for guiding future data creation , 5) estimating Out-of-Domain performance reliably . Comprehensive experiments for these applications lead to several interesting results, such as evaluation using just 5% instances (selected via ILDAE) achieves as high as 0 . 93 Kendall correlation with evaluation using complete dataset and computing weighted accuracy using difficulty scores leads to 5 . 2% higher correlation with Out-of-Domain performance. We release the difficulty scores 1 and hope our work will encourage research in this important yet understudied field of leveraging instance difficulty in evaluations. Transformer-based language models (Devlin et al., 2019; Liu et al., 2019; Clark et al., 2020) have improved state-of-the-art performance on numerous natural language processing benchmarks (Wang et al., 2018, 2019; Talmor et al., 2019); however, recent studies (Zhong et al., 2021; Sagawa et al., 2020) have raised questions regarding whether these models are uniformly better across all instances. This has drawn attention towards instance-1 https://github.com/nrjvarshney/ILDAE Figure 1: Illustrating five applications of Instance-Level Difficulty Analysis of Evaluation data (ILDAE). level analysis of evaluation data (Rodriguez et al., 2021; Vania et al., 2021; Mishra and Arunkumar, 2021) which was previously limited to training data (Swayamdipta et al., 2020; Xu et al., 2020; Mishra and Sachdeva, 2020). Furthermore, it is intuitive that not all instances in a dataset are equally difficult . However, instance-level difficulty analysis of evaluation data (ILDAE) has remained underex-plored in many different ways: what are the potential applications and broad impact associated with ILDAE? In this work, we address the above question by first computing difficulty scores of evaluation instances (section 2) and then demonstrating five novel applications of ILDAE (Figure 1). 1. Efficient Evaluations: We propose an approach of conducting efficient-yet-accurate evaluations. Our approach uses as little as 5% evaluation instances (selected via ILDAE) to achieve up to 0 . 93 Kendall correlation with evaluations conducted using the complete dataset . Thus, without considerably impacting the effectiveness of evaluations, our approach saves computational cost and time. 2. Improving Evaluation Datasets: We first show that trivial' and erroneous' instances can be identified using our difficulty scores and then present a model-and-human-in-the-loop technique to modify/repair such instances resulting in improved quality of the datasets. We instantiate it with SNLI dataset (Bowman et al., 2015) and show that on modifying the trivial instances, the accuracy (averaged over 27 models) drops from 77.58% to 26.49%, and on repairing the erroneous instances, it increases from 13.65% to 69.9%. Thus, improving the dataset quality. 3. Model Analysis: We divide evaluation instances into different regions based on difficulty scores and analyze models' performance in each region. We find that a single model does not achieve the highest accuracy in all difficulty regions. This implies that the model that achieves best overall performance may not be the best in each difficulty region. Such analyses could benefit in model selection. For instance, in scenarios where a system is expected to encounter hard instances, the model that performs well in high difficulty regions could be selected. 4. Dataset Analysis: ILDAE reveals several important characteristics of datasets that can be leveraged in future data creation processes. For instance, we find that in SNLI and MNLI datasets, contradiction' instances receive lower average difficulty score than en-tailment' and neutral' instances . Thus, more difficult contradiction examples can be created to develop high-quality task-specific datasets. 5. OOD Correlation: We compute weighted accuracy leveraging the difficulty scores and show that it leads to 5 . 2% higher Kendall correlation with Out-of-Domain (OOD) performance than the standard accuracy that treats all instances equally . Thus, ILDAE helps in getting a more reliable estimation of models' OOD performance. Interpretation: Human perception of difficulty may not always correlate well with machine's interpretation. Thus, difficulty scores must be computed via a model-in-the-loop technique so that they directly reflect machine's interpretation. predictive correctness since a difficult instance less likely to be predicted correctly than a relatively easier instance. We incorporate the above desiderata and consider model's prediction confidence in the ground truth answer (indicated by softmax probability assigned to that answer) as the measure of its predictive correctness. Furthermore, we compile an ensemble of models trained with varying configurations and use their mean predictive correctness to compute the difficulty scores. We do this because model's predictions fluctuate greatly when its training configuration is changed (Zhou et al., 2020; McCoy et al., 2020) and relying on predictive correctness of only one model could result in difficulty scores that show poor generalization. To this end, we use the following three training configurations to compile predictions from an ensemble of models: Data Size: Instances that can be answered correctly even with few training examples are inherently easy and should receive lower difficulty score than the ones that require a large training dataset. To achieve this, we train a model each with 5, 10, 15, 20, 25, 50, and 100 % of the total training examples and include them in our ensemble. Data Corruption: Instances that can be answered correctly even with some level of corrup-tion/noise in the training dataset should receive low difficulty score. To achieve this, we train a model each with different levels of noise (2, 5, 10, 20, 25% of the examples) in the training data, and add them to our ensemble. For creating noisy examples, we randomly change the ground-truth label in case of classification and multiple-choice datasets and change the answer span for extractive QA datasets. Training Steps: Instances that can be consistently answered correctly from the early stages of training should receive low difficulty score. Here, we add a model checkpoint after every epoch during training to our ensemble. This results in a total of N = E  (7+5) models in our ensemble where E corresponds to the number of training epochs, and 7 , 5 correspond to the number of data size and data corruption configurations respectively. We infer the evaluation dataset using these N models and calculate the average predictive correctness for each instance. Finally, we compute the difficulty score by subtracting this averaged correctness value from 1 . This ensures that an instance that is answered correctly with high confidence under many training configurations gets assigned a low difficulty score as it corresponds to an easy instance. In contrast, an instance that is often answered incorrectly gets assigned a high difficulty score. Algorithm 1 summarizes this approach. We use RoBERTa-large model (Liu et al., 2019) for this procedure and train each model for E = 10 epochs, resulting in N = 120 predictions for each evaluation instance. Our difficulty computation method is general and can be used with any other model or configurations; we use RoBERTa-large as it has been shown to achieve high performance across diverse NLP tasks (Liu et al., 2019). In addition, we show that difficulty scores computed using our procedure also generalize for other models (3.5.1). We note that difficulty computation is not our primary contribution. Prior work (Swayamdipta et al., 2020; Xu et al., 2020) has explored different ways to achieve this. However, our approach uses 120 predictions from models trained with different configurations for its computation and hence is more reliable. Equipped with difficulty scores of evaluation instances, we now demonstrate five applications of ILDAE in the following sections. Success of BERT (Devlin et al., 2019) has fostered development of several other pre-trained language models such as RoBERTa (Liu et al., 2019), XLNet (Yang et al., 2019b), DistilBERT (Sanh et al., 2019), ALBERT (Lan et al., 2020). Though, it has resulted in the availability of numerous model options for a task, comparing the performance of such a large number of models has become computationally expensive and time-consuming. For example, in real-world applications like online competitions, the naive approach that evaluates candidate models on the entire test dataset would be too expensive because they receive thousands of model submissions and contain a sizable number of evaluation instances. Moreover, some applications also require additional evaluations to measure Out-of-Domain generalization and robustness making it even more expensive. Can we make the evaluations efficient ? We address the above question and explore if the performance of candidate models can be accurately compared with a carefully selected smaller subset of the evaluation dataset. Reducing the number of instances would save computational cost and make the evaluations efficient. To this end, we propose an approach that selects evaluation instances based on their difficulty scores. We compare performance of candidate models only on these selected instances and show that without considerably impacting the result of evaluations, our approach saves computational cost and time. Instance Selection: We argue that the instances with extreme difficulty scores (very low and very high scores) would not be very effective in distinguishing between the candidate models . This is because the former instances are trivial and would be answered correctly by many/all candidate models, while the latter ones are hard and would be answered correctly by only a few/none models. Therefore, given a budget on the number of evaluation instances, we select a majority of them with moderate difficulty scores. However, to distinguish amongst very weak and amongst very strong candidates, we also include a small number of instances with extreme difficulty scores. Figure 2 illustrates our approach. ficulty scores are pre-computed. Furthermore, we do not compute separate difficulty scores for each candidate model as it would defy the sole purpose of efficient' evaluations. Instead, we compute difficulty scores using only one model (RoBERTa-large) and exclude it from the list of candidate models for a fair evaluation of our approach. For our instance selection approach to work in this setting, the difficulty scores should generalize for other models. We empirically prove this generalization capability and demonstrate the efficacy of our efficient evaluations approach in 3.5. Performance Metric: We measure the efficacy of an instance selection technique by computing accuracies of candidate models on the selected instances and calculating their Kendall's correlation (Kendall, 1938) with accuracies obtained on the full evaluation dataset. High correlation implies that the performance scores obtained using the selected instances display the same behavior as the performance scores obtained using the complete dataset. Hence, high correlations values are preferred. Datasets: We experiment with a total of 23 datasets across Natural Language Inference, Duplicate Detection, Sentiment Analysis, Question Answering, Commonsense Reasoning, and several other tasks. Refer to Appendix section B for an exhaustive list of datasets for each task. Candidate Models: We use BERT (Devlin et al., 2019), DistilBERT (Sanh et al., 2019), ConvBERT (Jiang et al., 2020) , XLNET (Yang et al., 2019a), SqueezeBERT (Iandola et al., 2020), ELECTRA (Clark et al., 2020) in our experiments. We also use different variants of ConvBert (small, medium-small, base) and ELECTRA (small, base) models. For comprehensive experiments, we train each of the above models with training data of three different sizes ( 2 k , 5 k , and 10 k examples) resulting in 27 candidate models for each dataset. We intentionally exclude RoBERTa from this list as we use it for computing the difficulty scores. the proposed instance selection approach with the following baselines: Random Selection : Select a random subset of instances from the evaluation dataset. Heuristic Selection : Select instances based on the length heuristic (number of characters in the instance text) instead of the difficulty scores. Adaptive evaluation (Weiss, 1982) is used in educational settings for evaluating performance of students. It uses Item Response Theory (IRT) (Baker 3415 % Instances  0.5% 1% 2% 5% 10% 20% Dataset  Random Heuristic Proposed Random Heuristic Proposed Proposed Proposed Proposed Proposed SNLI 0 . 55 0 . 09 0 . 38 0 . 17 0.68 0 . 13 0 . 68 0 . 05 0 . 58 0 . 08 0.78 0 . 08 0 . 83 0 . 04 0 . 88 0 . 04 0 . 91 0 . 01 0 . 93 0 . 02 PAWS Wiki 0 . 67 0 . 07 0 . 68 0 . 04 0.78 0 . 06 0 . 73 0 . 05 0 . 78 0 . 02 0.86 0 . 05 0 . 89 0 . 02 0 . 91 0 . 03 0 . 95 0 . 01 0 . 96 0 . 01 AgNews 0 . 12 0 . 26 0 . 14 0 . 27 0.47 0 . 05 0 . 25 0 . 34 0 . 41 0 . 14 0.52 0 . 1 0 . 65 0 . 07 0 . 75 0 . 06 0 . 8 0 . 04 0 . 89 0 . 03 QNLI 0 . 41 0 . 1 0 . 44 0 . 04 0.48 0 . 13 0 . 57 0 . 04 0 . 55 0 . 1 0.57 0 . 07 0 . 7 0 . 06 0 . 78 0 . 06 0 . 85 0 . 03 0 . 91 0 . 03 MRPC 0 . 04 0 . 09  0 . 03 0 . 18 0.21 0 . 16  0 . 02 0 . 09 0 . 05 0 . 2 0.29 0 . 21 0 . 36 0 . 15 0 . 45 0 . 08 0 . 58 0 . 12 0 . 65 0 . 14 SocialIQA 0 . 19 0 . 09 0 . 15 0 . 29 0.37 0 . 17 0 . 34 0 . 07 0 . 28 0 . 21 0.4 0 . 09 0 . 58 0 . 1 0 . 67 0 . 04 0 . 75 0 . 08 0 . 81 0 . 05 QQP 0 . 63 0 . 06 0 . 64 0 . 05 0.65 0 . 05 0 . 74 0 . 03 0 . 74 0 . 01 0.77 0 . 06 0 . 84 0 . 04 0 . 9 0 . 04 0 . 94 0 . 04 0 . 95 0 . 01 DNLI 0 . 58 0 . 05 0 . 59 0 . 1 0.58 0 . 11 0 . 68 0 . 1 0 . 71 0 . 04 0.76 0 . 07 0 . 84 0 . 04 0 . 92 0 . 05 0 . 94 0 . 03 0 . 96 0 . 01 COLA     0 . 01 0 . 18 0.25 0 . 26 0 . 24 0 . 45 0 . 41 0 . 41 0 . 63 0 . 23 0 . 75 0 . 08 0 . 78 0 . 02 SWAG 0 . 72 0 . 04 0 . 66 0 . 02 0.75 0 . 06 0 . 79 0 . 03 0 . 77 0 . 03 0.78 0 . 05 0 . 86 0 . 03 0 . 89 0 . 02 0 . 93 0 . 01 0 . 95 0 . 01 PAWS QQP    0 . 13 0 . 24 0 . 36 0 . 05 0.34 0 . 13 0 . 55 0 . 19 0 . 8 0 . 05 0 . 84 0 . 03 0 . 87 0 . 04 MNLI 0 . 7 0 . 04 0 . 71 0 . 03 0.73 0 . 07 0 . 8 0 . 02 0 . 8 0 . 04 0.82 0 . 08 0 . 89 0 . 03 0 . 93 0 . 02 0 . 95 0 . 02 0 . 96 0 . 01 Adv. NLI R1 0 . 0 0 . 08  0 . 07 0 . 06 0.17 0 . 27 0 . 02 0 . 13 0.09 0 . 11 0 . 08 0 . 2 0 . 13 0 . 18 0 . 3 0 . 18 0 . 47 0 . 05 0 . 59 0 . 05 Adv. NLI R2  0 . 08 0 . 04 -0.01 0 . 06  0 . 08 0 . 16  0 . 08 0 . 07 0.02 0 . 03  0 . 03 0 . 21 0 . 0 0 . 12 0 . 17 0 . 03 0 . 26 0 . 11 0 . 42 0 . 15 Adv. NLI R3  0 . 15 0 . 12 0.15 0 . 1 0 . 1 0 . 21  0 . 03 0 . 06 0 . 07 0 . 1 0.1 0 . 11 0 . 18 0 . 16 0 . 12 0 . 17 0 . 31 0 . 15 0 . 58 0 . 05 SST-2    0 . 08 0 . 15 0 . 16 0 . 35 0.29 0 . 25 0 . 4 0 . 2 0 . 52 0 . 16 0 . 65 0 . 13 0 . 81 0 . 08 ARC Easy    0 . 0 0 . 2  0 . 03 0 . 12 0.42 0 . 19 0 . 47 0 . 19 0 . 59 0 . 13 0 . 6 0 . 14 0 . 74 0 . 11 ARC Diff       0 . 15 0 . 29 0 . 28 0 . 13 0 . 33 0 . 31 0 . 3 0 . 26 Abductive NLI 0 . 08 0 . 26 0.17 0 . 05 0 . 16 0 . 09 0 . 19 0 . 19 0 . 26 0 . 08 0.3 0 . 07 0 . 42 0 . 13 0 . 57 0 . 08 0 . 61 0 . 07 0 . 68 0 . 07 Winogrande  0 . 19 0 . 11  0 . 03 0 . 06 0.0 0 . 17  0 . 11 0 . 09  0 . 05 0 . 12 0.11 0 . 15 0 . 09 0 . 14 0 . 03 0 . 1 0 . 14 0 . 1 0 . 21 0 . 14 CSQA 0 . 29 0 . 11 0 . 28 0 . 1 0.31 0 . 07 0 . 36 0 . 14 0 . 37 0 . 08 0.39 0 . 09 0 . 49 0 . 09 0 . 69 0 . 08 0 . 78 0 . 04 0 . 83 0 . 05 QuaRel       0 . 32 0 . 26 0 . 33 0 . 25 0 . 39 0 . 07 0 . 51 0 . 1 QuaRTz       0 . 34 0 . 19 0 . 36 0 . 04 0 . 34 0 . 12 0 . 37 0 . 08 Average 0 . 28 0 . 1 0 . 3 0 . 11 0.39 0 . 13 0 . 31 0 . 11 0 . 35 0 . 11 0.43 0 . 14 0 . 46 0 . 17 0 . 58 0 . 11 0 . 66 0 . 08 0 . 72 0 . 07 Table 1: Kendall correlation with full evaluation dataset achieved by various instance selection approaches for different percentage of instances. Each cell shows the mean and standard deviation obtained from 5 different runs.  cell indicates 0 selected instances. We show the expanded version of this table in supplementary. and Kim, 2004) from psychometrics that requires a large number of subjects and items to estimate system parameters (Lalor et al., 2016, 2018). Moreover, adaptive evaluation is computationally very expensive as it requires calculating performance after each response to select the next instance based on the previous responses of the subject. Thus, it is not fit for our setting as we intend to improve the computational efficiency. In contrast, our approach is much simpler and efficient as it does not incur any additional cost during the evaluation. We first study generalization of our computed difficulty scores and then show the efficacy of the proposed instance selection approach in conducting efficient evaluations. In Figure 3, we plot accuracy (averaged over all 27 candidate models) against difficulty scores (com-puted using RoBERTa-large). We find that with the increase in difficulty score, the accuracy consistently decreases for all datasets. We also study this behavior for each individual candidate model and find results supporting the above observation 2 (Fig-2 Further details are in appendix ure 6). This proves that the difficulty scores follow the desiderata mentioned in Section 2.1 for other models also and our intuitions behind instance selection for conducting efficient evaluations hold true. Note that these difficulty scores are computed using a specific model but our approach is general and will replicate this generalization capability if used with any other model. Table 1 shows Kendall correlation with full dataset evaluation achieved by various instance selection approaches for different percentages of instances. Proposed Approach Outperforms Baselines: Our proposed approach is consistently better than the Random and Heuristic approaches. For instance, with just 0 . 5% and 1% evaluation instances, our approach outperforms the baseline methods by  30% and  22 . 8% respectively. We show the expanded version of this table and performance of other instance selection techniques in Appendix. Correlation Change with % of Instances: As expected, Kendall correlation consistently increases as a higher percentage of instances are selected for evaluation. In case of SNLI, PAWS Wiki, QQP, DNLI, SWAG, and MNLI, just 2% instances 3416 are sufficient to achieve correlation of > 0 . 8 . For most datasets, with just 20% of the evaluation instances, our approach achieves Kendall correlation of > 0 . 8 . This suggests that the evaluations can be conducted with fewer instances without significantly compromising the accuracy of comparison. We further analyze performance of our approach for higher percentage of instances in Table 7. Thus, for practical settings where candidate models can't be compared on the entire dataset due to computational and time constraints, evaluating only on the selected instances can result in fairly accurate performance comparison. Performance on Multiple-Choice QA datasets: Though, we perform better than the baselines approaches on almost all datasets, we achieve a lower correlation value for multiple-choice question answering datasets such as QuaRel, QuaRTz, and Winogrande. We attribute this behavior to the close scores (accuracies) achieved by many candidate models even in case of full dataset evaluation. Thus, it is difficult to differentiate such models as they achieve nearly the same performance. Furthermore, in some difficult datasets such as Adversarial NLI (R1, R2, and R3), ARC Difficult, and Winogrande, many candidate models achieve accuracies very close to the random baseline ( 33% for NLI, 50% for Winogrande). So, comparing their performance even with full dataset does not provide any significant insights. Recent years have seen a rapid increase in the number and size of NLP datasets. Crowd-sourcing is a prominent way of collecting these datasets. Prior work (Gururangan et al., 2018; Tan et al., 2019; Mishra et al., 2020) has shown that crowd-sourced datasets can contain: (a) erroneous instances that have annotation mistakes or ambiguity, (b) too many trivial instances that are very easy to answer. This hampers the quality of the dataset and makes it less reliable for drawing conclusions. Can difficulty scores aid in improving the quality of evaluation datasets? We first show that erroneous and trivial instances can be identified using the difficulty scores and then present a human-and-model-in-the-loop tech-Dataset Identifying Erroneous and Trivial Instances: We inspect 50 instances each with very high and very low difficulty scores and find that a significant percentage of the former are either mislabeled or contain ambiguity and the latter are too easy to be answered. Table 2 shows examples of erroneous instances from SNLI, Winogrande, CSQA, and Abductive NLI. We find 72% of the inspected SNLI instances to be erroneous. Furthermore, we find that some high difficulty score instances are actually difficult even for humans because they require abilities such as commonsense reasoning. Table 4 (appendix) shows such instances. We also provide examples of trivial instances (Table 6) and note that such instances are trivial from model's perspective as they can be answered correctly (with high confidence) by simply latching on to some statistical cues present in the training data. Technique: Since the trivial instances are too easy to be answered, we propose to modify them in an adversarial way such that they no longer remain trivial. Specifically, we include a human-in-the-loop who needs to modify a trivial instance in a label-preserving manner such that the modified version fools the model into making an incorrect prediction. For adversarial attack, we use the strongest model from our ensemble of 120 models. It has two key differences with the standard adversarial data creation approach presented in (Nie et al., 2020; Kiela et al., 2021): (a) it requires modifying an already existing instance instead of creating a new instance from scratch. (b) it does not increase the size of the evaluation dataset as we replace an already saturated instance (trivial) with its improved not-trivial version. We use a human instead of leveraging automated ways to modify the trivial instances because our objective is to improve the quality of instances and prior work has shown that these automated techniques often result in unnatural and noisy instances. Therefore, such techniques could be cost-efficient but might not solve the sole purpose of improving quality. To further improve the quality, we provide instances with very high difficulty score (potentially erroneous) and ask a human to repair them such that the repaired versions follow the task definition. The human can either change the instance text or its answer to achieve the goal. Note that this scenario is model-independent. Table 3 shows original and modified instances from SNLI. Top two examples correspond to the trivial instances where the human modified the hypothesis in a label-preserving manner such that it fooled the model into making incorrect prediction. The bottom two correspond to the mislabeled instances where the human rectified the label. Figure 4 compares the performance of models on the original instances and the their modified/repaired versions. As expected, the performance drops on the previously trivial instances as they are no longer trivial and improves on the previously erroneous instances. We release the improved version of the dataset compiled via our technique. ILDAE reveals several useful characteristics of datasets such as which class label has the easiest instances. We study this for NLI datasets: SNLI, MNLI, DNLI, and Adversarial NLI (Figure 5). For SNLI and MNLI, we find that the contradiction instances receive lower average difficulty score than entailment and neutral instances. For Adversarial NLI, the order is reversed. For DNLI, all the labels get assigned nearly the same average difficulty. Such analysis can serve as a guide for future data creation as it indicates for which type of instances more data collection effort needs to be invested. It can also be used to compare average difficulty at dataset level. Furthermore, a new harder task-specific benchmark can be created by combining high difficulty instances from all the datasets of that task. We divide the evaluation instances into different regions based on the difficulty scores and analyze models' performance in each region. We find that a single model does not achieve the highest accuracy across all regions. Figure 6 illustrates this pattern for SNLI dataset. This implies that the model that achieves the highest performance on easy instances may not necessarily achieve the highest performance on difficult instances. The similar pattern is observed for other datasets (refer appendix). Such analysis would benefit in model selection. For instance, in scenarios where a system is expected to encounter hard instances, we can select the model that has the highest accuracy on instances of difficult regions. Whereas, for scenarios containing easy instances, the model that has the highest accuracy on instances of easy regions. Large pre-trained language models can achieve high In-Domain performance on numerous tasks. However, it does not correlate well with OOD performance (Hendrycks and Dietterich, 2019; Hendrycks et al., 2020). To this end, we present an approach to compute a weighted accuracy that shifts away from treating all the evaluations instances equally and assigns weight based on their difficulty scores. We define the weight w i of an Figure 7: Comparing Kendall correlation of standard unweighted accuracy and weighted accuracy with OOD accuracy. instance i with difficulty score d i as: w i = 1 +   d i N +   (cid:80) N j =1 d j where N corresponds to the total number of evaluation instances, and  is a hyper-parameter that controls influence of difficulty score on the weight. Then, weighted accuracy W is simply: W = N (cid:88) i =1 w i  v i where v i is 1 when the model's prediction is correct else 0. This implies that high accuracy may not always translate to high weighted accuracy. We take SNLI as the in-domain dataset and MNLI, DNLI, and HANS (McCoy et al., 2019) (Constituent, Lexical Overlap, Subsequence) as OOD datasets. We calculate unweighted and weighted accuracy of the 27 models (described in Section 3.3) and compare their Kendall correlation with the accuracy on OOD datasets. Figure 7 shows this comparison. It can be observed that weighted accuracy shows 5 . 2% higher correlation with OOD performance that the standard accuracy. Most improvement is observed in hard datasets i.e. HANS. Thus, weighting instances based on their difficulty score is more informative than the standard accuracy that treats all instances equally. We conducted Instance-Level Difficulty Analysis of Evaluation data (ILDAE) in a large-scale setup of 23 datasets and presented its five novel applications. With these applications, we demonstrated 3419 ILDAE's impact in several important areas, such as conducting efficient evaluations with fewer instances, improving dataset quality, and estimating out-of-domain performance reliably. We release our computed difficulty scores and hope that our encourage research in this important yet understudied field of leveraging instance difficulty in evaluations. We use existing public-domain text datasets, such as SNLI, Winogrande, and ARC, and follow the protocol to use and adapt research data to compute instance-level difficulty scores. We will release the computed difficulty scores, but will not share the original source data. We recommend readers to refer to the original source research papers. Any bias observed in difficulty scores computed using our methods can be attributed to the source data and our computation functions. However, no particular socio-political bias is emphasized or reduced specifically by our methods. We thank the anonymous reviewers for their insightful feedback. This research was supported by DARPA SAIL-ON and DARPA CHESS programs.
abstain method other abstain abstain abstain method other other abstain abstain abstain objective objective result objective abstain objective result objective method result objective objective objective objective result objective result objective objective abstain abstain abstain method method method method method abstain method method abstain method method method method abstain abstain abstain method method method method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain other other other abstain objective abstain method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result result abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain method method abstain abstain abstain abstain abstain objective objective method method method method abstain abstain other other