Leaderboards are widely used in NLP and push the field forward. While leaderboards are a straightforward ranking of NLP models, this simplicity can mask nuances in evaluation items (examples) and subjects ( NLP models). Rather than replace leaderboards, we advocate a re-imagining so that they better highlight if and where progress is made. Building on educational testing, we create a Bayesian leaderboard model where latent subject skill and latent item difficulty predict correct responses. Using this model, we analyze the ranking reliability of leaderboards. Afterwards, we show the model can guide what to annotate, identify annotation errors, detect overfitting, and identify informative examples. We conclude with recommendations for future benchmark tasks. Leaderboard evaluationsfor better or worseare the de facto standard for measuring progress in question answering (Rajpurkar et al., 2016) and in many NLP tasks (Wang et al., 2019a). An unfortunate side effect of leaderboard popularity is SOTA -chasing, often at the expense of carefully inspecting data and models (Linzen, 2020). For example, the same super-human models that top question answering leaderboards (Najberg, 2018) often fail spectacularly (Feng et al., 2018; Wallace et al., 2019a) by learning non-generalizable statistical patterns (McCoy et al., 2019; Niven and Kao, 2019). Finally, focusing solely on metrics conflates progress on a specific task with progress on real-world NLP problems behind the task (Bender and Koller, 2020). Plainly, focusing on headline SOTA numbers provide(s) limited value for scientific progress absent insight into what drives them and where they fail (Lipton and Steinhardt, 2019). In this work we take leaderboards as they are, and imagine how they might better support research. Leaderboards establish differences between models on a fixed task. Hence, leaderboards should enable and encourage the comparison of models and inspection of examples. And leaderboards should also signal when they have outlived their usefulness (Boyd-Graber and Brschinger, 2020). To help focus attention on examples and models of interest, we propose Difficulty and Ability Discriminating ( DAD ) leaderboards that explicitly model both task and submissions jointly , rather than either in isolation. 1 DAD 's underlying model is based on 1 Source code, data, and visualizations at irt.pedro.ai. Item Response Theory (Lord et al., 1968; Baker, 2001, IRT , reviewed in 2), a widely used (van Rijn et al., 2016) alternative in educational testing to simple summary statistics (Edgeworth, 1888). DAD can explicitly identify the difficulty and discriminability of items (Figure 1), 2 which in turn can lead to a more nuanced ranking of models, identifying poor items, and better understanding of a dataset and task. Throughout the paper, we use the question answering ( QA ) benchmark SQ u AD 2.0 (Rajpurkar et al., 2018). For example, DAD can identify questions that are challenging to models and questions that are wrong (incorrectly annotated). In addition to better understanding datasets, it is also helpful for efficiently selecting evaluation items to annotate. We conclude with recommendations for future leaderboards (7) and discuss where IRT in NLP can go next (8). Leaderboards are a product of the metrics, evaluation data, and subjects (machine or human) who answer items (Figure 2). For concreteness, let's assume that we have a question-answering task and two subjects: Ken, who is good at trivia, and Burt, who is not. In the simplest IRT models, each subject j has a random variable  j corresponding to their skill: Ken's is big, Burt's is small. But you cannot know that until you start asking them questions of varying difficulty  i . Harder questions have a higher difficulty (what is the airspeed of an unladen swallow) than easy ones (who is buried in Grant's tomb). The bigger the margin between a subject's skill  j and an item's difficulty  i ,  j   i , the more likely that subject j responds correctly p i,j ( r i,j = 1) . This is the simplest IRT model, which we call IRT -base . Generally, given n test items X = ( X 1 , . . . , X n ) and m subjects S = ( S 1 , . . . , S m ) , where each subject answers every item, we want to estimate subject skills and item difficulties. To discover the random variables that best explain the data, we turn to probabilistic inference (Pearl, 1988). Two additional random variables further improve DAD : discriminability  i and feasibility  i . We first consider discriminability and the margin between a question's difficulty  i and a subject's skill  j . A discriminative question is challenging but can still be answered correctly by a strong subject. If Ken's ability is higher than most items' difficulty (  j   i is large), item discriminability multiplies this gap by  i in a model called IRT -disc . Questions with low  i are low quality: they have annotation error or do not make sense. Another way of capturing poor quality questions is the feasibility  i . For example, if the question who was the first president has the answer Rajendra Prasad, the question has an unstated implicit assumption that subjects must guess what country or company the question is about. In the model IRT -feas , if a large fraction of subjects all get an item wrong, everyone's probability of getting the item right is capped at  i . In NLP terms, 1   i corresponds to the prevalence of annotation errors that lead to unsolvable items. Having introduced all of the constituent elements of the model, we can now present the full generative model: 1. For each subject j : (a) Draw skill  j  N (   ,   1  ) 2. For each item i : (a) Draw difficulty  i  N (   ,   1  ) (b) Draw discriminability  i  N (   ,   1  ) (c) Draw feasibility  i  U [0 , 1] 3. Draw subject i response on item j , r ij  p ij ( r ij |  j ,  i ,  i ) = p ij ( r ij = 1 |  j ) =  i 1 + e   i (  j   i ) . (1) For IRT -base,  i and  i are fixed to 1.0, while for IRT -disc, only  i is fixed. 3 Means   ,   ,   are drawn from N (0 , 10 6 ) and   ,   ,   from a (1 , 1) prior, as in Lalor et al. (2019) and recommended by Natesan et al. (2016). 4 3 In psychometrics, IRT -base is called a Rasch (Rasch, 1960) or 1 parameter logistic (1PL) model, IRT -disc is a 2PL model, and IRT -feas is a 4PL model with guessing set to zero. 4 We differ by allowing  < 0 to identify bad items. Because it is difficult to completely codify skill and difficulty into a single number, we can rewrite the exponent in Equation 1 as a sum over dimensions   i (  k  j,k   i,k ) , where each dimension captures the interaction between an item's difficulty and a subject's skill. For example, perhaps Burt could better exploit artifacts in one dimension (their skill for  j,k =5 is high but everywhere else is low) while Ken might not know much about a particular topic like potent potables (  j,k =2 is low but everywhere else is high). We call this model IRT -vec . 5 Multidimensional IRT models (Reck-ase, 2009) couldin addition to better modeling difficultyalso cluster items for interpretation; we briefly experiment with this (Appendix F), but leave more to future work (8). IRT 's fundamental assumption is that not all items and subjects are equal. This explains why leaderboards can fail while having normal looking accuracies. As a thought experiment, consider a dataset that is one third easy (  i  [0 , 1] ), one third medium difficulty (  i  [2 , 3] ), and one third hard (  i  [6 , 7] ). Suppose that Ken has skill  k = 4 while Burt has skill  b = 2 . A standard leaderboard would say that Ken has higher accuracy than Burt. But suppose there's a new subject that wants to challenge Ken; they are not going to reliably dethrone Ken until their skill  c is greater than six. This is a more mathematical formulation of the easy and hard dataset splits in question answering (Sugawara et al., 2018; Rondeau and Hazen, 2018; Sen and Saffari, 2020). In IRT -feas, this recapitulates the observation of Boyd-Graber and Brschinger (2020) that annotation error can hinder effective leaderboards. DAD helps systematize these observations and diagnose dataset issues. To estimate the latent parameters of our model, we use mean-field variational inference (Jordan et al., 1999). In variational inference, we propose a distribution over the latent variables, q  (  ) , that approximates the true but intractable posterior p (  ) . We then minimize the KL -divergence between these distributions, equivalent to maximizing the evidence lower-bound ( ELBO ) with respect to the variational parameters. In our case, q  (  ) is a mean-field distribution, which means it factorizes over each of the latent variables (the product is over the n  m subject-item pairs) q  (  ,  ,  ,  ,  ) = q (  ) q (  )  i,j q (  j ) q (  i ) q (  i ) Specifically, for our key latent variables z  {  ,  ,  } , the associated variational distributions are of the form q ( z ) = N ( u z , t  1 z ) . Recall that in the generative distribution, each latent z is drawn from a N (  z ,   1 z ) whose parameters are also latent variables; for these variables, we use the variational distributions q (  z ) = N ( u  z , t  1  z ) and q (  z ) = ( a  z , b  z ) . We optimize the ELBO with respect to the variational parameters  = { u z , t z , u  z , t  z , a  z , b  z ,  } for all z using ADAM (Kingma and Ba, 2015). With DAD 's leaderboard IRT model introduced, we next discuss how leaderboard subjects are statistically compared and alternative methodssuch as using IRT parametersto evaluate whether two models are truly different. Fundamentally, the objective of comparative evaluations like leaderboards is to decide whether model A is better than model B . A thread of NLP has rightfully advocated for adding rigor to these decisions using statistics (Traub, 1997, Classical Testing Theory) where the objective is to infer a true score T from the observed test score X = T + E given a measurement error E , uniform across subjects. However, in educational testinga field measuring skill and knowledge in humans IRT is a primary measurement instrument (Hamble-ton, 1991, p. 2). A major motivation for IRT is that subjects of different skill have different errors. IRT explicitly accounts for the bandwidth-fidelity dilemma (McBride, 1976): items can either accurately measure a narrow ability range (fidelity) or inaccurately measure large ability ranges (band-width). 6 This section and the next contrast methods for identifying the best model and advocate for IRT . Implicit in nearly all leaderboard evaluations is ranking models by a statistic such as the average accuracy. As we show in 4, nave rankings are noisier than IRT rankings. Leaderboards should: (1) reliably and efficiently rank better models ahead of worse models (Tague-Sutcliffe, 1992; Voorhees, 2003) and (2) guide inspection of items and subjects (5). The first ameliorates the unavoidable randomness of finite evaluations while the second enables error analysis (Wu et al., 2019) and model probing (Belinkov and Glass, 2019; Zhang et al., 2019). First we verify that IRT models accurately predict the responses of subjects (4.2). Next, a ranking stability analysis shows that IRT has modestly better reliability than classical rankings (4.2.3). Lastly, using IRT to actively sample items for annotation yields rankings with better correlation to complete test data (4.4). At first blush, the differences between IRT and logistic regression are minimal, but we include the comparison to address natural questions from the NLP community: (1) do the idiosyncrasies of the IRT formulation hurt accuracy? (2) should we add features to better understand phenomena in the questions? (3) why not use deep models? The next section argues that both IRT and logistic regression are accurate even without laboriously engineered task-specific features. Adding obvious features such as item words (e.g., questions) only minimally improves the accuracy. We explicitly omit less interpretable deep models since our goal is to make leaderboards more interpretable. Just as educational testing researchers validate IRT models by seeing if they predict subject responses correctly (American Educational Research Association, 2014), we validate how well DAD predicts whether SQ u AD models get questions right. We compare against a logistic regression linear model ( LM ) implemented with Vowpal Wabbit (Agarwal et al., 2014). Since integrating handcrafted features is easy, we incorporate features derived from subject ID s; item ID s; functions of the SQ u AD question, answer, and title; and IRT parameters (details in Appendix B). As in IRT , logistic regression predicts whether a subject correctly responds to an item. Later, we discuss ways to integrate more features into IRT (8). Experiments are on the SQ u AD 2.0 leaderboard. Development data are publicly available, and organizers provide test set responses. There are 161 development subjects, 115 test subjects, and 11 , 873 items ( 1 . 9 million total pairs). Experiments that do not need test responses use all development subjects; those that do use the smaller test subset. Following prior work (Wu et al., 2020), we evaluate IRT and linear models by holding out 10% of responses and computing classification metrics. 7 In SQ u AD , predicting whether a response is correct is an imbalanced classification problem (80.4% of responses in the development set are correct). Thus, we use ROC AUC , macro F1, and accuracy. IRT models that incorporate more priors into the generative story should be better, but are they? We compare four IRT models: IRT -base, IRT -disc, IRT feas, and IRT -vec (2). The more sophisticated models are better and all improve over the LM (Figure 3) and correlate well with each other (Ap-pendix C). To be clear, while higher accuracy than LM is good, our goal is to validate that IRT models are accurate; later, we inspect model errors and identify annotation errors (5). Integrating additional features into Bayesian models is not trivial, so we instead use the flexibility of linear models to identify useful features. Our leave-one-in ablation compares features (Figure 3): the top ablations both use IRT features, further validating IRT parameters. The subject and item identifier features are also strongly predictive, but item is the stronger of the two. Text-based features are weaker, but this suggests future work to better integrate them into IRT models (8). Leaderboards should produce reliable subject rankings: can DAD rank systems even with a tiny test set? Thus, we compare the correlation both of traditional average accuracy (3) and IRT rankings on the whole test set compared to the rankings of the same metric on a smaller test set. Our first experiment (4.3.1) examines the stability of existing items and subjects while the second (4.4) investigates stability of new evaluation data using sampling strategies. 7 Everywhere else in the paper, we train on all responses. Rankings should be reliable within the same dataset (e.g., on dev set) and generalize to similar datasets (e.g., with a test dataset). To test the first, we measure the ranking stability of mutually exclusive samples of the development data (Buckley and Voorhees, 2000). To test the second, we measure the correlation between development set sample rankings to test set rankings (Voorhees, 1998). Specifically, for a range of sample sizes 8 we (1) sample two partitions of the data, (2) compute the classical ranking 9 and the IRT ranking from a refit IRT -feas model, then (3) compute Kendall's correlation (Kendall, 1938) between the samples for each ranking (details in Appendix D). In both cases IRT rankings have higher correlation than classical rankings (Figure 4, left). Since the benefit is strongest at low sample sizes, IRT can improve the reliability of small-scale evaluations. The second experiment examines ranking generalization: IRT yields more reliable measures of subject skill, implying a greater consistency in subject rankings across evaluation settings. Figure 4 compares the development set sample rankings computed above to rankings obtained using subjects' test set responses (with the same IRT model). Across all sample sizes, subjects' IRT ability estimated on the development set correlates well test set ability. Crucially, this is better than the corresponding classical metrics like accuracy (Ap-pendix D quantifies the statistical significance of the difference), supporting our original motivation for using IRT . 10 8 The sample size must be less than half the size of the development data so that we can obtain two samples. 9 For SQ u AD , ordering by mean exact match score. 10 Since the maximum trial size was limited, we train one final model with the full data, see Table 3 in the Appendix D. 4.4 IRT Improves Cold Start Reliability IRT can also guide the construction of tests. Just as IRT practitioners prepare tests for humans, we too construct tests for machines. In educational testing, collecting responses from humans is expensive; likewise, although questions are cheap in search-based QA tasks (Nguyen et al., 2016; Kwiatkowski et al., 2019), annotating answers is expensive. Likewise, grading machine dialog responses is expensive and IRT helps (Sedoc and Ungar, 2020). To emulate this setting, we use computerized adaptive testing (Weiss and Kingsbury, 1984) to iteratively select SQ u AD items to annotate. As in human test preparation, we use existing annotations to infer item parameters and iteratively infer the ability of new subjects. This experiment splits m subjects into a training group (80%) and a testing group (20%). The training group represents subjects for which we have full item predictions and annotations; the testing group represents a new group of subjects that we need to rank. To efficiently rank, we should iteratively choose items to annotate that yield the most information about the ranking if all the data were annotated. This experiment compares how well several item selection strategies work. For each selection method, we (1) choose a sample size, (2), sample from the development set, (3) compute the ranking of subjects, and (4) compute Kendall's rank correlation (Figure 5). 11 Which item selection strategies should we compare? As a baseline, we use nave random sampling. Like prior work, we compare selecting items with the highest difficulty and the highest discriminability (Lalor et al., 2019) as well as the sum of the 11 We compute correlations with the complete development set on ten trials to build 95% confidence intervals. two. 12 We propose that items should be selected according to their Fisher information content (Weiss, 1982) I i (  j ) = ( p  ij ) 2 p ij (1  p ij ) =  2 i p ij (1  p ij ) (2) as derived by Lord et al. (1968, p. 70). Intuitively, if we do not yet know the true skill  j , we should pick items whose expected response we are most uncertain about. Our uncertainty (entropy) is maximized when the likelihood of a correct re-12 We train an IRT -disc model to simplify sampling (e.g., avoiding a tradeoff between feasibility and discriminability). sponse p ij is the same as the likelihood of an incorrect response 1  p ij , which corresponds to the maximal value of I i (  j ) ; it is also sensible this value increases as discriminability  i increases. To infer the maximally informative items, we estimate the ability  j of each subject using the currently selected items, use the ability to compute the information of each yet-to-be-annotated item for each subject, and then aggregate the informativeness Info ( i ) =  j I i (  j ) (3) by item i summed over subjects j . This approach is similar to uncertainty sampling and reduces to it for the IRT -base model (Lewis and Gale, 1994). We initially seed with the twenty-five most discriminative items (details in Appendix D). Like computerized adaptive testing (Moreno et al., 1984), Figure 5 shows that at lower sample sizes three of the IRT sampling methods are better than random samplingdifficulty does worse. The other IRT methods have comparable correlation. Thus, by using IRT , DAD can both improve rankings and guide annotation. DAD also helps qualitative analysis of items and subjects. First, IRT identifies overfitting and generalizes partitioning datasets by difficulty. Then we show thatlike in educational testing IRT identifies good and bad items. Several works curate easy and hard QA subsets based on how many models answer correctly (Ron-deau and Hazen, 2018) or heuristics (Sugawara et al., 2018). IRT can create similar subsets using IRT -feas, the best 1D model. Difficulty finds where subjects improve while discriminability and feasibility can surface items that may be invalid. For example, one low feasibility question (Figure 9) asks what are two examples of types of Turing machines? which has two problems: (1) the answer omits five types and (2) span-based evaluation precludes selecting non-contiguous types. After excluding items with negative discriminabilitythey are likely erroneous we sort items into bins. We break both difficulty and discriminability into four binstaking the 25 th , 50 th , and 75 th percentilescreating eight total bins. Then we select representative SQ u AD subjects with their exact match scores (Figure 6). Let's examine a feasible item with positive difficulty and discriminability like what reform was attempted following the Nice treaty? 13 In this case, the annotator's span is too longresulting in almost no correct answers and a low fuzzy match (token F1). In contrast, one highly discriminative question succeeds because there are multiple plausible guesses to who did the Normans team up with in Anatolia? 14 While both the Armenian state and Turkish forces are superficially plausible answers, only Turkish forces is correct; nonetheless, some models are fooled. Using IRT to guide subject analysis is helpful; next, we test how efficient it is in identifying annotation error. To test if IRT can identify annotation error, we inspect sixty SQ u AD development set items. We select ten items from each of these groups: the most negative discriminability, discriminability nearest to zero, the highest discriminability, the least difficult, most difficult, and IRT model errors. For each, we annotate whether the item was correct, was cor-rect yet flawed in some way, or simply wrong (Fig-ure 7). 15 Inter-annotator agreement between three authors on this three-way annotation with Krippen-dorff's  (Krippendorff, 2004; Artstein and Poesio, 2008) is 0 . 344 . Despite only modest agreement, just as in the development of education tests, negative discriminability is predictive of bad items. When discriminability is negative, then the probability of getting the answer right is higher when ability is lower , which is undesirable: Ken consistently loses to Burt on those items. This could identify bad items in evaluation sets for removal. DAD draws together two primary threads: we use IRT to understand datasets, which has been applied to other NLP tasks, and apply it to improving leaderboards. Finally, we explore how the insights of IRT can improve not just the analysis of test sets but to improve the construction of test sets. IRT in NLP IRT is gaining traction in machine learning research (Martnez-Plumed et al., 2016, 2019) where automated metrics can be misleading (Sedoc et al., 2019): machine translation (Hop-kins and May, 2013) and chatbot evaluation (Sedoc 15 Annotation guidelines provided in supplementary materials; Figure 7 uses the first set of annotations which were later augmented by two additional sets of annotations. and Ungar, 2020). Concurrent with our work, Vania et al. (2021) compare NLP test sets with IRT . Closest to our work in NLP is Otani et al. (2016), who rank machine translation subjects and compute correlations with gold scores. Similarly, Martnez-Plumed and Hernndez-Orallo (2020) use IRT on non-language AI video game benchmarks. Just as we use IRT to identify difficult or easy items, Lalor et al. (2016) create challenge sets for textual entailment. We test IRT as a way to guide annotation, but it can also train NLP models; for example, deep models learn easy examples faster (Lalor et al., 2018) and maintain test accuracy when training data are down-sampled (Lalor et al., 2019). Improving Leaderboards The rise NLP leaderboards has encouraged critical thought into improving them (Linzen, 2020), improving evaluation more broadly (Eger et al., 2020), and thoughtful consideration of their influence on the direction of research (Sculley et al., 2018; Dotan and Milli, 2020). DAD aims make leaderboard yardsticks (Hernandez-Orallo, 2020) more reliable, interpretable, and part of curating the benchmark itself. In line with our reliability goal, just as statistical tests should appear in publications (Dror et al., 2018; Dodge et al., 2019), they should be freebies for leaderboard participants (Ethayarajh and Juraf-sky, 2020). Alternatively, Hou et al. (2019) posit that leaderboards could be automatically extracted from publications. How to aggregate multi-task benchmarks (Wang et al., 2019b,a; Fisch et al., 2019) and multi-metric benchmarks (Ma et al., 2021) is an open question whichalthough we do not addressis one use for IRT . This work implicitly argues that leaderboards should be continually updated. As a (static) leaderboard ages, the task(s) overfit (Recht et al., 2019) whichalthough mitigable (Blum and Hardt, 2015; Anderson-Cook et al., 2019)is best solved by continually collecting new data (Kiela et al., 2021). Ideally, new data should challenge models through adversarial collection (Wallace et al., 2019b; Nie et al., 2020) and related methods (Gard-ner et al., 2020). However, if making an easy leaderboard more difficult is possible, the leaderboard has outlived its helpfulness and should be retired (Voorhees, 2000). Part of our work centers on alternate task efficacy rankings, but this navely assumes that task efficacy is the sole use case of leaderboards. Indeed, focusing solely these factors can mislead the public (Paullada et al., 2020) and may not reflect human language capabilities (Schlangen, 2020). Leaderboards are also well positioned to provide incentive structures for participants to prioritize fairness (Bender and Friedman, 2018) and efficiency (Strubell et al., 2019; Schwartz et al., 2020; Min et al., 2021) or incorporate testing of specific capabilities (Ribeiro et al., 2020; Dunietz et al., 2020). To enable these more nuanced analyses, leaderboards should accept runnable models rather than static predictions (Ma et al., 2021). Active Learning Beyond IRT , the analysis of training dynamics and active learning (Settles, 2009) is helpful for actively sampling specific items or identifying low-quality items (Brodley and Friedl, 1999). For example, Swayamdipta et al. (2020) and Pleiss et al. (2020) propose alternative training dynamics-based methods for identifying difficult items as well annotation errors. Even closer to goals, Rahman et al. (2020) use active learning to build a test collection. Explicitly measuring how effectively examples separate the best subject from the rest allows test set curators to fo-cus on the bubble (Boyd-Graber and Brschinger, 2020), prioritizing examples most likely to reveal interesting distinctions between submitted systems. Alternate Formulations IRT is an example of convergent evolution of models that predict subject action given an item. Ideal point models (Poole and Rosenthal, 2017) consider how a legislator (subject) will vote on a bill (item) and use a similar mathematical formulation. The venerable ELO model (Glickman and Jones, 1999) and modern extensions (Herbrich et al., 2007) predict whether a player (subject) will defeat an opponent (item) with, again, a similar mathematical model. Certain IRT models can also be formulated as nonlinear mixed models (Rijmen et al., 2003), where the item parameters are fixed effects and the latent subject parameters are random effects. This allows for comparisons between IRT models and other mixed effects models under a consistent framework. IRT base and IRT -disc can be formulated as nonlinear mixed models, and IRT -feas can be formulated as a discrete mixture model over items. As we discuss further in the next section, DAD 's application of IRT can further be improved by adopting interpretable extensions of these models. This paper advocates incorporating decades of research in crafting education tests to improve how we evaluate the capabilities of NLP models. We propose and validate an alternate IRT ranking method for leaderboard evaluations, show it can guide annotation, detect annotation error, and naturally partition evaluation data. Just as educators moved from classical testing to IRT , the NLP community should consider future evaluations with IRT . Although there is much to gain through IRT evaluation, there are limitations which make it hard to implement. First, it requires access to item-level responses for all examples for all subjects which are often only available to organizers. Second, Urbano (2016) notes that sampling mutually exclusive subsets has drawbackssamples are not entirely independent. We see a few directions for future work. First, this paper is intended to validate IRT and its usefulness as an active part of the leaderboard lifecycle; the natural next step is to implement it in a leaderboard. Second, our IRT models do not incorporate the item content (e.g., example text) to predict responses, but in principle could; Bayesian models with meta-data (Card et al., 2018) and ideal point models from political science (Poole and Rosenthal, 1985) that incorporate bills and speeches do exactly this (Ger-rish and Blei, 2011; Nguyen et al., 2015; Kraft et al., 2016). Analogously, IRT for leaderboards can and should also incorporate text from passages, questions, and answers to better model what makes questions difficult. Such a model can also predict which characteristics would create discriminating or difficult items. Lastly, multidimensional IRT models to evaluate multiple skills could aid multitask or multi-metric leaderboards like MRQA (Fisch et al., 2019) and Dynaboard (Ma et al., 2021). For their work on early iterations of leaderboard visualizations, we thank Jacob Bremerman and Wei Wei Chi. For insightful discussions and ideas we thank Shi Feng, Doug Oard, Joo Sedoc, Mike Wu, and Patrick Lewis. We thank Peter Rankel for recommendations on statistical testing methods. For discussion and feedback on visualizations, we thank Leo Zhicheng Liu, Calvin Bao, and classmates in UMD 's Fall 2020 Information Visualiza-tion course. For suggestions on topic modeling, we thank Philip Resnik and Maria Antoniak. For feedback on prior versions of this paper, we thank our anonymous ACL reviewers and members of the UMD CLIP lab. Boyd-Graber and Rodriguez's work at UMD were supported by NSF Grant IIS -1822494. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the sponsor. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
abstain abstain result method method result abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain other method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain method abstain method abstain abstain method method abstain abstain abstain method abstain method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain method abstain method method method method abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain objective other other method method other method method other other abstain other abstain method other other other method other other other other other other other other other other other other other abstain result objective abstain abstain abstain abstain abstain abstain method abstain abstain abstain other other other other other other other other other