Evaluation in NLP is usually done by comparing the scores of competing systems independently averaged over a common set of test instances. In this work, we question the use of averages for aggregating evaluation scores into a final number used to decide which system is best, since the average, as well as alternatives such as the median, ignores the pairing arising from the fact that systems are evaluated on the same test instances. We illustrate the importance of taking the instance-level pairing of evaluation scores into account and demonstrate, both theoretically and empirically, the advantages of aggregation methods based on pairwise comparisons, such as the BradleyTerry (BT) model, a mechanism based on the estimated probability that a given system scores better than another on the test set. By re-evaluating 296 real NLP evaluation setups across four tasks and 18 evaluation metrics, we show that the choice of aggregation mechanism matters and yields different conclusions as to which systems are state of the art in about 30% of the setups. To facilitate the adoption of pairwise evaluation, we release a practical tool for performing the full analysis of evaluation scores with the mean, median, BT, and two variants of BT (Elo and TrueSkill), alongside functionality for appropriate statistical testing. Research is driven by evaluation results, with attention and resources being focused on methods identified as state of the art (SotA). The proper design of evaluation methodology is thus crucial to ensure progress in the field. In NLP, evaluation usually consists in comparing the averaged scores of competing systems over a common set of test instances. Indeed, averaging scores independently for each system and declaring the one with the highest average to be best is particularly 1 2 3 4 5 Index of test instances 2.5 5.0 7.5 10.0 12.5 15.0 17.5 E v a l u a t i o n s c o r e s o f s y s t e m s ( h i g h e r i s b e tt e r ) System A Mean (same for all) System B Medians System C Figure 1: Motivating example (synthetic data). Here, we critically assess the specific choice of the average to aggregate evaluation scores. In particular, we emphasize that there is a natural in-stance-level pairing between the evaluation scores of systems, which aggregation mechanisms such as the mean or median fail to take into account: as they produce a score for each system independently, systems that have the same set of scores (but potentially in different order) cannot be distinguished. Consider the three systems A , B , and C compared on five test instances in Fig. 1. Despite a complex pairing structure, they all have the same mean score across test instances. Moreover, even though B is better than A on all test instances but one, the median of A is greater than the median of B . In this work, we discuss an alternative aggregation mechanism: the BradleyTerry (BT) model 2302 (Bradley and Terry, 1952). BT compares systems for each test instance and estimates the latent strength of systems based on how frequently one system scores higher than another. Such paired mechanisms have already been successfully used to aggregate human judgments (Novikova et al., 2018; Sedoc and Ungar, 2020); for example, WMT evaluation protocols regularly employ TrueSkill (Herbrich et al., 2007), a Bayesian variant of BT (Sakaguchi et al., 2014). Contributions. We contribute the first comprehensive analysis of the BT model (especially vis--vis mean and median) as an aggregation mechanism for comparing system scores in NLP. (i) We illustrate the importance of accounting for instance-level pairing and discuss the conditions under which the mean, median, and BT disagree about the ordering of systems. In Sec. 3, we draw parallels with the field of statistical testing, where paired statistical tests are recommended when comparing paired variables. Thus, we argue that paired aggregation mechanisms such as BT are more robust alternatives to the mean and median. We support this argument with simulations in Sec. 4. (ii) We show that the differences between mean, median, and BT matter in practice. By re-evalu-ating 296 real NLP evaluation setups across four tasks and 18 evaluation metrics, different aggregation mechanisms yield different conclusions as to which systems are SotA in about 30% of the setups (Sec. 5). These results hold when replacing BT by the Elo (Elo, 1978) and TrueSkill variants. (iii) We discuss further advantages and potential limitations of BT, alongside possible resolutions, in Sec. 7. (iv) We recommend replacing the mean by BT in future evaluations of NLP systems. To ease the adoption of more robust aggregation mechanisms, we release Pairformance , 1 a practical tool for performing full analyses of evaluation scores with mean, median, BT, and two variants of BT (Elo and TrueSkill). The tool reports paired evaluation results alongside appropriate statistical testing for all five aggregation mechanisms and various visualization functionalities to elucidate the pairing structure between system scores. Code and data for replicating our analyses and experiments is available online. 2 1 https://github.com/epfl-dlab/ pairformance 2 https://github.com/epfl-dlab/BT-eval 2 Aggregation of evaluation results In this section, we briefly present the three aggregation mechanisms we consider. 1. At least two systems, A and B , to compare, with latent strengths  A and  B that we aim to estimate. 2. A test set T = (cid:8) ( x l , y l ) : l = 1 ,..., n (cid:9) consisting of n test instances, where x l is the input and y l is the ground-truth target output. 3. An evaluation metric M for scoring system outputs based on target outputs y l , resulting in the sequence of evaluation scores MA = (cid:104) M ( A ( x l ) , y l ) : l = 1 ,..., n (cid:105) for system A . 4. An aggregation mechanism  that decides whether system A is better than B based on the evaluation scores of the two systems. We use  T , M ( A , B ) = ( MA , MB ) to denote the comparison mechanism between A and B on the test set T with evaluation metric M . Here,  outputs its guess about which system is the best (or declares the comparison inconclusive if the difference is not statistically significant). For simplicity, we drop the dependency on T and M in the notation, simply writing ( A , B ) . For example in text summarization, x l is a source document from the test set, y l its corresponding reference summary, and M might be ROUGE (Lin, 2004). The decision mechanism  usually compares the individual systems' mean evaluation scores, where the system with the highest mean score (here mean ROUGE score) is declared better. Consistent evaluation result. We say that the outcome of such an evaluation is consistent if it recovers the ordering of systems implied by the inherent strengths of systems: ( A , B ) = A   A >  B . Probabilistic model. As commonly done in the literature on statistical testing, we view the evaluation scores of a system A as n indexed random variables: X ( l ) A , l = 1 ,..., n , where n is the size of the test set. Note that this sequence of random variables is not necessarily i.i.d. Furthermore, even though systems A and B are independent, their evaluation scores are not, since there is an instance-level pairing. Intuitively, knowing the score of A on an instance ( x l , y l ) can provide information about the expected 2303 performance of B . We now introduce three aggregation mechanisms  . We investigate their properties in subsequent sections. Mean. This is the current standard: the system with the highest average score is declared the strongest. We denote this aggregation mechanism as MEAN . The average score of system A is computed as EA = 1 n n (cid:80) l = 1 X ( l ) A . Median. The median is an interesting alternative to the mean because it is robust to outliers. Here, the system with the highest median score is declared to be the strongest. The median score MA of a system A is the central value in the sorted list of evaluation scores of A . We denote this aggregation mechanism as MEDIAN . Bradley-Terry. The third option examined here is the BradleyTerry (BT) model (Bradley and Terry, 1952). While MEAN and MEDIAN compute scores for systems A and B independently, BT is a function of the joint random variable (cid:16) X ( l ) A , X ( l ) B (cid:17) . BT estimates the relative strengths   A and   B of the two systems A and B , by comparing the evaluation scores for each test instance: P ( A > B ) =   A   A +   B . (1) Intuitively, P ( A > B ) is the probability that, for any given test instance, A scores higher than B . The BT model chooses   A and   B in order to best explain the observations. The system with the highest   is declared strongest. When considering only two systems, the latent strength   A is the number of instances for which A scores better than B (and similarly for   B ). When the number of systems is greater than two, BT solves an iterative optimization algorithm that is guaranteed to converge to a unique solution (Bradley and Terry, 1952). We give details about BT and its computation in the general case in Appendix E. We denote as BT the decision mechanism based on the BT model. While it is much less common than MEAN and MEDIAN , we will see below that BT satisfies interesting properties making it a more robust alternative. Since the roles played by A and B are symmetrical, we now assume without loss of generality that system A is better, i.e.,  A >  B . Proposition 1. If  A >  B then  MEAN consistent  EA  EB > 0 ,  MEDIAN consistent  MA  MB > 0 ,  BT consistent  MA  B > 0 , where ES and MS are the mean and median of the evaluation scores of system S, and MA  B is the median of the differences between the evaluation scores of A and B. Note that ES , MS , and MA  B are all random variables. The proof is given in Appendix B. Note that, whereas the expectation is linear ( EA  EB = EA  B ), the median is not (in general, MA  MB (cid:54) = MA  B ). Robustness to ouliers. The mean is not robust to outliers: EA  B can be swayed above or below the threshold of 0 by a small number of test instances for which the difference between system scores is large. On the contrary, the median is a robust statistic that cannot be easily influenced by outliers. Similarly, BT is robust to outliers because its decision is based on the median of differences MA  B . Importance of pairing. The critical difference between BT , MEAN , and MEDIAN , is that only BT preserves the pairing information. Both MEAN and MEDIAN compute a statistic from the (unordered) set of scores X ( l ) A and X ( l ) B independently and then compare the aggregate statistics, losing the pairing structure. If the pairing actually does not matter, any permutation of the indices of system scores leaves the distribution of paired evaluation scores unchanged. This happens, for example, when both X ( l ) A and X ( l ) B are i.i.d. 3 However, in the general case, the pairing matters. One particular example is when there exist different types of test instances and systems behave differently for different types, e.g., when there are easy instances on which all systems have higher scores. For example, consider the three systems and their evaluation scores on five test instances in Fig. 1. System A is worse than C on all instances but one, so C > A according to BT , yet the median of A is greater than the median of C (10 vs. 7). At the same time, B outperforms C on all instances 3 More generally, when the two sequences of random variables are exchangeable. but one, so B > C according to BT . For MEDIAN and MEAN , which ignore the pairing, A and B are completely equivalent, even though there is a clear difference regarding which system is more likely to be the best. This difference is revealed in the pairing structure. In general, any mechanism ignoring the pairing cannot capture the difference between A and B . Choosing an aggregation mechanism. In Prop. 1, we stated the conditions for each mechanism to be consistent . Choosing an aggregation mechanism for a specific evaluation setup boils down to deciding what condition is more likely to hold in the setup. Note that none of the conditions implies any other condition in Prop. 1. When comparing BT against MEAN (or MEDIAN ), there are three possible scenarios: (i) BT agrees with MEAN (or MEDIAN ), (ii) BT is consistent but MEAN (or MEDIAN ) is not, and (iii) MEAN (or MEDIAN ) is consistent but BT is not. In case (ii), for most instances, the better system has a higher score than the worse system, but MEAN (or MEDIAN ) fails. For example, MEAN may be swayed by outliers, and MEDIAN may be swayed by jumps in score lists as in the example above. In case (iii), for most instances, the better system has a lower score than the worse system, yet particular variations in the marginals make the MEAN or MEDIAN get the ordering correct. This is a very peculiar scenario: for MEAN , it implies that on the few instances on which the better system did better, the difference between evaluation scores was large enough to lift the mean of the better system above the other. We argue that if one really believes that the evaluation setup is likely to be in case (iii), then one does not trust the evaluation setup in the first place. It corresponds to assuming that the observed scores are inconsistent for the majority of test instances. If this is the case, one should rather improve the evaluation setup (e.g., metric, test set) in order to be more representative of the phenomena that one desires to capture. Overall, the condition making BT consistent appears to be the most natural one. Trusting MEAN or MEDIAN more than BT implies holding an unintuitive belief about the evaluation setup, namely that the better system does worse than the worse system on a majority of test instances. From another perspective, the random variables EA  EB ( MEAN ) and MA  MB ( MEDIAN ) are less likely to be (correctly) greater than zero in the presence of (i) complex pairing structures or (ii) outliers. The variable MA  B ( BT ), on the contrary, is not affected by complex pairings or outliers. Fig. 2 summarizes the problem of ignoring the pairing and offers a graphical criterion to understand the decisions made by MEAN , MEDIAN , and BT . In each plot, the densities are estimated by placing test instances at coordinates given by the evaluation scores of the two systems. The evaluation scores of A (green) are on the x -axis, and the evaluation scores of B (blue) on the y -axis. We also plot the marginal distributions of evaluation scores, from which we can read off means and medians. When the mean of X ( l ) B is greater than that of X ( l ) A , the two extended lines representing the means meet in the upper triangle (above the line XA = XB ), and analogously for the median. But mean and median being only functions of the marginals, they completely ignore the pairing. Fig. 2 illustrates this by depicting three completely different pairing structures where the marginals (and thus the means and medians) of A and B remain unchanged. (In Appendix A.1, we explain how to generate infinitely many such examples.) On the contrary, BT , being a property of the pairing (the 2D density), predicts that B is better than A when there is more mass in the upper triangle, i.e., more instances for which B scores higher than A . In the middle figure, the pairing indicates that A is better than B , in disagreement with the decisions of MEAN and MEDIAN . The above discussion about the differences between MEAN , MEDIAN , and BT has interesting parallels with statistical testing. When comparing the means of two systems over the same test set, the recommended statistical test is the paired t -test (Fisher, 1935). When comparing medians instead of means, the appropriate test is the sign test, which measures whether the median of the difference is significantly differerent from zero. Interestingly, the statistic of the sign test is precisely the one in the condition for BT to be consistent (see Prop. 1). Wilcoxon's signed-rank test (Wilcoxon, 1945) is often used as an alternative to the sign test because it has more statistical power (at the cost of making more assumptions). However, 2305 10 20 30 40 50 60 Scores of A 10 20 30 40 50 60 S c o r e s o f B Median Mean XA = XB 10 20 30 40 50 60 Scores of A 10 20 30 40 50 60 S c o r e s o f B Median Mean XA = XB 10 20 30 40 50 60 Scores of A 10 20 30 40 50 60 S c o r e s o f B Median Mean XA = XB Figure 2: These 2D plots represent the distribution of test instances with coordinates given by the scores of the two systems being compared, i.e., the x -axis is the score X ( l ) A of system A on some test instance ( x l , y l ) , and the y -axis is the score X ( l ) B of system B on the same instance. Divine et al. (2018) showed that Wilcoxon's signed-rank test does not always properly account for the pairing of data, unlike the sign test. When performing statistical testing, it seems obvious that we should use the paired version of tests when the data is naturally paired (Rankel et al., 2011). Even works discussing statistical testing in NLP recommend Wilcoxon's signed-rank test (Gra-ham, 2015; Owczarzak et al., 2012; Dror et al., 2018). Yet, to obtain aggregated scores for systems, the community still mostly uses aggregation mechanisms that ignore the pairing, such as MEAN . MEDIAN is the outlier-resistant version of MEAN , and BT is the paired variant of MEDIAN . Whenever one recommends a paired test of medians, such as the sign test or Wilcoxon's signed-rank test, to obtain p -values, one should use BT to compare system scores. Next, we perform simulations to extend the analysis of the previous section to (i) N > 2 systems, (ii) finitely many test samples, (iii) a practical implementation of BT (for N > 2 systems, BT is an iterative optimization algorithm, as discussed in Appendix E). We synthesize evaluation scores with various properties starting with systems of predefined implicit strengths  i . To create situations where the pairing of evaluation scores matters, we introduce multiple test instance types. For each type, systems perform differently but still have the same relative strength ( P ( A > B ) ), differing only by an added offset. For example, the evaluation scores obtained by A and B could be sampled from N (  A , ) and N (  B , ) for one test instance type, and by N (  A + (cid:15), ) and N (  B + (cid:15), ) for another type, with (cid:15) being the offset. We sample evaluation setups by varying the following properties: the number of systems, the number of test instances, the percentage of outliers, the numbers of test instance types, and the level of noise. This results in 2,880 simulated evaluation setups. In Appendix A.2, we give the detailed algorithm and parameters used to generate the data. In Fig. 3, we report Kendall's  between the latent scores  i and the aggregated scores estimated by MEAN , MEDIAN , and BT . When the evaluation setup does not present any difficulty (Fig. 3(a, b)), all aggregation mechanisms work equally well (within each other's 95% error bounds), improving with more samples (Fig. 3(b)) and deteriorating with more systems (Fig. 3(a)). Unsurprisingly, MEAN fails in the presence of outliers, whereas MEDIAN and BT are unaffected (Fig. 3(c, e, f)). When several types of test instances are considered, MEDIAN begins to fail (Fig. 3(d)), which is made worse when outliers are also present (Fig. 3(f)). Overall, BT is more robust and does not fail when the pairing matters Fig. 3(g, h). In this section, we perform large-scale experiments using real evaluation scores from four NLG tasks. For summarization, we use the TAC-08, TAC-09, TAC-11 and CNN/DM (Hermann et al., 2015) datasets. For machine translation, we use the shared tasks of WMT-17 (Bojar et al., 2017), WMT-18 (Ma et al., 2018), and WMT-19 (Ma et al., 2019). For image captioning, we use the MSCOCO (Lin et al., 2014) dataset, and for dialogue, we use the PersonaChat and TopicalChat (Mehri and Eskenazi, 2020) datasets. The evaluation scores are obtained with a total of 18 different evaluation metrics: BLEU-[1,2,3,4] (Papineni et al., 2002), ROUGE-[1,2,L] (Lin, 2004), ROUGE-WE-[1,2] (Ng and Abrecht, 2015), JS-[1,2] (Lin et al., 2006), S3-[pyr, resp] (Peyrard et al., 2017), CIDEr (Vedan-tam et al., 2015), Chrfpp (Popovic, 2017), METEOR (Lavie and Agarwal, 2007), MoverScore (Zhao et al., 2019), and BERTScore (Zhang et al., 2020). Some metrics are only available for some task; e.g., CIDEr, METEOR are only available for the image captioning task. We provide details about datasets, metrics, and their statistics in Appendix A.3. Overall, across datasets and metrics we have 296 evaluation setups, 73,471 pairs of systems, and 91,197 test instances. We also experiment with sub-sampling different sizes of test sets (see Appendix A.3) to simulate varying train/dev/test splits or cross-validation. In Table 1, we report the disagreement between aggregation mechanisms over all the data with three measures: the percentage of pairs ranked in a different order (rescaled version of Kendall's  ), the percentage of setups where the state-of-the-art (SotA) systems are different, and the percentage of setups where the top 3 systems are different (com-pared as sets). A significant fraction of pairs of systems (about 10%) are ranked differently by different mechanisms. More importantly, top systems are often different (in about 40% of setups for top 1 and 50% for top 3). We can conclude that the choice of aggregation mechanism has a real impact on evaluation outcome. The observed disagreement between the three aggregation metrics implies that we are not in the case depicted by Fig. 3(a) and Fig. 3(b), i.e., the pairing matters and there are outliers in real data. In the next paragraphs, we break down the disagreement per evaluation metric, task, and test set size. Detailed results are provided in Appendix C. Which metrics are impacted most? We report in Fig. 4(a) the percentage of disagreement between aggregation mechanisms per metric averaged over datasets, when subsampling test sets of different sizes uniformly (see Appendix A.3 for details). While most metrics are available for all four tasks, METEOR and CIDEr are only available for the captioning task. Therefore, the observed disagreements for these metrics may be a feature of the task instead of the metrics. Interestingly, recent metrics 2307 Disagree (cid:54) = SotA (cid:54) = Top-3 MEAN vs. MEDIAN 4% 18% 30% MEAN vs. BT 9% 40% 49% MEDIAN vs. BT 9% 41% 55% Table 1: Disagreement between aggregation mechanisms. such as BERTScore and MOVERScore seem less affected. On the other hand, BLEU variants are the most impacted, particularly when comparing MEAN or MEDIAN against BT . The disagreement between MEAN and MEDIAN is stable across metrics. In general, MEAN and MEDIAN are more in agreement with one another than they are with BT , which indicates that pairing issues have a stronger effect than outliers. Which tasks are impacted most? Fig. 4(b) summarizes an analysis as above, but across tasks instead of metrics. Again, to control for the fact that some tasks may have larger datasets, we subsample uniformly from various test set sizes. The results are averaged over evaluation metrics. Machine translation and summarization suffer the least while dialogue and image captioning display larger disagreement between aggregation mechanisms. This suggests important future research directions to improve the evaluation setups in these tasks. Importance of dataset size. In Fig. 4(c), we report disagreement across test set sizes, while averaging over datasets and evaluation metrics. It is reassuring to observe that with larger test sets, the different mechanisms tend to agree more, such that it matters less which one is actually chosen. However, for MEAN vs. BT and MEDIAN vs. BT , the disagreement does not continue to decrease below 10% with more test instances. For MEAN and BT the disagreement is lower but exhibits the same behavior, never falling below a certain threshold. Different perspectives on uncertainty. In standard evaluation setups, not only system scores are reported but also whether the differences are statistically significant (Dror et al., 2018). Therefore, we ask how often differences that are statistically significant for one test are also statistically significant for another. The details of this experiments are presented in Appendix D and show, perhaps unsurprisingly, different behavior for different tests. In particular, the paired t -test is the one that most often finds differences to be significant (for 41% of pairs); Mood's test, an unpaired test to compare medians, finds significance for only 21% of pairs; and the sign test and Wilcoxon's sign-rank test (re-lated to BT ) are in between (for 35% and 40% of the pairs, respectively). Sources of disagreement. Based on the analysis of Sec. 3, we know that the difference between MEAN and MEDIAN is due to the presence of statistical outliers, while the difference between MEDIAN and BT is due to the presence of different test instance types (Fig. 3). With real NLP datasets, in Fig. 4, we observe some discrepancy between MEAN and MEDIAN , indicating the presence of outliers. There is even more disagreement between MEDIAN and BT , indicating the presence of different types of test instances, as illustrated in Fig. 3. 6 Related work Several studies have made a critical assessment of the standard evaluation methodologies. For example, Freitag et al. (2020) demonstrate the advantages of carefully choosing which references to use for NLG evaluation. Mathur et al. (2020) show that outliers matter in practice. Recently, Graham et al. (2020) draws attention on test set size. Several works have emphasized the importance of careful statistical testing (Rankel et al., 2011; Owczarzak et al., 2012; Graham, 2015; Dror et al., 2018). They recommend paired statistical tests. Finally, Novikova et al. (2018) report that rela-tive rankings yield more discriminative results than absolute assessments, which further motivates aggregation mechanisms like BT . Aggregations. Pairwise comparison mechanisms date back to Thurstone (1927). Subsequently, the Bradley-Terry (BT) model has become a standard pairwise comparison model (Bradley and Terry, 1952). In NLP, BT-inspired mechanisms have sometimes been used to aggregate human assessments. For instance, Deriu et al. (2020) ranked chatbots regarding their ability to mimic conversational behavior of humans. Item response theory (IRT) has a similar formulation as BT, but also estimates the difficulty of each test instances using a latent-variable Bayesian model (Dras, 2015). IRT has been applied to perform dataset filtering (Lalor et al., 2016, 2019), evaluate chatbots from human assessments (Sedoc and Ungar, 2020), and aggregate human assessments in machine translation (Dras, 2015). Elo (Elo, 1978) and TrueSkill (Herbrich et al., 2007) are famous extensions of the BT model commonly used to rate players in the context of gaming or sports events. Elo views player strengths as normally distributed random variables. TrueSkill is a Bayesian variant of Elo. Since 2015, the Workshop on Machine Translation (WMT) has been using TrueSkill to rank models based on human assessments following the methodology of Sakaguchi et al. (2014). We provide a detailed presentation and comparison of BT, Elo, and TrueSkill in Appendix G, and make both Elo and TrueSkill available as alternatives to BT in the released tool. The arguments in favor of BT made in this work transfer to its variants, including IRT, Elo, and TrueSkill, and the conclusions drawn from the experiments of Sec. 5 still hold when replacing BT by Elo or TrueSkill (Appendix G). Our work extends previous works that has considered BT variants by analyzing the potential causes for disagreement with MEAN and MEDIAN and by measuring the disagreement in real NLP evaluation setups. We briefly discuss some possible questions raised by the use of BT -like metrics, with more details provided in Appendix E, F, G, and H. Extension to other evaluation setups. The experiments of Sec. 5 focus on reference-based NLG evaluation metrics. However, the arguments laid out throughout the paper apply beyond this setup. Any comparison of systems based on score aggregation is susceptible to suffer from outliers and complex pairing structures (e.g., Fig. 2). Future work should replicate our experimental setup for reference-free NLG (Zhao et al., 2020), classifica-tion, or regression tasks. Type imbalance. Imagine a test set with a majority of easy instances and few hard ones. A system A could perform slightly worse than B on easy instances but much better on hard ones and will be declared worse by BT . If one views this decision as problematic then one should probably acknowledge that the test set is not representative of what should be measured. If hard instances matter more there should be a majority of them in the test set. Hoping that MEAN will be swayed to output the intuitive ordering of systems from a minority of test instances is a peculiar expectation to have about the evaluation setup. To diagnose such pathological cases, our tool, Pairformance, offers the possibility to view pairwise plots (as in Fig. 2) and histograms 2309 of score differences. More generally, better aggregation mechanisms such as BT do not solve all potential problems of evaluation methodologies. Other aspects (such as choosing evaluation metrics or meaningful, representative, and large test sets) are all independent of the choice of aggregation mechanism, but also critical to the quality of the evaluation. Transitivity. BT is not computed independently for each system, and it can happen that adding or removing a baseline impacts the scores of other systems. We explain this phenomenon in Appendix F and show that it is rarely a problem in real data. More generally, we discuss the connection with Arrow's impossibility theorem in the context of the aggregation of social preferences (Arrow, 1950). The Pairformance tool gets around this difficulty by offering the possibility of analyzing each pair of systems independently. Relaxing assumptions. BT assumes that the relative strengths of systems remain constant across test instances. This might not always be true, especially when some systems are crafted for some specific kind of instances but perform badly on others. In such cases, BT still produces meaningful and easily interpretable results but fails to capture the latent structure of system strengths. Several refinements of BT are possible; e.g., item response theory extends BT by modeling instance difficulty, and Elo and TrueSkill allow system strengths to be stochastic and vary across instances. These refinements come at the cost of introducing new parameters, and it remains unclear how to choose these parameters in practice. Future work should investigate systematic ways to choose these parameters. Tool description. We release Pairformance, a tool for performing full diagnostic analyses based on an evaluation dataframe made of the evaluation scores of systems and baselines. It can perform the analysis based on MEAN , MEDIAN , BT , Elo, and TrueSkill. For each aggregation technique, it outputs a full pairwise analysis of all pairs of systems. For MEAN and MEDIAN it compares score differences for pairs of systems. For BT , Elo, and TrueSkill, it estimates the probability that one system is better than another. All analysis is accompanied by appropritate statistical testing. See Fig. 5 for an example based on the BT mechanism. Furthermore, the tool can plot the histogram of paired differences X ( l ) A  X ( l ) B , allowing for the direct iden-j hu pb m t . tification of pathological patterns such as those discussed above. We performed a critical assessment of the standard NLP evaluation methodology based on averaged scores, which ignores the natural instance-level pairing of evaluation scores when comparing systems. We showed the importance of the pairing and demonstrated the advantages of paired mechanisms such as BradleyTerry ( BT ) over more standard aggregation schemes such as the mean or median. The choice of aggregation mechanism matters in real evaluation setups, and we therefore recommend BT as a robust aggregation mechanism. To facilitate adoption, we release Pairformance, a new tool to perform full analyses of system scores using BT and two of its variants, Elo and TrueSkill. We thank the anonymous reviewers for their insightful comments and suggestions, which greatly improved the final version of the paper. With support from Swiss National Science Foundation (grant 200021_185043), European Union (TAILOR, grant 952215), and gifts from Google, Facebook, Microsoft.
abstain method objective result method abstain abstain abstain other abstain objective abstain abstain abstain abstain abstain abstain objective abstain method abstain abstain result abstain abstain abstain method method abstain abstain other abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain other abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method method other abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method objective method objective other other