Ordinal Classification (OC) is an important classification task where the classes are ordinal. For example, an OC task for sentiment analysis could have the following classes: highly positive, positive, neutral, negative, highly negative . Clearly, evaluation measures for an OC task should penalise misclassifica-tions by considering the ordinal nature of the classes (e.g., highly positive misclassified as positive vs. misclassifed as highly negative ). Ordinal Quantification (OQ) is a related task where the gold data is a distribution over ordinal classes, and the system is required to estimate this distribution. Evaluation measures for an OQ task should also take the ordinal nature of the classes into account. However, for both OC and OQ, there are only a small number of known evaluation measures that meet this basic requirement. In the present study, we utilise data from the SemEval and NTCIR communities to clarify the properties of nine evaluation measures in the context of OC tasks, and six measures in the context of OQ tasks. In NLP and many other experiment-oriented research disciplines, researchers rely heavily on evaluation measures. Whenever we observe an improvement in the score of our favourite measure, we either assume or hope that this implies that we have managed to moved our system a little towards what we ultimately want to achieve. Hence it is of utmost importance to examine whether evaluation measures are measuring what we want to measure, and to understand their properties. This paper concerns evaluation measures for Ordinal Classification (OC) and Ordinal Quantification (OQ) tasks. In an OC task, the classes are ordinal, not nominal. For example, Task 4 (Senti-ment Analysis in Twitter) Subtask C in SemEval-2016/2017 is defined as: given a set of tweets about a particular topic, estimate the sentiment conveyed by each tweet towards the topic on a five-point scale ( highly negative, negative, neutral, positive, highly positive ) (Nakov et al., 2016; Rosenthal et al., 2017). On the other hand, an OQ task involves a gold distribution of labels over ordinal classes and the system's estimated distribution. For example, Task 4 Subtask E of the SemEval-2016/2017 workshops is defined as: given a set of tweets about a particular topic, estimate the distribution of the tweets across the five ordinal classes already mentioned above (Nakov et al., 2016; Rosenthal et al., 2017). The Dialogue Breakdown Detection Challenge (Higashinaka et al., 2017) and the Dialogue Quality subtasks of the NTCIR-14 Short Text Conversation (Zeng et al., 2019) and the NTCIR-15 Dialogue Evaluation (Zeng et al., 2020) tasks are also OQ tasks. 1 Clearly, evaluation measures for OC and OQ tasks should take the ordinal nature of the classes into account. For example, in OC, when a highly positive item is misclassified as highly negative , that should be penalised more heavily than when it is misclassified as positive . Surprisingly, however, there are only a small number of known evaluation measures that meet this requirement. In the present study, we use data from the SemEval and NTCIR communities to clarify the properties of nine evaluation measures in the context of OC tasks, and six measures in the context of OQ tasks. Some of these measures satisfy the aforementioned basic requirement for ordinal classes; others do not. 1 In terms of data structure, we observe that the relationship between OC and OQ are similar to that between paired data and two-sample data in statistical significance testing. In OC, we examine which item is classified by the system into which class, and build a confusion matrix of gold and estimated classes. In contrast, in OQ, we compare the system's distribution of items with the gold distribution, but we do not concern ourselves with which item in one distribution corresponds to which item in the other. Section 2 discusses prior art. Section 3 provides formal definitions of the measures we examine, as this is of utmost importance for reproducibility. Section 4 describes the data we use to evaluate the measures. Sections 5 and 6 report on the results on the OC and OQ measures, respectively. Finally, Section 7 concludes this paper. As we have mentioned in Section 1, Task 4 Subtask C of the SemEval-2016/2017 workshops is an OC task with five ordinal classes (Nakov et al., 2016; Rosenthal et al., 2017). While SemEval also features other OC tasks with fewer classes (e.g., Task 4 Subtask A from the same years, with three classes), we use the Subtask C data as having more classes should enable us to see more clearly the difference between measures that consider ordinal classes and those that do not. 2 Note that if there are only two classes, OC is reduced to nominal classification. Subtask C used two evaluation measures that consider the ordinal nature of the classes: macroaveraged Mean Absolute Error (MAEM ) and the standard Mean Absolute Error (MAE  ) (Baccianella et al., 2009). At ACL 2020, Amigo et al. (2020) proposed a measure specifically designed for OC, called Closeness Evaluation Measure (CEMORD ), and discussed its axiomatic properties. Their meta-evaluation experiments primarily focused on comparing it with other measures in terms of how each measure agrees simultaneously with all of preselected gold measures. However, while their results showed that CEMORD is similar to all of these gold measures, the outcome may differ if we choose a different set of gold measures. Indeed, in the context of evaluating information retrieval evaluation measures, Sakai and Zeng (2019) demonstrated that a similar meta-evaluation approach called unanimity (Amigo et al., 2018) depends heavily on the choice of gold measures. Moreover, while Amigo et al. (2020) reported that CEMORD also performs well in terms of consistency of system rankings across different data (which they refer to as robustness), experimental details were not provided in their paper. Hence, to complement their work, the present study conducts extensive and re-2 SemEval-2018 Task 1 (Affect in Tweets) featured an OC task with four classes (Mohammad et al., 2018). However, the run submission files of this task are not publicly available. producible experiments for OC measures. Our OC meta-evaluation experiments cover nine measures, including MAEM , MAE  , and CEMORD . As we have mentioned in Section 1, Task 4 Subtask E of the SemEval-2016/2017 workshops is an OQ task with five ordinal classes (Nakov et al., 2016; Rosenthal et al., 2017). 3 Subtask E used Earth Mover's Distance (EMD), remarking that this is currently the only known measure for ordinal quantification (Nakov et al., 2016; Rosenthal et al., 2017). Subsequently, however, Sakai (2018a) proposed a new suite of OQ measures based on Order-aware Divergence (OD), 4 and compared them with Normalised Match Distance (NMD), a normalised version of EMD. Sakai utilised data from the Third Dialogue Breakdown Detection Challenge (DBDC3) (Higashinaka et al., 2017), which features three ordinal classes, and showed that his Root Symmetric Normalised OD (RSNOD) measure behaves similarly to NMD. However, his experiments relied on the run submission files from his own team, as he did not have access to the entire set of DBDC3 submission files. On the other hand, the organisers of DBDC3 (Tsunomori et al., 2020) compared RSNOD, NMD, and the official measures of DBDC (namely, Mean Squared Error and Jensen-Shannon Divergence, which ignore the ordinal nature of the classes) using all the run submission files from DBDC3. They reported that RSNOD was the overall winner in terms of system ranking consistency and discriminative power , i.e., the ability of a measure to obtain many statistical significant differences (Sakai, 2006, 2007, 2014). In addition to the aforementioned two Subtask E data sets from SemEval, the present study utilises three data sets from the Dialogue Quality (DQ) Subtasks of the recent NTCIR-15 Dialogue Evaluation (DialEval-1) Task (Zeng et al., 2020). Each DQ subtask is defined as: given a helpdesk-customer dialogue, estimate the probability distribution over the five-point Likert-scale Dialogue Quality ratings (See Section 4). Our OQ meta-evaluation experiments cover six measures, including NMD and RSNOD. 3 The Valence Ordinal Classification subtask of SemEval-2018 Task 1 (Affect in Tweets) is also an OQ task, with seven classes (Mohammad et al., 2018). However, the submission files of this task are not publicly available. 4 See also Sakai (2017) for an earlier discussion on OD. In the OC tasks of SemEval-2016/2017, a set of topics was given to the participating systems, where each topic is associated with N tweets. ( N varies across topics.) Given a set C of ordinal classes represented by consecutive integers, each OC system yields a | C |  | C | confusion matrix for each topic. From this, we can calculate evaluation measures described below. Finally, the systems are evaluated in terms of mean scores over the topic set. Let c ij denote the number of items (e.g., tweets) whose true class is j , classified by the system into i ( i, j  C ) so that N = (cid:80) j (cid:80) i c ij . Let c  j = (cid:80) i c ij , c i  = (cid:80) j c ij , and C + = { j  C | c  j > 0 } . That is, C + is the set of gold classes that are not empty . We compute MAE's as follows. Unlike the original formulation of MAEM by Bac-cianella et al. (2009), ours explicitly handles cases where there are empty gold classes (i.e., j s.t. c  j = 0 ). Empty gold classes actually do exist in the SemEval data used in our experiments. It is clear from the weights used above ( | i  j | ) that MAEs assume equidistance , although this is not guaranteed for ordinal classes. Hence Amigo et al. (2020) propose the following alternative: CEMORD = (cid:80) j  C (cid:80) i  C prox ij c ij (cid:80) j  C prox jj c  j , (3) where prox ij =  log 2 (max { 0 . 5 , K ij } /N ) , and K ij = (cid:40) c  i / 2 + (cid:80) jl = i +1 c  l ( i  j ) c  i / 2 + (cid:80) i  1 l = j c  l ( i > j ) . (4) Our formulation of prox ij with a max operator ensures that it is a finite value even if K ij = 0 . We also consider Weighted  (Cohen, 1968). We first compute the expected agreements when the system and gold labels are independent: e ij = c i  c  j /N . Weighted  is then defined as:  = 1  (cid:80) j  C (cid:80) i  C w ij c ij (cid:80) j  C (cid:80) i  C w ij e ij , (5) where w ij is a predefined weight for penalising misclassification. In the present study, we follow the approach of MAEs (Eqs. 1-2) and consider Linear Weighted  : w ij = | i  j | . However, it should be noted here that  is not useful if the OC task involves baseline systems such as the ones included in the aforementioned SemEval tasks: that is, a system that always returns Class 1, a system that always returns Class 2, and so on. It is easy to mathematically prove that  returns a zero for all topics for all such baseline systems. We also consider applying Krippendorff's  (Krippendorff, 2018) to OC tasks. The  is a measure of data label reliability, and can handle any types of classes by plugging in an appropriate distance function. Instead of the | C |  | C | confusion matrix, the  requires a | C | N class-by-item matrix that contains label counts n i ( u ) , which represents the number of labels which say that item u belongs to Class i . For an OC task, n i ( u ) = 2 if both the gold and system labels for u is i ; n i ( u ) = 1 if either the gold or system label (but not both) for u is i ; n i ( u ) = 0 if neither label says u belongs to i . Thus, this matrix ignores which labels are from the gold data and which are from the system. For comparing two complete sets of labels (one from the gold data and the other from the sys-tem), the definition of Krippendorff's  is relatively simple. Let n i = (cid:80) u n i ( u ) ; this is the total number of labels that Class i received from the two sets of labels. The observed coincidence for Classes i and j ( i, j  C, i (cid:54) = j ) is given by O ij = (cid:80) u n i ( u ) n j ( u ) , while the expected coincidence is given by E ij = n i n j / (2 N  1) . The  is defined as:  = 1  (cid:80) i (cid:80) j>i O ij  2 ij (cid:80) i (cid:80) j>i E ij  2 ij , (6) where, for ordinal data,  2 ij = ( j (cid:88) k = i n k  n i + n j 2 ) 2 , (7) and for interval data,  2 ij = | i  j | 2 (Krippendorff, 2018). We shall refer to these two versions of  as  -ORD and  -INT, respectively. Unlike  , the  's can evaluate the aforementioned baseline systems without any problems. The three measures defined below ignore the ordinal nature of the classes. That is, they are axiomatically incorrect as OC evaluation measures. First, let us consider two different definitions of Macro F1 found in the literature (Opitz and Burst, 2019): to avoid confusion, we give them different names in this paper. For each j  C + , let Prec j = c jj /c j  if c j  > 0 , and Prec j = 0 if c j  = 0 (i.e., the system never chooses Class j ). Let Rec j = c jj /c  j . Also, for any positive values p and r , let f 1( p, r ) = 2 pr/ ( p + r ) if p + r > 0 , and let f 1( p, r ) = 0 if p = r = 0 . Then: F 1 M = 1 | C + | (cid:88) j  C + f 1( Prec j , Rec j ) . (8) Now, let Prec M = (cid:80) j  C + Prec j / | C + | , Rec M = (cid:80) j  C + Rec j / | C + | , and HMPR = f 1( Prec M , Rec M ) . (9) HMPR stands for Harmonic mean of Macroaveraged Precision and macroaveraged Recall . Opitz and Burst (2019) recommend what we call F1 M over what we call HMPR. Again, note that our formulations use C + to clarify that empty gold classes are ignored. Finally, we also consider Accuracy: 5 Accuracy = (cid:80) j  C c jj N . From Eqs. 2 and 10, it is clear that MAE  and Accuracy ignore class imbalance (Baccianella et al., 2009), unlike the other measures. In an OQ task, a comparison of an estimated distribution and the gold distribution over | C | ordinal classes yields one effectiveness score, as described below. The systems are then evaluated by mean scores over the test instances , e.g., topics (Nakov et al., 2016; Rosenthal et al., 2017) or dialogues (Zeng et al., 2019, 2020). Let p i denote the estimated probability for Class i , so that (cid:80) i  C p i = 1 . Similarly, let p  i denote the true probability. We also denote the entire probability distributions by p and p  , respectively. Let cp i = (cid:80) k  i p k , and cp  i = (cid:80) k  i p  k . Normalised Match Distance (NMD) used in the NTCIR Dialogue Quality Subtasks (Zeng et al., 2019, 2020) is given by (Sakai, 2018a): NMD ( p, p  ) = (cid:80) i  C | cp i  cp  i | | C |  1 . 5 Since the system and the gold data have the same total number of items to classify (i.e., N ), accuracy is the same as microaveraged F1/recall/precision. This is simply a normalised version of EMD used in the OQ tasks of SemEval (See Section 2.2) (Nakov et al., 2016; Rosenthal et al., 2017). We also consider two measures that can handle OQ tasks from Sakai (2018a). First, a Distance-Weighted sum of squares for Class i is defined as: DW i = (cid:88) j  C | i  j | ( p j  p  j ) 2 . Note that the above assumes equidistance. Let C  = { i  C | p  i > 0 } . That is, C  is the set of classes with a positive gold probability. Order-aware Divergence is defined as: OD ( p (cid:107) p  ) = 1 | C  | (cid:88) i  C  DW i , (13) with its symmetric version SOD ( p, p  ) = ( OD ( p (cid:107) p  ) + OD ( p  (cid:107) p )) / 2 . Root (Symmetric) Normalised Order-aware Divergence is defined as: RNOD ( p (cid:107) p  ) = (cid:115) OD ( p (cid:107) p  ) | C |  1 , (14) RSNOD ( p, p  ) = (cid:115) SOD ( p, p  ) | C |  1 . (15) The other three measures defined below ignore the ordinal nature of the classes (Sakai, 2018a); they are axiomatically incorrect as OQ measures. Normalised Variational Distance (NVD) is essentially the Mean Absolute Error (MAE): NVD ( p, p  ) = 1 2 (cid:88) i  C | p i  p  i | . (16) Root Normalised Sum of Squares (RNSS) is essentially the Root Mean Squared Error (RMSE): RNSS ( p, p  ) = (cid:114)(cid:80) i  C ( p i  p  i ) 2 2 . (17) The advantages of RMSE over MAE is discussed in Chai and Draxler (2014). The Kullback-Leibler divergence (KLD) for system and gold probability distributions over classes is given by: KLD ( p (cid:107) p  ) = (cid:88) i  C s . t . p i > 0 p i log 2 p i p  i . (18) As this is undefined if p  i = 0 , we use the more convenient Jensen-Shannon divergence (JSD) instead, which is symmetric (Lin, 1991): JSD ( p, p  ) = KLD ( p (cid:107) p M ) + KLD ( p  (cid:107) p M ) 2 , (19) where p Mi = ( p i + p  i ) / 2 . Table 1 provides an overview of the SemEval and NTCIR task data that we leveraged for our OC and OQ meta-evaluation experiments. From SemEval-2016/2017 Task 4 (Sentiment Analysis in Twitter) (Nakov et al., 2016; Rosenthal et al., 2017), we chose Subtask C as our OC tasks, and Subtask E as our OQ tasks for the reason given in Section 2.1. 6 Moreover, for the OQ meta-evaluation experiments, we also utilise the DQ (Dialogue Quality) subtask data from NTCIR-15 DialEval-1 (Zeng et al., 2020). As these subtasks require participating systems to estimate three different dialogue quality score distributions, namely, A-score (task accomplishment), E-score (dialogue effectiveness), and S-score (cus-tomer satisfaction), we shall refer to the subtasks as DQ-A, DQ-E, and DQ-S hereafter. We utilise both Chinese and English DQ runs for our OQ meta-evaluation (22 runs in total), as the NTCIR task evaluates all runs using gold distributions that are based on the Chinese portion of the parallel dialogue corpus (Zeng et al., 2020). As the three NTCIR data sets are larger than the two SemEval data sets both in terms of sample size and the num-6 We do not use the Arabic data from 2017 as only two runs were submitted to Subtasks C and E (Rosenthal et al., 2017). ber of systems, we shall focus on the OQ meta-evaluation results with the NTCIR data; the results with Sem16T4E and Sem17T4E can be found in the Appendix. Table 2 shows, for each OC task, the Kendall's  rank correlation values (Sakai, 2014) between two system rankings for every pair of measures. We can observe that: (A) the  's, the two Macro F1 measures (F1 M and HMPR), MAEM and  produce similar rankings; (B) MAE  and Accuracy (i.e., the two measures that ignore class imbalance) produce similar rankings, which are drastically different from those of Group A; and (C) CEMORD produces a ranking that is substantially different from the above two groups, although the ranking is closer to those of Group A. The huge gap between Groups A and B strongly suggests that MAE  and Accuracy are not useful even as secondary measures for evaluating OC systems. It should be noted that the SemEval 2016/2017 Task 4 Subtask C actually reported MAE  scores in addition to the primary MAEM scores, and the Measure Mean  Measure Mean  Sem16T4C system rankings according to these two measures were completely different even in the official results. For example, in the 2016 results (Table 12 in Nakov et al. (2016)), while the baseline run that always returns neutral is ranked at 10 among the 12 runs according to MAEM , the same run is ranked at the top according to MAE  . Similarly, in the 2017 results (Table 10 in Rosenthal et al. (2017)), a run ranked at 10 (tied with another run) among the 20 runs according to MAEM is ranked at the top according to MAE  . Our results shown in Table 2 generalise these known discrepancies between the rankings. For each measure, we evaluate its system ranking consistency (or robustness (Amigo et al., 2020)) across two topic sets as follows (Sakai, 2021): (1) randomly split the topic set in half, produce two system rankings based on the mean scores over each topic subset, and compute a Kendall's  score for the two rankings; (2) repeat the above 1,000 times and compute the mean  ; (3) conduct a randomised randomised paired Tukey HSD test at  = 0 . 05 with 5,000 trials on the mean  scores to discuss statistical significance. 7 Table 3 (a) and (c) show the consistency results with the OC tasks. For example, Part (a) shows that when the 100 topics of Sem16T4C were randomly split in half 1,000 times,  statistically significantly outperformed all other measures, as indicated by a  (cid:93) . Table 3 (b) and (d) show variants of these experiments where only 10 topics are used in each topic subset, to discuss the robustness of measures to small sample sizes. If we take the averages of (a) and (c), the top three measures are the two  's and  , while the worst two measures are CEMORD and Accuracy; we obtain the same result if we take the averages of (b) and (d). Thus, although Amigo et al. (2020) reported that CEMORD performed well in terms of robustness, this is not confirmed in our experiments. Recall that  has a practical inconvenience: it cannot distinguish between baseline runs that always return the same class. While SemEval16T4C contains one such run (which always returns neutral ), SemEval17T4C contains as many as five such runs (each always returning one of the five ordinal classes). This is probably why  performs well in Table 3(a) and (b) but not in (c) and (d). In the information retrieval research community, discriminative power (Sakai, 2006, 2007, 2014) is a widely-used method for comparing evaluation measures (e.g., Anelli et al. (2019); Ashkan and Metzler (2019); Chuklin et al. (2013); Clarke et al. (2020); Golbus et al. (2013); Lu et al. (2016); Kanoulas and Aslam (2009); Leelanupab et al. (2012); Robertson et al. (2010); Valcarce et al. (2020)). Given a set of systems, a p -value for the difference in means is obtained for every system pair (preferrably with a multiple comparison procedure (Sakai, 2018b)); highly discriminative measures are those than can obtain many small p -values. While highly discriminative measures are not necessarily correct , we do want measures to be sufficiently discriminative so that we can draw some useful conclusions from experiments. Again, we use randomised paired 7 The Tukey HSD (Honestly Significant Differences) test is a multiple comparison procedure: that is, it is like the t test, but can compare the means of more than two systems while ensuring that the familywise Type I error rate is  . The randomised version of this test is free from assumptions such as normality and random sampling from a population (Sakai, 2018b). Tukey HSD tests with 5,000 trials for obtaining the p -values. Figure 1 shows the discriminative power curves for the OC tasks. Curves that are closer to the origin (i.e., those with small p -values for many system pairs) are considered good. We can observe that (i) CEMORD , Accuracy, MAEM , and MAE  are the least discriminative measures in both tasks. (ii) Among the other measures that perform better,  performs consistently well. Again, the fact that  distinguishes itself from others in the SemEval16T4C results probably reflects the fact that the data set contains only one run that always returns the same class, which cannot be handled properly by  . Table 4 summarises the properties of the nine measures we examined in the context of OC tasks. Column (IV) shows that, for example, the Group A measures produce similar rankings. Based on this table, we recommend (Linear Weighted)  as the primary measure for OC tasks if the tasks do not in-(I) DQ-A RSNOD RNOD NVD JSD NMD RNSS 0.835 0.913 0.939 0.905 0.636 RSNOD -0.870 0.861 0.827 0.766 RNOD -0.939 0.939 0.723 NVD --0.931 0.680 JSD --0.714 (II) DQ-E RSNOD RNOD NVD JSD NMD RNSS 0.931 0.922 0.913 0.913 0.688 RSNOD -0.957 0.948 0.948 0.758 RNOD -0.957 0.991 0.749 NVD --0.948 0.758 JSD --0.758 (III) DQ-S RSNOD RNOD NVD JSD NMD RNSS 0.861 0.974 0.957 0.922 0.558 RSNOD -0.887 0.887 0.853 0.662 RNOD -0.983 0.948 0.584 NVD --0.965 0.584 JSD --0.619 Table 5: System ranking similarity in terms of Kendall's  for each OQ task (NTCIR). volve multiple baseline runs that always return the same class. Such runs are unrealistic, so this limitation may not be a major problem. On the other hand, if the tasks do involve such baseline runs (as in SemEval), we recommend  -ORD as the primary measure. In either case, it would be good to use both  and  -ORD to examine OC systems from multiple angles. According to our consistency and discriminative power experiments, using  -INT instead of  -ORD (i.e., assuming equidistance) does not seem beneficial for OC tasks. Table 5 shows, for each OQ task from NTCIR, the Kendall's  between two system rankings for every pair of measures. It is clear from the NMD column that NMD is an outlier among the six measures. In other words, among the only axiomatically correct measures for OQ tasks, RNOD and RSNOD are the ones that produce rankings that are similar to those produced by well-known measures such as JSD and NVD (i.e., normalised MAE; see Eq. 16). Also, in Table 5(I) and (III), it can be observed that the ranking by RSNOD lies somewhere between that by NMD (let us call it Group X) and those by the other measures (Group Y). However, this is not true in Table 5(II), nor with our SemEval results (See Appendix Table 8). Table 6 shows the system ranking consistency results with the OQ tasks from NTCIR. These experiments were conducted as described in Section 5.2. If we take the averages of (a), (c), and (e) (i.e., experiments where the 300 dialogues are split in half), the worst measure is NMD, followed by RSNOD. Moreover, the results are the same if we take the averages of (b), (d), and (f) (i.e., experiments where two disjoint sets of 10 dialogues are used), we obtain the same result. Hence, among the axiomatically correct measures for OQ tasks, RNOD appears to be the best in terms of system ranking consistency, and that introducing symmetry (Com-pare Eqs. 14 and 15) may not be a good idea from a statistical stability point of view. Note that, for comparing a system distribution with a gold distribution, symmetry is not a requirement. Figure 2 shows the discriminative power curves for the OQ tasks from NTCIR. We can observe that: (i) NMD performs extremely poorly in (I) and (III), which is consistent with the full-split consistency results in Table 6(a) and (e); (ii) RNOD outperforms RSNOD in (I) and (III). Although RSNOD appears to perform well in (II), if we consider the 5% significance level (i.e., 0.05 on the y -axis), the number of statistically significantly different pairs (out of 231) is 117 for RNOD, 116 for RSNOD, NMD, and NVD, and 115 for RNSS and JSD. That is, RNOD performs well in (II) also. These results also suggest that introducing symmetry to RNOD (i.e., using RSNOD instead) is not beneficial. Table 7 summarises the properties of the six measures we examined in the context of OQ tasks. Column (III) indicates that NMD is an outlier in terms of system ranking. Based on this table, we recommend RNOD as the primary measure of OQ tasks, as evaluating OQ systems do not require the measures to be symmetric. As a secondary measure, we recommend NMD (i.e., a form of Earth Mover's Distance) to examine the OQ systems from a different angle, although its statistical stability (in terms of system ranking consistency and discriminative power) seems relatively unpredictable. Although the NTCIR Dialogue Quality subtasks (Zeng et al., (I) (II) (III) (IV) (V) NMD (cid:88) (cid:88) X poor poor RSNOD (cid:88) (cid:88) Y poor fair RNOD (cid:88) Y good fair NVD (cid:88) Y good fair RNSS (cid:88) Y good fair JSD (cid:88) Y good fair Table 7: Summary of the properties of OQ measures. 2019, 2020) have used NMD and RSNOD as the official measures, it may be beneficial for them to replace RSNOD with RNOD. We conducted extensive evaluations of nine measures in the context of OC tasks and six measures in the context of OQ tasks, using data from SemEval and NTCIR. As we have discussed in Sections 5.4 and 6.4, our recommendations are as follows. OC tasks Use (Linear Weighted)  as the primary measure if the task does not involve multiple runs that always return the same class (e.g., one that always returns Class 1, another that always returns Class 2, etc.). Otherwise, use  -ORD (i.e., Krippendorff's  for ordinal classes) as the primary measure. In either case, use both measures. All of our evaluation measure score matrices are available from https://waseda.box.com/ ACL2021PACKOCOQ , to help researchers reproduce our work. Among the above recommended measures, recall that Linear Weighted  and RNOD assume equidistance (i.e., they rely on w ij = | i  j | ), while  -ORD and NMD do not. Hence, if researchers want to avoid relying on the equidistance assumption (i.e., satisfy the ordinal invariance property (Amigo et al., 2020)),  -ORD can be used for OC tasks and NMD can be used for OQ tasks. However, we do not see relying on equidistance as a practical problem. For example, note that the Linear Weighted  is just an instance of the Weighted  family: if necessary, the weight w ij can be set for each pair of Classes i and j according to practical needs. Similarly, w ij = | i  j | (Eq. 12) for RNOD (and other equidistance-based measures) may be replaced with a different weighting scheme (e.g., something similar to the prox ij weights of CEMORD ) if need be. Our final and general remark is that it is of utmost importance for researchers to understand the properties of evaluation measures and ensure that they are appropriate for a given task. Our future work includes evaluating and understanding evaluation measures for tasks other than OC and OQ. This work was partially supported by JSPS KAK-ENHI Grant Number 17H01830.
abstain abstain abstain abstain abstain abstain abstain abstain result method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain result method objective abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain other abstain method abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain method method abstain abstain other abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain other abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method method other abstain method method abstain abstain abstain other abstain abstain result abstain abstain abstain method other