Evaluation of open-domain dialogue systems is highly challenging and development of better techniques is highlighted time and again as desperately needed. Despite substantial efforts to carry out reliable live evaluation of systems in recent competitions, annotations have been abandoned and reported as too unreliable to yield sensible results. This is a serious problem since automatic metrics are not known to provide a good indication of what may or may not be a high-quality conversation. Answering the distress call of competitions that have emphasized the urgent need for better evaluation techniques in dialogue, we present the successful development of human evaluation that is highly reliable while still remaining feasible and low cost. Self-replication experiments reveal almost perfectly repeatable results with a correlation of r = 0 . 969 . Furthermore, due to the lack of appropriate methods of statistical significance testing, the likelihood of potential improvements to systems occurring due to chance is rarely taken into account in dialogue evaluation, and the evaluation we propose facilitates application of standard tests. Since we have developed a highly reliable evaluation method, new insights into system performance can be revealed. We therefore include a comparison of state-of-the-art models (i) with and without personas, to measure the contribution of personas to conversation quality, as well as (ii) prescribed versus freely chosen topics. Interestingly with respect to personas, results indicate that personas do not positively contribute to conversation quality as expected. Evaluation of open-domain dialogue is particularly challenging and has been cited in high-profile competitions as a known open problem (Dinan et al., 2019). Challenges arise primarily from the fact that in real-world conversations there exists such a vast number of possible appropriate responses. Subsequently, dialogue evaluation that relies on comparison with pre-created reference dialogues incur substantial false-negative rates as many appropriate responses are unfairly penalized simply for not corresponding closely with references. In addition, evaluation faces further challenges with respect to the ability to fully take into account dialogue history. 1 In this paper, we present a new method of open-domain dialogue evaluation based on human assessment of live conversations with models that avoids the need for pre-created reference dialogues and ensures full familiarity with dialogue history, ticking two important boxes in terms of validity. Although live human evaluation of models has the advantage of being highly valid, reliability unfortunately cannot be assumed and developing methods of evaluation for language tasks that achieve high rater consistency has been challenging, often resulting in low levels of agreement between annotators (Finch and Choi, 2020; Callison-Burch et al., 2011, 2012; Bojar et al., 2013, 2014; Mehri and Eskenazi, 2020b). Despite challenges in this respect, our proposed method provides highly reliable evaluation, achieving a correlation of r = 0 . 969 in self-replication experiments. Additionally, the evaluation can be carried out cheaply and on a large scale through strict quality controlled crowd-sourcing, as well as including score standardization for fairer ranking of competing models. We make the data and code publicly available to aid future research. 2 2 Problems in Past Evaluations A common issue occurs that can potentially impact the validity of results is filtering the set of systems to be evaluated via automatic metric scores . Since metric scores are known to be a poor substitute 1 The protocol employed in this work was approved by the DCU Research Ethics Committee. for human assessment, this only results in the possibility that the best system according to human judges is inadvertently filtered out at this stage. For example, ConvAI2 (Dinan et al., 2019) ranked models firstly using automatic metrics before top models according to metric scores were assessed by crowd-sourced workers on Mechanical Turk, while similarly in the sixth Dialog System Technology Challenge (DSTC6) systems were filtered according to metric scores prior to human evaluation. In terms of the live evaluation, competitions such as ConvAI2 report such evaluations as highly challenging, with many of the resulting dialogues reported to be senseless, offensive , or simply not in line with instructions and ultimately live evaluation results have been discarded. Despite challenges, competitions that operate in the public domain, making data and evaluation techniques available to researchers (such as ourselves) should be applauded for such efforts. On the other hand, competitions that (for one reason or another) do not release data and evaluation techniques into the public domain have reported relative success in terms of human evaluation. However until such methods can be accessed and independently verified through replication studies, they will unfortunately have little impact . The first Amazon Alexa Socialbot Grand Challenge required human assessors to score how coherent and engaging conversations were on a 15 rating scale by two distinct groups: volunteer Amazon employees (experts), and general Alexa users (crowds) (Ram et al., 2018), are reported to achieve a correlation of overall scores for the two types of human assessors at 0 . 93 . The absolute average rating across all chatbots was reported to be 20% lower for experts compared to general users. In an additional effort to evaluate models, conversational user experience, coherence, engagement, domain coverage, topical diversity , and conversational depth were assessed (15 scale), with combined scores reported to correlate with those of general users at r = 0 . 66 . In addition to methods and data not being publicly available, correlations are difficult to interpret since no detail is provided about the number of judgments on which the correlation is calculated for example. In addition to competitions that generally aim to include human evaluation of systems, automatic metrics are often proposed for dialogue evaluation, themselves requiring a human evaluation data set on which to evaluate the proposed metric. However, inappropriate statistics are often applied. For example, Pang et al. (2020) propose a holistic metric to automatically evaluate four distinct aspects of dialogue, and a human evaluation experiment is deployed on Mechanical Turk using a 15 rating scale. The mean correlation between human assessors is reported as r = 0 . 61 . However, mean correlations are unfortunately difficult to interpret, since correlation coefficients are not additive , averages calculated in the usual way cannot be assumed to reflect central tendency, and unfortunately, the distribution of correlations is not reported (Alexan-der, 1990). Mehri and Eskenazi (2020b) propose USR (Un-Supervised and Reference-free), an unsupervised model that predicts the quality of dialog for a range of criteria using various rating scales: understandable (01 rating scale), natural (13), maintains context (13), interesting (13), uses knowledge (0 1); overall quality (15). Despite human evaluation being carried out by experts inter-annotator agreement levels varied depending on criteria being measured, ranging from as low as 0 . 298 . Additionally, although correlations between human assessments are reported as significant at p < 0 . 01 , despite such statistics often being reported for correlations, they are unfortunately not very meaningful in terms of their impact on correlation interpretation and can be somewhat misleading. Contrary to common expectations, even small effect sizes (low r ) can produce very low p-values (strong significance) in such tests. Aiming to achieve a significant correlation is an extremely low bar to reach in terms of consistency, since a low p-value in this case simply rejects the null hypothesis that the correlation is zero . In addition to the above issues, human evaluation of dialogue systems rarely take into account the fact that differences in performance can occur simply by chance. The method of human evaluation we propose provides a means of applying standard tests for statistical significance to avoid concluding differences that are highly likely to have occurred simply by chance. Crowd-sourcing with highly accurate quality control provides a potential mechanism to ensure the three most important criteria that makes an eval-6417 uation meaningful while still remaining feasible: validity, reliability and scalability. Subsequently, we ask crowd-workers to carry out live text-based chat with models prior to that same worker also rating the quality of the immediately preceding conversation. A continuous (0100) rating scale is employed with three main motivation points (Graham et al., 2013; Novikova et al., 2018; Li et al., 2019; Santhanam and Shaikh, 2019; Santhanam et al., 2020; Mille et al., 2020; Barrault et al., 2020; Howcroft et al., 2020). Firstly, continuous scales reduce potential bias when comparing the performance of competing models by enabling score standardization . The score distribution of each human assessor is standardized according to overall mean and standard deviation of all ratings provided by that assessor, thus removing any adverse effects of those employing overly harsh (or indeed lenient) scoring strategies. Secondly, the 0100 rating scale allows standard significance tests to score distributions of models to help determine which models significantly outperform others. Thirdly, and possibly most importantly, a continuous rating scale facilitates highly accurate quality control of crowd-sourced workers so that the evaluation can scale while still maintaining validity at a low cost. Each human assessor is firstly asked to carry out a live conversation with a randomly selected model, comprised of a minimum of 10 conversational inputs, before rating the quality of the conversation that just took place under a number of criteria shown in Figure 1. Note that the measurement criteria we employed are not immutable and we encourage to extend or adjust the criteria for future studies as necessary. A continuous rating scale is advantageous for several reasons but employment of such a scale raises the question of how it should be labeled. In evaluation of language tasks, adjectival scale labels, such as poor, low, medium, high, perfect/ okay, good, excellent , and so on, are often employed despite their likely contribution to annotator inconsistency (Loukina et al., 2020; Sorodoc et al., 2017). This is despite evidence of adjectival scale labels being problematic in terms of bias resulting from positively and negatively worded items not being true opposites of one another, and items intended to have neutral intensity in fact proving to have Robotic : It was obvious that I was talking to a chatbot as opposed to another human user. specific conceptual meanings. Alexandrov (2010) provides a summary of issues associated with adjectival labels. To avoid any such causes of inconsistency, we structure each rating as a simple Likert declarative statement and ask human assessors to rate the degree to which they agree with each of these statement, making it possible to keep the rating scale constant while only changing the statement for each measurement criteria. We ask judges to rate each conversation under the seven aforementioned measurement criteria (Figure 1) along with a continuous rating scale labeled only at each extreme with strongly disagree (left); strongly agree (right). We structure Human Intelligence Tasks (HITs) so that a sufficiently rich score distribution is collected from each individual worker who participated, asking each to hold six conversations, comprised of a shuffled arrangement of five dialogue models and a single quality control model. Many approaches to quality controlling the crowd employ gold-standard items as quality checks (Liu et al., 2013; Lasecki et al., 2014). This approach is however highly likely to allow low quality data to pollute the resulting evaluation, since any worker willing to assign high scores to all items will undeservedly pass this check. 3 The approach also runs in contrast to our aim of the same individual who took part in a live conversation to also assess its quality, as it relies on the use of pre-created gold standard conversations. Our quality control approach overcomes these challenges by deploying models in live conversations that have known distinct performance levels instead of asking workers to assess the quality of pre-existing known high quality conversations. Within a HIT, the five models m can produce some quality level of conversation and the model l produces known lower quality dialogues (lower than the five models). For a single worker who takes part in conversations with m and l , we then check how consistently the worker rated the conversations of l lower than m . This results in a quality control mechanism that does not ask workers to be consistent with other workers or to correctly rate gold standard dialogues but only assesses worker consistency by how consistently they distinguish between known distinct performance models and only with respect to their own conversation ratings . From a practical standpoint, creating a low performance model, l , is additionally far less challenging and costly than pre-creating a known set of high quality dialogues, and degraded models operate fully automatically. Low quality models produce outputs via generation of random responses with meaning distortion also applied. For random response degradation: Low quality responses are generated by random sampling responses from training set dialogues with the intention of disregarding any previous input from the user, so responses from the model are likely to be perceived as low quality since they have low relevance. To reduce the quality of conversations further, we apply meaning distortion : each response, r , is altered to distort its meaning by randomly selecting a sequence of words within that response and replacing it with a sequence of words sampled from a distinct training set dialogue, with the length of the replaced word sequence being determined by the number of words in r . The specific details are provided in Appendix A.1, and Figure 4 in Appendix A.4 gives a typical example. Hits subsequently consist of a total of six dialogues comprised of five genuine models and a single quality control model that generates meaning distorted and random responses. Crowd-sourced workers converse with each model before rating conversation quality (model order is shuffled and blind). Statistical significance tests are then applied to score distributions of workers for the ratings they attributed to ordinary models, m , relative to the low quality model, l . The resulting p -value is then employed as a means of rating worker consistency, and any worker with p > = 0 . 05 shows no significant difference between low and ordinary model quality and is filtered out. Scores are collected from workers who rate models on a 0100 rating scale, and we refer to these scores as raw scores. Scores for negative attributes, i.e. robotic and repetitive , are then reversed for ease of further comparison, 100  the original rating. A distribution of scores is extracted for each worker and raw scores are standardized according to each worker's mean and standard deviation, in order to iron out any differences in worker scoring strategy. Average standardized scores for each criteria are calculated, and an overall score is calculated as the average of all measurement criteria. In order to assess the reliability of the proposed method of human evaluation, we carry out a meta-evaluation in which we firstly examine individual human assessor consistency, before conducting a self-replication experiment. A number of models are required to function as a sample set of test systems, and for this purpose we employ available pre-trained models from ParlAI: 4 Poly-Encoder Transformer (Humeau et al., 2019), Bi-Encoder Transformer (Dinan et al., 2018), Sequence to Sequence (Sutskever et al., 2014), Key-Value Memory Networks (Miller et al., 2016), and a LSTM-based Model (Hochreiter and Schmidhuber, 1997). Within the evaluation setting of ConvAI2, each model is with a persona consisting of approximately five textual statements to emulate a personality. However, to increase the number of models and to provide an interesting comparison, we additionally include a version of each of the above models without any persona, resulting in 10 competing models. Hits were posted on the crowd-sourcing platform Amazon Mechanical Turk. 5 Firstly, and in order to evaluate the open-domain models in as realistic a setting as possible, we allow workers to choose the topic of conversation and input their chosen topic in a text field. The open nature of conversations should be noted however as something that influences the difficulty of producing consistent results 4 https://parl.ai/docs/zoo.html 5 https://www.mturk.com 6419 Topic Workers Ave. in our self-replication experiment. The fact that we allow human assessors to freely choose the topic of conversation means that differences in ratings could result from legitimate differences in performance when different topics are chosen by human assessors. We nonetheless test our evaluation allowing the user to choose the topic as this is part of our core aim for developing evaluation of dialogue truly in the open domain. Besides choosing a topic, we additionally asked workers to input their opinion of the topic they chose to discuss with models, categorizing the topic as either liked, ambivalent about it, or disliked . For example, if the topic they chose to discuss was dogs , we were curious to know if this was motivated by the fact that the worker liked or disliked dogs or indeed that they had chosen to discuss something they had no particular feeling about. Table 2 shows subsequent proportions (%) of workers, and the detailed instructions are introduced in Figure 5 in Appendix A.4. Perhaps unsurprisingly, the vast majority of workers chose to discuss something they liked (84% for workers who passed quality control). Nonetheless 7% of good workers were ambivalent about the topic they chose and 9% chose a topic they reported as disliking. Table 1 shows the number of workers who participated in the initial data collection run who freely chose the topic of conversation with models (Free run 1), amounting to 1,525 dialogues  7 criteria = 10,675 human ratings. The details of payment to each worker and the total experiment cost are provided in Appendix A.2. Table 1 also shows the proportion of workers who passed quality checks, numbers of dialogues assessed in total before and after quality filtering, as well as the average time taken for workers to complete a hit and average time taken to assess dialogues. As mentioned previously, we carry out a second data collection run with precisely the same settings (Free run 2) to measure the reliability of results and Table 1 shows equivalent statistics with respect to Free run 2 in which a total of 1,480 dialogues  7 ratings = 10,360 human ratings were collected in total. Although the overall aim of our evaluation is to produce reliable results at the system level, which we test later in Section 4.2, we firstly examine ratings of workers at the level of individual dialogue ratings. Technically speaking, the most meaningful reliability measures for continuous ratings scales test consistency of aggregate (system-level) results because although a high level of random error is expected in individual continuous rating scale scores, when aggregates are calculated for large samples of ratings, positive and negative error that is truly random effectively cancels itself out, and does not negatively impact consistency. In other words, the rating scale we employ does not rely on consistency on the level of individual ratings. We nonetheless examine individual rater consistency, since it is the standard approach, but keep in mind that results in this part of our meta-evaluation are not crucial when testing reliability for an evaluation carried out via a continuous rating scale where consistency in overall system-level results are more important. of agreement between pairs of workers who failed quality control is close to random as the distribution is approaching uniformity across almost the range of possible coefficients. In contrast, for pairs of workers who pass quality control, the peak of agreement is between an r of 0 . 6 and 0 . 7 , showing high agreement in general between such annotator pairs. Some of the observed disagreement is likely to be the result of legitimate differences between scores of two workers who chose distinct topics to discuss with the same model however, an unavoidable source of inconsistency when testing models with respect to the open domain. Interestingly, in 5% of dialogues, worker pairs assigned the same hit happened to both freely choose an identical topic to discuss with the same model. Furthermore, remaining disagreement at the level of individual ratings might not be problematic at the level of overall scores in relation to aggregation of ratings collected on a continuous rating scale. Table 3 shows results of the system-level evaluation resulting from the initial data collection run on Mechanical Turk (Free run 1), where competing models are ordered by overall highest average z-score. 6 Table 3 additionally shows consistency of the evaluation between each experiment run via the Pearson correlation of scores for each measurement criteria as well as consistency overall. Across 6 Average standardized scores for models in Free run 2 are additionally included in Table 9 in Appendix A.4; as well as equivalent average raw scores for models are in Table 10 in Appendix A.4. 94 in almost all cases with the exception of robotic which nonetheless achieved a correlation of over 0 . 7 . Besides individual criteria, of crucial importance is the consistency of overall results, as this is the means by which models would ordinarily be ranked in terms of overall performance. As can be observed from Table 3, the correlation reached in terms of overall scores for systems is 0 . 969 , which is very close to a perfect correlation, showing extremely high levels of reliability for the evaluation, evidence that the approach overcomes substantial challenges with respect to annotator consistency and expected difficulties with respect to evaluating models in the open domain, where assessors are legitimately free to choose distinct topics of conversation. In any empirical evaluation, statistical significance tests should be applied to take into account the fact that small differences in scores between systems can occur simply by chance. We provide pairwise significance test results in Figure 3, where we apply standard significance test, Wilcoxon rank-sum to rating distributions for each pair of competing models for each data collection run, and corresponding results for run 2 in Figure 6 in Appendix A.4. Results showed a very high proportion of identical conclusions, 84% , drawn from pairwise significance tests applied to data from the two data collection runs at p < 0 . 1 . Results for p < 0 . 05 , additionally showed high correspondence between pairwise significance test conclusions, only marginally lower with 82% of the same conclusions being drawn for pairs of models in the two data collection runs. We additionally provide correlations between measurement criteria and overall scores in Table 8 of Appendix A.4. Since we have verified the reliability of the human evaluation, we take a closer look at the results and investigate dialogue quality when models employ a persona. Results in Table 3 reveal that perhaps unexpectedly in general are either rated more favorably by human assessors when they carry out dialogues without a persona or a tie occurs between models with and without a persona. In contrast to the initial experiment in which workers were permitted to choose the topic of conversation, we further investigate the performance of models in a slightly easier setting where the topic under discussion is known to the model, by selecting a statement from its persona, which we refer to as an ice-breaker topic statement . An ice-breaker topic statement is then provided to human assessors at the beginning of each conversation, and the assessor is instructed to talk about this topic with the model. We therefore provide the topic of conversation to workers in the form of an ice-breaker topic statement , corresponding to a randomly selected persona statement belonging to the agent. Again, we run this experiment on MTurk, this time contrasting results for our initial data collection run where workers freely chose a topic with one in which workers were instructed to talk about the ice-breaker statement with models. Numbers of workers who participated in the Icebreaker run are provided in Table 1, while a breakdown of results for each model and overall average scores are shown in Table 4 as well as the correlation between scores for systems when a topic is freely chosen. Interestingly, in terms of absolute differences in raw scores, the best performing model achieves higher fluency, consistency and is deemed less repetitive when evaluated in icebreaker conversations compared those with freely chosen topics. Raw average scores for models in the Ice-breaker run are additionally provided in Table 11 in Appendix A.4. Relatively speaking, in terms of system rankings, no meaningful difference in relative performance is observed when models are tested in a scenario where the worker chooses a topic and when one is prescribed with an ice-breaker statement, as can be seen from the strong correlation between scores for models in Free run 1 and Ice-breaker evaluation as shown in Table 4. Additionally, significance test results for the Ice-breaker evaluation are provided in Figure 7 in Appendix A.4. In this experiment, we employed four prevailing word-overlap-based metrics as described in the following, whose scores are computed on the ConvAI2 test set. BLEU BLEU (Bilingual Evaluation Understudy) evaluate the quality of a system output by computing the n-gram precision according to human-generated references (Papineni et al., 2002). It also uses the brevity penalty to penalize short outputs. ROUGE-L ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a recall-adaptation of BLEU, whose wildly-applied variant is ROUGE-L (Lin and Hovy, 2003). It computes the precision and recall using longest common subsequence (LSC) instead of n-gram, and the F1 score of precision and recall is reported as the final score. METEOR METEOR (Metric for Evaluation of Translation with Explicit ORdering) was firstly proposed to overcome flaws of BLEU, like no usage of recall (Denkowski and Lavie, 2011). It computes the unigram precision and recall, and have a different mechanism of choosing the brevity penalty. GLEUGLEU (Google-BLEU) is a variety of BLEU (Wu et al., 2016) which computes the n-gram precision and recall instead of the standalone precision. The minimum of precision and recall is reported as the final GLEU score. The following introduces two reference-free automatic metrics we employed: FED and USR. Their 6423 FED m FED l USR USR-MLM FED FED (Fine-grained Evaluation of Dialog) is a pre-trained-model based metric to evaluate a textual conversation history (Mehri and Eskenazi, 2020a). Given a conversation c , a pre-trained model m , two predefined responses r p and r n ( p = positive and n = negative), the FED score is L m ( r p | c )  L m ( r n | c ) where L m ( r | c ) computes the likelihood that the model m will generate a response r to a conversation c . We employed medium and large DialoGPT (Zhang et al., 2020) as FED scorers, where the full list of predefined positive and negative responses are available in Table 7 in Appendix. USR USR (an UnSupervised Reference-free metric) uses the pre-trained model RoBERTa (Liu et al., 2019) to assess the quality of a conversation (Mehri and Eskenazi, 2020b). It consists of three sub-metrics: USR-MLM is to evaluate the understandability and naturalness, USR-DR(c) and USR-DR(f) are to evaluate the interestingness and consistency. The sub-metric scores then produce an overall score through a regression model. We compute the correlation between commonly applied automatic metrics and our human evaluation methods, including word-overlap-based metrics and reference-free metrics, as shown in Tables 5 and 6 respectively. As can be seen from Table 5, unfortunately no word-overlap metric achieves a strong positive correlation with human assessment, confirming once again that the invalidity of system rankings currently produced by automatic metric scores. In terms of reference-free metrics, results correspond better and are more encouraging. FED has the ability of distinguishing repetitive models, but for other criteria, it correlates weakly or even negatively with human. Meanwhile, despite USR only correlating marginally with human in terms of consistency and topic loyalty, USR-DR(f) correlates closest to human among the three sub-metrics, while it performs best on evaluating consistency and topic loyalty. Development of reliable evaluation of open-domain dialogue has been highlighted as a known open-problem. We overcome previous challenges and provide a new human evaluation methodology shown as highly consistent, with results for models correlating at r = 0 . 969 in two separate data collection runs. Our evaluation has the advantage of highly accurate quality control of crowd-sourcing, differences in scoring strategies to be ironed out via score standardization, applicability of standard significance testing while increasing the reliability of results. Support was provided by Noah's Ark Lab, Huawei, and Science Foundation Ireland in the ADAPT Centre for Digital Content Technology ( www.adaptcentre.ie ) at Trinity College Dublin and Dublin City University funded under the SFI Research Centres Programme (Grants 13/RC/2106_P2; 13/RC/2106) co-funded under the European Regional Development Fund.
abstain abstain abstain objective abstain abstain objective objective abstain abstain abstain abstain abstain abstain abstain abstain objective other objective abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain other abstain abstain abstain method other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain other method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain result other