The lack of meaningful automatic evaluation metrics for dialog has impeded open-domain dialog research. Standard language generation metrics have been shown to be ineffective for evaluating dialog models. To this end, this paper presents USR , an U n S upervised and R eference-free evaluation metric for dialog. USR is a reference-free metric that trains unsupervised models to measure several desirable qualities of dialog. USR is shown to strongly correlate with human judgment on both Topical-Chat (turn-level: 0.42 , system-level: 1.0 ) and PersonaChat (turn-level: 0.48 and system-level: 1.0 ). USR additionally produces interpretable measures for several desirable properties of dialog. The lack of meaningful automatic evaluation metrics is a significant impediment for open-domain dialog generation research. Standard language generation metrics have been shown to be ineffective for dialog evaluation (Deriu et al., 2019; Liu et al., 2016). Without well-accepted, meaningful automatic metrics, open-domain dialog researchers have come to rely on human evaluation. Due to its timeand cost-intensive nature, human evaluation is typically only used for the final dialog model. As such, during development dialog systems are generally optimized for poorly-correlated automatic metrics (e.g., F-1, BLEU, PPL) which can result in sub-par human evaluation scores (Di-nan et al., 2019). To facilitate development of open-domain dialog models with meaningful automatic metrics, this paper presents the U n S upervised and R eference free ( USR ) evaluation metric for dialog. Standard automatic metrics for evaluating dialog generation (e.g., BLEU, F-1, METEOR, ROUGE) have several shortcomings that make them unsuitable for dialog evaluation: (1) The one-to-many nature of dialog (Zhao et al., 2017) makes word-overlap metrics ineffective for scoring valid system output that deviates from the ground-truth response (Liu et al., 2016; Gupta et al., 2019). (2) Human evaluation of dialog typically measures multiple properties (e.g., appropriate, interesting, consis-tent). Automatic metrics on the other hand, condense the multi-faceted nature of dialog quality to a single uninterpretable metric. (3) There are many definitions of what a good dialog is and, as such, it is not feasible to construct a one size fits all metric. Depending on the task and the data, the desired qualities of a dialog system may differ (Walker et al., 1997; Deriu et al., 2019). USR is a reference-free metric that consists of several interpretable sub-metrics which are combined in a configurable manner. Rather than relying on a ground-truth reference response, unsupervised models are trained to measure desired qualities of dialog (e.g., interesting, natural). As such, USR (1) alleviates the one-to-many issue of standard metrics, (2) produces interpretable measures for desirable properties of dialog, and (3) provides a configurable mechanism for combining several sub-metrics into an overall quality score. To evaluate the performance of USR, human quality annotations were collected for models trained on the Topical-Chat (Gopalakrishnan et al., 2019) and the PersonaChat corpora (Zhang et al., 2018). USR is shown to strongly correlate with human judgment on both Topical-Chat (turn-level Spearman: 0.42 , system-level Spearman: 1.0 ) and PersonaChat (turn-level Spearman: 0.48 and system-level Spearman: 1.0 ). The strong correlation with human judgment across two datasets and a variety of model types shows that USR is a valuable tool for the dialog community. Further, since USR does not require any explicit supervision, it has the potential to generalize to several dialog tasks and datasets. The contributions of this paper as as follows: (1) a strongly-correlated, unsupervised and reference free metric is proposed for evaluating open-domain dialog systems, (2) a thorough human quality annotation is carried out and is released 1 to facilitate future benchmarking of dialog evaluation metrics. Standard automatic metrics for language generation correlate poorly with human judgement of dialog (Liu et al., 2016; Lowe et al., 2017; Gupta et al., 2019). For example, the F-1 score can be gamed by outputting the most frequent words, regardless of the context (Dinan et al., 2019). The poor performance of present metrics is largely due to the one-to-many nature of dialog (Zhao et al., 2017). To avoid comparing to a single reference response, several authors have proposed using multiple reference responses. Multiple reference responses can be obtained with retrieval models (Galley et al., 2015; Sordoni et al., 2015) or through data collection (Gupta et al., 2019). These multi-reference metrics show improvement in performance, but it is infeasible to thoroughly cover the space of potential responses. As such, this paper addresses the one-to-many issue of dialog by presenting a reference-free metric. Lowe et al. (2017) train ADEM to produce a quality score conditioned on the dialog context, the reference response and the generated response. Venkatesh et al. (2018) present a framework for evaluation of Alexa prize conversations, which attains moderate correlation with user ratings. Both of these methods are trained on explicit quality annotations. In contrast, USR requires no explicit supervision and will more easily generalize to new datasets and tasks. Li et al. (2017) proposes a reference-free dialog evaluator which is trained to discriminate between human and generated responses. This work is similar to USR in that it evaluates the quality of a response without a reference or quality annotation training data. Using the evaluation model as a reward during reinforcement learning exhibited strong performance. However, correlation with human judgement was not evaluated. Intuitively, it appears insufficient to rely on a discriminator as a meaningful evaluation of dialog since this assumes that all human responses are perfect and all generated responses are imperfect. To evaluate the correlation of automatic metrics with human judgment, human quality annotation was carried out across two open-domain dialog corpora. Generated responses were obtained from several models described in Section 3.3. For each dialog context, an additional human response was also written. Human annotation was then carried out on sixty dialog contexts, with six responses per context for Topical-Chat (four system outputs, one newly-annotated human output, one original ground-truth response) and five for PersonaChat (one less system output). Each response was given six different scores: Understandable (0-1), Natural (1-3), Maintains Context (1-3), Interesting (1-3), Uses Knowledge (0-1), Overall Quality (1-5). Three annotators labeled each response. The task instructions were very detailed in order to minimize subjectivity in the quality annotations. For example, individuals may differ in their definition of Interesting (e.g., some individuals find football interesting, others do not). Thus, the instructions contained a clear, albeit somewhat rigid definition, of Interesting. The instructions for Overall Quality annotation, however, were less rigid and therefore those annotations contain some amount of annotator-specific subjectivity. The data collection and experiments with PersonaChat were carried out to assess the generality of the USR metric. As such, the annotation questions used were specifically tailored to Topical-Chat, but are still suitable for PersonaChat. The Topical-Chat dataset (Gopalakrishnan et al., 2019) is a large collection of human-human knowledge-grounded open-domain conversations that consists of 11,319 dialogs and 248,014 utterances. Following the same experimental setup as Gopalakrishnan et al. (2019), heuristics are employed to identify the most relevant fact for each response. As such, the task is to produce a response conditioned on both a dialog context and a fact. The PersonaChat dataset (Zhang et al., 2018) is a corpus of human-human persona-conditioned conversations that consists of 10,907 dialogs and 162,064 utterances. Each worker is asked to condition their responses on a persona, which we consider to be analogous to the facts in the Topical-Figure 1: On the Topical-Chat corpus, six responses are obtained for each dialog context. Four use the trained Transformer model with different decoding strategies. One is a new human-generated response. One is the original ground-truth. A similar setup was employed for PersonaChat, albeit with different models. A Transformer (Vaswani et al., 2017) is trained to produce the response, r , conditioned on dialog context, c , and fact, f . The input to the transformer is the concatenation of c and f , similar to Gopalakrishnan et al. (2019). The transformer consists of 6 layers, a hidden size of 512, randomly-initialized word embeddings of size 300, a dropout rate of 0.1 and it is trained for 50 epochs. A single Transformer model is trained, which matches the automatic metrics reported by Gopalakrishnan et al. (2019). Different decoding strategies are used to obtain four different outputs from this model. In addition to standard argmax sampling, nucleus sampling (Holtzman et al., 2019) is used at three different rates: p = { 0 . 3 , 0 . 5 , 0 . 7 } . The outputs from these four decoding strategies are listed with the original ground-truth utterance and a new human-generated response, for a total of six responses for each context, as shown in Figure 1. Three models were used to generate system outputs: a sequence-to-sequence model (Seq2Seq), an LSTM language model (LM) and a Key-Value Profile Memory Network (KV-MemNN). We use the pre-trained models provided in ParlAI 2 for the ConvAI2 competition (Dinan et al., 2019). A fourth open-source model was also used to produce output for quality annotation, however it 2 https://github.com/facebookresearch/ ParlAI/tree/master/projects/convai2 was ultimately excluded from the released dataset and experiments due to possible data leakage. Quality annotation was performed by six dialog researchers. Using a crowdsourcing platform, such as Amazon Mechanical Turk (AMT), would have allowed for more efficient and scalable annotation. However, crowdsourcing was not used because (1) the annotation instructions are lengthy, (2) a preliminary annotation pass was carried out, followed by a group discussion, (3) having many annotations from a few annotators allows examination of annotator-specific subjectivity. Annotators were provided with a set of instructions (Appendix A). A small preliminary annotation pass was carried out, with each individual annotating 5 dialog contexts (for a total of 30 re-sponses). The inter-annotator agreement was computed for each of the questions. The instructions were refined after the preliminary pass and a discussion meeting (e.g., Maintains Context was changed to be a 3-point rating instead of a 2-point rating). After the instructions were modified, the full annotation pass was carried out. Each response was rated according to the qualities mentioned at the beginning of this section. Instructions for each of qualities are summarized below:  Understandable (0 1): Is the response understandable given the previous context? Natural (1 3): Does the response seem to be something that a person would naturally say? Maintains Context (1 3): Does the response serve as a valid continuation of the preceding conversation? Interesting (1 3): Is the response dull or interesting? Uses Knowledge (0 1): Given the fact that the response is conditioned on, how well does the response use that fact? Overall Quality (1 5): Given your answers above, what is your overall impression of the quality of this utterance? The instructions contained detailed descriptions and examples of what constitutes a response in each category (e.g., what makes a response score Metric Spearman Pearson Topical-Chat Understandable 0.5102 0.5102 Natural 0.4871 0.4864 Maintains Context 0.5599 0.5575 Interesting 0.5811 0.5754 Uses Knowledge 0.7090 0.7090 Overall Quality 0.7183 0.7096 PersonaChat Understandable 0.2984 0.2984 Natural 0.4842 0.4716 Maintains Context 0.6125 0.6130 Interesting 0.4318 0.4288 Uses Knowledge 0.8115 0.8115 Overall Quality 0.6577 0.6603 Table 1: Inter-annotator agreement for all the metrics. For all the correlations presented in this table, p < 0 . 01 . 2 on Maintains Context ). These instructions were written to minimize subjectivity in the annotations, which results in clear, agreed upon definitions. For Topical-Chat, the full annotation consisted of 60 dialog contexts randomly sampled from the frequent test set , for a total of 360 responses scored on six different qualities. For PersonaChat, 60 dialog contexts were sampled from the ConvAI2 validation set, with a total of 300 responses scored on six different qualities. Each response was labeled by three different annotators. Annotators were randomly assigned to each dialog context. Inter-annotator agreements for the different ratings across both datasets are presented in Table 1. The correlation between each pair of annotations is computed and the average correlation over all the pairs is reported. Correlation is used instead of Cohen's Kappa in order to better account for the ordinal nature of the ratings (i.e., 4 should correlate better with 5 than 1 ), and to maintain consistency with the evaluation of the automatic metrics. Most inter-annotator correlations are above 0 . 4 , which indicates moderate to strong agreement. The low agreement for Understandable on PersonaChat is likely a consequence of the simple language in the dataset. Most responses are understandable, except for those requiring background knowledge (e.g., that cod' is an acronym for Call of Duty' ). Since the annotators have differing background knowledge, the few occasions where they fail to understand an utterance will differ, hence the lower agreement. The agreement for Overall Quality is relatively high (0.71 for Topical-Chat and 0.66 for PersonaChat) which suggests that any ambiguity in the specific dialog qualities is mitigated when the annotator is asked for an overall impression. Table 2 presents the scores for the different systems on each of the six qualities. Across both datasets and all qualities, the new human generated response strongly outperforms all other response types, even the original ground truth. This may be because the new human generated response was written with this quality annotation in mind, and as such is optimized for turn-level evaluation. On the other hand, the workers who produced the original ground-truth response, were more concerned with the quality of the overall dialog than with the quality of each individual response. On the Topical-Chat corpus, argmax decoding has a moderately higher performance over the nucleus sampling (Holtzman et al., 2019) methods. This should not be taken as an indication that argmax decoding is the superior method, since the hyperparameters (e.g., temperature) were not tuned for nucleus sampling. It should be noted that the objective was not to train and evaluate the best performing models, but instead to produce responses of varying qualities and obtain accurate human judgements of these responses. A regression was trained to map from the five ratings to the overall score in order to analyze the relationship between them. For better interpretability of the regression weights, the scores were normalized (using z-score) before training the regression. For better interpretability, a softmax was computed over the weights. Since individuals may differ in their definition of a good response , a specific regression is trained for each of the five annotators who labeled responses for the Topical-Chat corpus. Figure 2 displays the weights attributed to each of the five qualities by each of the annotators. Annotators attributed different weights to the specific features. For example, A3 emphasized naturalness while A2 paid more attention to whether a response was grounded on knowledge. Despite the differences across annotators, a good response was generally expected to be natural, maintain context, and be interesting. These annotator-specific weights demonstrate that individuals define good dialog differently. Future work could explore per-System Und (0-1) Nat (1-3) MCtx (1-3) Int (1-3) UK (0-1) OQ (1-5) Topical-Chat Original Ground-Truth 0.95 2.72 2.72 2.64 0.72 4.25 Argmax Decoding 0.60 2.08 2.13 1.94 0.47 2.76 Nucleus Sampling (0.3) 0.51 2.02 1.90 1.82 0.42 2.40 Nucleus Sampling (0.5) 0.48 1.92 1.93 1.72 0.34 2.29 Nucleus Sampling (0.7) 0.52 2.01 1.87 1.80 0.37 2.39 New Human Generated 0.99 2.92 2.93 2.90 0.96 4.80 PersonaChat Original Ground-Truth 0.99 2.89 2.82 2.67 0.56 4.36 Language Model 0.97 2.63 2.02 2.24 0.08 2.98 LSTM Seq2Seq 0.92 2.64 2.49 2.29 0.47 3.47 KV-MemNN 0.93 2.70 2.18 2.56 0.17 3.25 New Human Generated 1.00 2.97 2.88 2.87 0.96 4.80 Table 2: Average scores for the six different responses, on the six quality: Understandable, Natural, Maintains Context, Interesting, Uses Knowledge and Overall Quality. sonalized dialog evaluation wherein the evaluation metric is tailored to a specific individual. A potential criticism of this quality annotation could be that certain dialog qualities are missing. To address concerns about the completeness of the set of five qualities, a regression can be trained to produce the overall score conditioned on the quality ratings. The Spearman correlation between the predicted score and the original overall score is 0.9654 , which signifies that the set of qualities is thorough and contains enough information to reflect the overall quality of the response. This section describes the automatic metrics explored for evaluating generated responses. Section 4.1 describes several existing metrics that were studied. Section 4.2 presents USR, a novel unsupervised and reference-free metric. Several existing and easily-applicable metrics for dialog evaluation are compared. the list of available metrics is not exhaustive. Only the most commonly used and the most accessible are addressed. F-1 score computes the word-overlap between the generated response and the ground-truth, by taking the harmonic mean of the precision and recall. It is one of the four metrics used by the creators of the Topical-Chat dataset (Gopalakrishnan et al., 2019), along with perplexity and unique uni-gram/bigram counts. Dinan et al. (2019) described a simple adversarial example that attains a high F-1 score on PersonaChat. We produce a similar example for the Topical-Chat dataset and find that always outputting a concatenation of the ten most common tokens in the dataset ( . i the , that a to it is of ) attains an F-1 score of 25.6 which is a +3.6 improvement over the Transformer presented by Gopalakrishnan et al. (2019). BLEU (Papineni et al., 2002) is a well-known word overlap metric that computes n-gram precision between the generated sequence and the reference. Because precision favors shorter sentences, BLEU also adds a brevity penalty that punishes shorter sentences. BLEU has been found to correlate poorly with human judgment (Liu et al., 2016; Lowe et al., 2017; Gupta et al., 2019). METEOR (Denkowski and Lavie, 2014) was designed as an improvement on BLEU using a harmonic mean of precision and recall, as well as stemming and synonyms. ROUGE-L (Lin, 2004) identifies the longest common subsequence between the generated and reference sequence to better account for sentence-level structure when computing word overlap. Greedy Matching (Rus and Lintean, 2012) is an embedding-based metric that greedily matches each word in the generated sequence to a reference word based on the cosine similarity of their embeddings. The final score is then an average over all the words in the generated sequence. Embedding Average (Wieting et al., 2015) computes a sentence embedding for both the generated sequence and the ground-truth response by taking an average of word embeddings. The score is then a cosine similarity of the average embedding for both the generated and reference sequence. Vector Extrema (Forgues et al., 2014) follows a similar setup to Embedding Average, where the score is the cosine similarity between sentence embeddings. Rather than taking an average over word embeddings, this method identifies the maximum value for each dimension of the word embedding. Taking the maximum is motivated by the idea that common words will be de-emphasized as they will be closer to the origin. Vector Extrema has been shown to perform better on dialog tasks than other metrics (Gupta et al., 2019; Liu et al., 2016). Skip-Thought (Kiros et al., 2015) uses a recurrent neural network to produce a sentence-level embedding for the generated and reference sequences. A cosine similarity is then computed between the two embeddings. The implementation provided by Sharma et al. (2017) is used. BERTScore (Zhang et al., 2019) uses a pre-trained BERT (Devlin et al., 2018) model to greedily match each word in a reference response with one word in the generated sequence. By doing so, it computes the recall of the generated sequence. BERTScore was shown to have strong system-level and segment-level correlation with human judgment on several machine translation and captioning tasks. However, although it is a more sophisticated metric, it still compares word similarity between a reference and a generated sequence. While this method may work well for tasks where there is a limited space of outputs for each input (e.g., captioning, translation), it is ineffective at dealing with the one-to-many nature of dialog. This section proposes describes the USR metric, an unsupervised, reference-free evaluation metric for dialog. USR leverages pre-trained language models, specifically RoBERTa (Liu et al., 2019), to measure properties of dialog. USR is designed to be reference-free because there is no one right answer due to the inherent one-to-many nature of dialog (Zhao et al., 2017). Several sub-metrics were developed for the different qualities of dialog (e.g., Natural, Interesting, Uses Knowledge). While USR measures the overall quality of a response, its sub-metrics assess specific dialog qualities and therefore facilitate better understanding of a model's performance. The masked language modelling (MLM) metric uses a fine-tuned RoBERTa (Liu et al., 2019) model to estimate the likelihood of a response. RoBERTa is pre-trained on a massive amount of English data and fine-tuned on the corpus being evaluated (either Topical-Chat or PersonaChat), making it capable of identifying unnatural and incorrect responses. The likelihood estimated by the fine-tuned RoBERTa model is used as an automatic metric for evaluating the understandability and naturalness of responses. The RoBERTa-base model (Liu et al., 2019) is fine-tuned on the training set of the Topical-Chat corpus (Gopalakrishnan et al., 2019) using the implementation open-sourced by Wolf et al. (2019a). The language model is fine-tuned on only the dialog, without any of the facts, for a single epoch. RoBERTa uses both past and future context to predict a probability distribution for a masked word. The input sequence to MLM is a concatenation of a dialog context, c , and a response, r . One word at a time, each word in r is masked and its log likelihood is computed. Given the masked log-likelihood for the i -th word of r as l i , the value of the metric is then computed to be  (cid:80) | r | i l i . Figure 3 visualizes this process. Recent research has highlighted the complementary nature of dialog retrieval and generation with respect to multi-tasking (Wolf et al., 2019b) and pre-training (Mehri et al., 2019). Because of this complimentary nature, using dialog retrieval (DR) for evaluating generative models is an intuitive choice, especially for metrics like Maintains Context and Uses Knowledge . The fine-tuned RoBERTa model described in Section 4.2.1 is further fine-tuned for the retrieval task. This task is set up in the same manner as Figure 3: Visualization of the masked language modelling (MLM) metric. the Ubuntu dialog corpus (Lowe et al., 2015). The model is trained given a context x , a response r , and a binary label y indicating whether r is the true response or randomly sampled. The context x may consist of the dialog history and the fact, denoted c , or just the fact, denoted f . Two different versions of the dialog retrieval (DR) metric are trained, with different values of x . The DR metric score is defined to be the probability P ( y = 1 | x, r ) a given DR metric model produces. Though the DR metric is trained for the task of retrieval, this is done in an unsupervised manner. The retrieval task is an unsupervised task since it requires no additional labels during training (e.g., explicit quality annotations). The DR metric is appropriate for Maintains Context , Interesting and Uses Knowledge . If a retrieval model predicts that a generated response is contextually relevant to a dialog context, it indicates that the response Maintains Context . Likewise, if a retrieval model predicts that the response r is contextually relevant to fact f , it signifies that r most likely Uses Knowledge . Interesting is the measure of whether the response is dull/generic or if it provides some in-teresting/engaging information. The DR metric is trained to distinguish between a ground-truth response ( y = 1 ) and a randomly sampled response ( y = 0 ). Generic responses are applicable to many contexts, and will often appear as both ground-truth responses and randomly sampled responses. As such, the model will likely learn to assign a low probability distribution to these generic responses and will often output P ( y = 1 | r, x ) = 0 . 5 . As such, generic responses will generally be scored lower than other contextually relevant, interesting responses. The DR metrics will learn to favor responses that are unique to a given context x , rather than being applicable to many different contexts. Given meaningful automatic metrics for each of the five dialog qualities, USR combines the scores into an overall measure that correlates well with Overall Quality ratings. In Section 3.5, a regression model was trained to reproduce the overall score from each of the specific quality scores. The predictions of this regression model attained a 0.9654 Spearman correlation with the original scores. This same regression is used by USR on top of the automatic metrics presented in Sections 4.2.1 and 4.2.2. USR combines its sub-metrics into one measure of overall quality. This combination is configurable, adaptable to different datasets or tasks. For example, if a specific application prefers natural responses over interesting ones, the weights of the regression model can be adjusted. Analysis demonstrated that individuals used different weights when producing the overall score (Figure 2). USR might be able to be personalized for specific individuals by adjusting the weights of the regression model. This section evaluates all of the automatic metrics described in Section 4, by comparing them to human judgement. The best sub-metrics for each dialog quality are used as input for the regression model of the USR metric. While the best performing sub-metrics are not consistent across the two datasets, the USR metric nonetheless exhibits strong results. The annotations for the original ground-truth are not used for evaluation, in order to accurately compare referenced and reference-free metrics. Table 3 shows turn-level correlations of the best automatic metrics for each dialog quality on Topical-Chat. USR is shown to strongly outperform both word-overlap and embedding-based metrics across all of the dialog qualities. Interestingly, the best non-USR metric is consistently either METEOR or BERTScore  possibly because both methods are adept at comparing synonyms during evaluation. For some dialog qualities, the overall USR metric outperforms the best sub-metric. For example, USR does better for Maintains Context than USR-DR. This is likely because the information from the other sub-metrics (e.g., Uses Knowledge ) is valuable and effectively leveraged by USR. Table 4 reports the turn-level correlations of the best automatic metrics for each dialog quality on the PersonaChat corpus. Across all dialog qualities, USR strongly outperforms the word-overlap and embedding-based metrics. Conversations in PersonaChat generally consist of individuals communicating facts from their own persona in a relevant and coherent manner. As such, when models trained on PersonaChat produce subpar outputs, it is generally because the outputs either (1) do not effectively use the persona or (2) are not rele-vant/coherent to the dialog context. This explains why the correlations are significantly higher for Maintains Context and Uses Knowledge . As a consequence of PersonaChat's strong dependency on both the dialog context and the persona, USR-DR (x = c) which uses both the dialog context and the persona to perform dialog retrieval, generally outperforms all other metrics. Table 5 shows turn-level correlation with the Overall Quality ratings on Topical-Chat, for all of the automatic metrics. USR shows a strong improvement over all other methods. This strong performance can be attributed to two factors: (1) Metric Spearman Pearson Understandable BERTScore (base) 0.0685 0.0672 USR MLM 0.1186 0.1313 USR 0.1324 0.1241 Natural VectorExtrema 0.1375 0.1458 USR DR (x = c) 0.2291 0.1733 USR 0.2430 0.1862 Maintains Context METEOR 0.2564 0.2500 USR DR (x = c) 0.5625 0.6021 USR 0.5280 0.6065 Interesting BERTScore (base) 0.0491 0.0325 USR DR (x = c) 0.2634 0.0606 USR 0.0171 0.0315 Uses Knowledge METEOR 0.1719 0.1678 USR DR (x = c) 0.6309 0.4508 USR 0.3177 0.4027 Table 4: Turn-level correlations on Persona-Chat. the ability of MLM and DR to accurately quantify qualities of a generated response without a reference response, and (2) the ability of USR to effectively combine MLM and DR into a better correlated overall metric. USR shows a similar improvement over all other metrics on PersonaChat, as shown in Table 6. However, DR (x = c) outperforms USR despite the fact that four out of the five sub-metrics input into the USR regression are DR (x = c). This result is probably due to PersonaChat's strong dependancy on both dialog context and persona, both of which DR (x = c) explicitly leverages. We compute the system-level correlation between all automatic metrics and the Overall Quality ratings. USR significantly ( p < 0 . 01 ) outperforms all other metrics with a Spearman correlation of 1.0 on both datasets and Pearson correlations of 0.92 (Topical-Chat) and 0.82 (PersonaChat). The full set of system-level correlations can be found in the appendix. These results demonstrate USR's effectiveness. It strongly outperforms other metrics on both turn-level and system-level correlations. Gopalakrishnan et al. (2019) use the F-1 score as their primary automatic evaluation metric when presenting Topical-Chat. The results demonstrate a significant difference between USR and the F-1 score, suggesting that USR is a better metric for the Topical-Chat corpus. USR achieves statistically significant correlations with human judgement. The results hold across two datasets, Topical-Chat (Gopalakrishnan et al., 2019) and PersonaChat (Zhang et al., 2018). USR is configurable. Notably it is composed of several specific dialog quality sub-metrics. These sub-metrics are combined in a configurable manner, using a regression. For other tasks, datasets or even users, this regression can be adjusted, allowing qualities to be removed or re-weighted. Additional sub-metrics could be added. USR should be used alongside human evaluation. USR was created to facilitate development and tuning of dialog models. As such, USR can be used for model selection and hyperparameter tuning. USR should not be used to claim superior performance over another method. USR may not work with non-generative models, which were not addressed here. Responses produced by a model that is too similar to the evaluation models (e.g., to DR) are a particular concern. This paper presents USR , an U n S upervised and R eference-free evaluation metric for dialog. To address the shortcomings of standard metrics for language generation, USR (1) is reference-free, (2) is composed of multiple sub-metrics that evaluate specific qualities of dialog, (3) has a definition of good dialog that is configurable. Thus the metric may be adapted to different tasks and datasets. USR is shown to strongly correlate with human judgment on Topical-Chat (turn-level: 0.42 , system-level: 1.0 ) and PersonaChat (turn-level: 0.48 , system-level: 1.0 ). We thank the following individuals for their help with annotation: Evgeniia Razumovskaia, Felix Labelle, Mckenna Brown and Yulan Feng.
abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective other other other other other other method other other other other other method other other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain other