Existing automatic evaluation systems of chatbots mostly rely on static chat scripts as ground truth, which is hard to obtain, and requires access to the models of the bots as a form of white-box testing. Interactive evaluation mitigates this problem but requires human involvement. In our work, we propose an interactive chatbot evaluation framework in which chatbots compete with each other like in a sports tournament, using flexible scoring metrics. This framework can efficiently rank chatbots independently from their model architectures and the domains for which they are trained. Evaluation of dialogue systems is an open problem. Existing automatic evaluation metrics for chitchat systems are similar to those for other text generation tasks (e.g., machine translation (Papineni et al., 2002), question-answering (Rajpurkar et al., 2016), summarization (Lin, 2004)), which depends on calculating word overlaps with reference responses. However, for chitchats, there are usually many alternative but plausible responses given a situation, perhaps more than any other text generation task mentioned above. A limited number of reference responses are not sufficient to determine how good a generated response is. Moreover, such static settings are not good at assessing an interactive, context-sensitive system. Interactive human evaluation metrics usually involve a Likert scale evaluation after a multi-turn conversation with the bot to be assessed. While this method is a step up from the previous static evaluation, it is difficult for human judges to give a concrete score to any bot. Comparing the performance of two bots is easier. Thus ACUTE-EVAL (Li et al.,  Kenny Q. Zhu is the corresponding author, and is partially supported by NSFC Grant No. 91646205, and SJTU-CMBCC Joint Research Scheme. 2019) asks the judges to make a binary judgment of who is better in conversations between two identical bots or between a human and a bot. A more advanced version of that is Spot The Bot (Deriu et al., 2020) which models the human evaluation of a conversation after the Turing test. However, such a process is still time-consuming and costly, compared with automatic evaluations. In our opinion, a good method for evaluating multi-turn conversational model/system should satisfy the following requirements: i) be as efficient and inexpensive as possible; ii) can truly reflect a model's ability to conduct a human conversation; iii) evaluation results should correlate well with human judgments; iv) can be used to compare and rank the capabilities of a set of models/systems. Toward that goal, in this work, we propose an automatic interactive evaluation framework, which is called ChatMatch (CM) for chitchat agents. This framework can be used to rank a number of bots with little time and minimum human effort. Above all, we want to emphasize the significance of direct interactions between bots in the evaluation. People tend to believe that human-bot conversations are more reliable and produce more comprehensive evaluations of chatbots' capabilities. This is not always true. As human annotators know their counterpart is a robot, they tend to ask common and goal-directed questions. On the other hand, some bot-bot chat logs in our experiments show that, surprisingly, conversations between different bots may expose their strengths and weaknesses never seen in human-bot conversations. Figure 1 gives two small chat fragments, illustrating such differences. While talking about hobbies, human keeps asking the bot some blunt questions, which leads to dull responses from the bot. However, in a bot-bot setting, two bots, including the same bot in the previous conversation, start explaining their hobbies to each other, producing a more interesting conversation. Human : What's your hobby? PLATO-2 : I've been playing the drums. Human : Why? PLATO-2 : It's just something I've always done. ! " # $ (a) Chat snippet between human and bot PLATO-2 : I love writing and playing the guitar but those are pretty common hobbies. Seq2Seq : I tend to write blogs and stuff mostly, but I am more of a literature teacher. PLATO-2 : Literature is cool, I've never been into reading deeply. Seq2Seq : Yeah, it's very relaxing and soothing. ! " # $ (b) Chat snippet between two bots Figure 1: Snippets from human-bot and bot-bot chat logs Our framework consists of two components: competition and scoring , which interoperate with each other. The competition is modeled after most sports tournaments such as soccer or ping pong. There are three levels of competitions. From bottom up, they are: game-level, match-level and tournament-level. Each match consists of several games. During a game, two bots will converse freely with each other and a virtual judge will score their performances according to a set of user-defined criteria such as consistency and fluency, etc. These criteria are flexible and extensible. The main contributions of this paper are:  We propose the first interactive evaluation framework for chatbots which is based solely on bot-bot conversations and modeled after sports competitions (Section 2.1). We designed three algorithms to score diversity , consistency , relevance , three important dimensions in a bot's chatting abilities. The entire scoring process is fully automated and efficient. In our experiments, the system can rank seven bots in less than three minutes on average (Section 2.2, Section 4.2). Our experiments show that the results produced by our framework closely correlate with the human evaluation results. Results also show that our framework outperforms several recent strong baseline evaluation systems (Section 4). In this section, we first introduce the general framework of ChatMatch, which is modeled as a sports tournament, then discuss some possible scoring metrics (or dimensions) that can be used by the virtual judges in these matches. We adopt a double round-robin sports tournament, where all bots participating in the competition converse directly with each other twice. This is better than a knock-out system because it assesses a bot's ability to deal with both strong and weak bots. If there are n chatbots to be evaluated, there will be (cid:0) n 2 (cid:1) matches in total. Each game is started by a player whose first utterance is provided by the system. The choice of the first utterance can be different depending on the domain of the bots and the ability we want to rank about the bots. For example, if we want to test the ability on movies, we can set a movie-related first utterance. To end the conversation, we set a fixed number of exchanges 1 2.2 Scoring 2.2.1 Game-level Scoring Inspired by Finch and Choi (2020), we score each dialogue turn based on seven aspects: consistency , fluency , knowledge , specificity , diversity , relevance and proactivity . Table 1 documents the definition of these dimensions and gives a brief view of tools that we used for automatic evaluation. Fluency, Knowledge, Proactivity and Specificity are scored for each turn separately and aggregated at the end of the conversation. We choose the most widespread reference-free approach, perplexity, to evaluate the fluency of each generated turn. Following Bao et al. (2021), we use the average of Distinct-1 and Distinct-2 (Li et al., 2016), which 1 An exchange of conversation is two turns, one from each speaker. computes the lexical variety, to approximate the specificity of a response. As for evaluating knowledge and proactivity, we count the number of entities and the number of questions of each bot. In need of considering the context while evaluating diversity, consistency and relevance, we propose more involved rules in Algorithm 1, Algorithm 2 and Algorithm 3. Table 2 shows the symbols we use in the algorithms. ishing repetition in one dialogue. At each turn t , we first check if there exists any repetitive question. Here in Figure 2, we can easily find turn 3 and turn 7 repeated turn 1 and turn 5 respectively. They will then be penalized one point for repetition. Repetition is not penalized if the previous turn is already marked as a repetitive question. For example, in Figure 2, although turn 4 is considered a repetition of turn 2, we are not going to penalize it as turn 3 is a repetitive question. Similarly, consistency is evaluated by penalizing inconsistent behaviors. The detection of inconsistency is always triggered after the detection 7581 of repeated questions. If the answers to the same questions are different, we will penalize the current turn, such as turn 8 in Figure 2. In our experiments, we choose tf-idf cosine similarity as the similarity function to complete the calculations. The actual diversity and consistency scores are the negation of the amount of repetition and inconsistency. Relevance is assessed as a bonus to reward a bot if it is able to memorize the important relevant concepts that have shown up before in the conversation. We sort the concepts that have shown up in chat history by their IDF scores. For example, in turn 9, A mentions the concept word student presented by B in turn 2. With this turn, A will win a bonus point. At the end of each game, each bot gets seven raw scores, one for each dimension. A bot receives one point on a dimension if it gets a higher raw score compared with the other bot in the game and zero point otherwise. The final score of each game for each bot is the sum of the points on these seven dimensions, which will not surpass 7. One naive method for adding up the scores which come from each game is to mimic the rules of sports tournaments: one match which consists of two games, each started with a different bot, decides winning or losing between two bots. Then for each match, we score W points for the winner, T points for a tie and L points for the loser. The value of W , T and L will be discussed in Section 4.4. For Tounrnament level, we count the points by summing up the scores gained in every match. Another choice is to use TrueSkill (Herbrich et al., 2007) algorithm to rank them. TrueSkill system is a ranking system which is based on Bayesian inference.This scoring system takes into account the uncertainty of each chatbot by considering their winning percentage and possible fluctuations. In this algorithm, the ability of each bot is regarded as a normal distribution. Each game result leads to an update of the bots' ability. In order to guarantee a stable ranking, we randomly shuffle the order of the game three times and get the final ranking on average. This section describes the chatbots that we experiment with, the set of baseline approaches that are compared to ChatMatch, and some implementation details used in the following experiments. We pick seven chitchat chatbots trained or finetuned on ConvAI2 (Dinan et al., 2019) to be evaluated in our experiments as Table 3 shows. All chatbots are running and evaluated on a Intel (R) Xeon (R) CPU E5-2678 v3 @ 2.50GHz with NVIDIA GeForce RTX 2080 and a 12GB RAM. DD Image Seq2Seq model. (Shuster et al., 2020) We choose four automatic evaluation methods and one manual evaluation method to compete with CM. For the baselines which depend on static scripts, we first make our seven bots generate responses for the evaluation using the test set of Dai-lyDialog (Li et al., 2017) as static scripts. Then we apply the following baselines: PPL: Perplexity Lower perplexity means that the generated sequence is more likely to be close to a human sentence. TA: Token Accuracy Token Accuracy is used to measure the generation accuracy of each token, which refers to the ratio of the number of correctly predicted tokens to the total number of predicted tokens. BS: BERTScore We use a commonly used automatic evaluation metric for text generation, BERTScore from Zhang et al. (2019). BERTScore will return a similarity score between referenced answer and generated response from the bot. HAE: Holistic and Automatic Evaluation We combine the four separate metrics which measures Fluency, Context Coherence, Logical Self Consistency and Diversity from Pang et al. (2020) by distributing each bot a per-metric score(1-7)first and then summing them up to get a final score. STB: Spot The Bot The recently proposed interactive manual evaluation metric Spot The Bot (Deriu et al., 2020) asks human judges to decide whether the speaker is human or bot with a mix of human-bot and bot-bot chat logs. In order to obtain rankings that can be reliably used as ground truth, we asked a group of human judges who are fluent in English to chat with each of the seven bots and then manually assess the ability of the bots. Seven dimensions, namely fluency, knowledge, proactivity, specificity, diversity, consistency, and relevance, are used to help them complete the ranking task which are rated on a 5-point Likert-scale. We trained the human judges by providing them one positive example and one negative example for each dimension, following the suggestions for improving the quality of human evaluation provided by Clark et al. (2021). Each judge can decide to stop the conversation whenever they feel confi-dent enough to score on these seven dimensions. We set the minimum number of exchanges to be 20. For each dimension, four judges participate in ranking bots' corresponding ability. After, we also ask four judges to provide their overall ranking based on their general impression. More details are shown in Appendix B. We use Kendall ranking correlation (  ) to evaluate the agreement among human judges and also between evaluation approaches and general human judgment. In the rest of the paper,  i and  g are used to denote inter-judge agreement and correlation between ranking produced by methods and the ground-truth ranking respectively. Table 4 shows  i on individual dimension and overall ranking. We believe human judgements on each dimension are reliable enough as all of them are greater than 0.6. Later, we will analyze different  g considering human overall rankings as ground-truth ranking. These settings are determined by empirics. The starting utterance is always set to a daily routing sentence since the players of our tournament are chitchat bots. For each game, the weight for individual dimension to be equal. Each tournament has 42 games (21 matches) in total. In this section, we first present the end-to-end results from our automatic framework and other baselines. Then we show that CM can generalize to different set of bots by applying CM on any 4-bot combination of the bot pool and check the results. Finally, we analyze the design of the framework with ablation tests in detail. We deploy the five baseline methods and our framework CM on the 7 chatbots. For simplicity, we convert the raw scores to rankings from 1 to 7 (the lower the better) by each of these methods. Figure 3 depicts these rankings, along with human ranking (HR) as a reference. BB ranks the top among all by CM and HR. This is not surprising because BB is a well-known competitive bot that performs well in many chitchat test sets and in different domains. The general trends exhibited by CM and STB track the human evaluation more closely, whereas the trends of TA and BS are almost the reverse of human judgment. PPL is a popular approach for evaluating the fluency of generated utterances, which can only evaluate whether the sequence generated by the model is close to human language without considering the context. It can only reflect a part of the ability of a chatbot. We also find that HAE tends to give a higher score to the shorter response, so BB and other bots that tend to generate longer sequences are not evaluated properly. We further compute  g of the baselines and ours against the human rankings and include them in Table 5. The correlation between our metric and human judgment is 0.81, This indicates that CM's evaluation results are very close to the average judgment made by four different human judges. Spot The Bot correlates well with human judgments as they depend on human annotations themselves. However, common automatic evaluation 7583 Fluency Knowledge Proactivity Specificity Diversity Consistency Relevance Overall  i 0.60 0.62 0.60 0.62 0.62 0.61 0.61 0.63 Table 4: Inter-judge agreement on seven different dimension as well as overall ranking. metrics such as PPL and HAE present poor agreements with human judgments. TA, which assesses whether the generated response matches the ground truth of the dataset, is even negatively correlated with human judges. The same goes for BS, which computes token similarity with contextual embed-dings. Since there may be many other plausible responses than the reference response itself, it is difficult to correctly evaluate the ability of the chatbot with static scripts. The efficiency of the competing methods is also assessed in Table 5. Though slower than other automatic methods, CM is much faster than methods requiring human efforts. It takes a human judge 90 minutes on average to complete the conversations with all 7 bots, decide ratings on seven dimensions and then give their overall ranking. To put it in perspective, we ask three human judges to evaluate the same 7 bots by Spot The Bot framework. It takes on average one hour for each human judge to complete the ranking. To justify that CM framework works for different sets of bots, we construct (cid:0) 74 (cid:1) = 35 test groups each of which consists of four randomly chosen bots from our bot pool. Next, we implement the double-round CM on these test groups. Among 35 test groups, we found  g of 29 test groups are higher than 0.60 while the average  g equals 0.73. This indicates that CM is capable of producing reliable rankings regardless of the number and the combination of participating bots. Table 6 shows the full results of our generalizability tests. We can tell that our framework is capable of predicting rankings for most of the combinations of bots. However, for a group of bots whose capabilities are relatively close, on which is even difficult for humans to reach an agreement(with  i relatively low in Table 6), it is still difficult for our framework to rank them accurately. Some of the chatlogs among these difficult bots are shown in Figure 5. Developing more precise metrics will be our next step work. The ChatMatch framework essentially consists of two main components: i) the bot-bot chat tournament set-up, and ii) the scoring metrics. Given its success in the end-to-end experiments, one natural question to ask is whether the high correlation with the human evaluation comes from the bot-bot set-up or the seven scoring metrics. If the scoring metrics are significant, which ones are more useful? In this subsection, we seek to answer these questions and also explore other factors or parameters in CM that might contribute to its effectiveness, such as the number of exchanges and starting utterance. rics to evaluate the human-bot chat logs collected from our human judges who chat with bots directly. For each bot, we obtain the per-metric rankings (1-7) first and then we sum up the 7 per-metric rankings for each bot to get their overall scores. The final ranking is decided by the overall scores. Self-chat conversations : we use our seven metrics to evaluate the generated 100-exchange self-chat logs and obtain the final rankings. The scoring process is similar to what we have done for the human-bot chat logs above. As the results in Table 7 show, CM gets the highest agreement among the three frameworks while implementing seven metrics on self-chat logs correlates weakly to human judgments. To figure out why our seven metrics do not work well with Human-bot setup and Self-chat setup, we calculate the average number of inconsistencies and relevant concepts caught by our metrics, regardless of the speaker. We can tell from Table 7 that Inconsistency is hardly detected in Self-chat logs as bots tend to chat about the same things within this setup. This can also explain why relevant concepts are often popular in self chat logs. Under this circumstance, these two metrics are not capable of distinguishing bots' real abilities. Evaluating relevance with our metric is difficult on human-bot logs. Human judges are often switching topics by raising some questions to shorten the evaluation time. That is why we are not able to evaluate bots' ability for memorizing some long-distance concepts. To understand the role and effects of each scoring dimension at game level, each time we set one dimension coefficient to zero and others remain one. We call these experiments minus x " experiments where x is one of the metrics under testing. Table 8 shows the agreement with human judges that comes from each minus x " experiment. Eliminating any of the metrics presents an effect on evaluation as  g has dropped. However, removing diversity does the most harm to evaluation as  g has dropped from 0.81 to 0.24. That is because diversity is usually the first thing that comes to human judges' minds while doing the evaluation. Improving lexical diversity and reducing repetition are still major challenges encountered by chat-bots developers. Additionally, to compare our scoring metrics with other unsupervised metrics, we try to add the metrics including Fluency, Context Coherence, Logic Self Consistency and Diversity from HAE (Pang et al., 2020) into CM. Since these four metrics are all reference-independent, we can test them with our bot-bot tournament framework. Results 7585 Fluency -Knowledge -Proactivity -Specificity -Diversity -Consistency -Relevance All  g 0.62 0.71 0.71 0.62 0.24 0.71 0.71 0.81 Table 8: Correlation between ranking produced by CM while eliminating different scoring metrics and ground truth ranking. shown in Table 9 indicate that the four metrics do not work well with bot-bot settings and metrics need to be carefully designed to suit bot-bot chatting. Metric  g CM (with HAE's metrics) 0.14 CM (with our metrics) 0.81 Table 9: Comparison with CM using HAE's metrics. 4.4.3 Effect of the Starting Utterance As most of the chitchat bots we test in the competition are for open domain, we use three types of starting utterances, namely greetings , declarative statements , and questions . We show one example for each type in Table 10. Example  g Greetings Hi! How are you? 0.62 Declarative Not feeling well this morning. 0.71 Question What did you do last week?  0.81 Table 10: The effects of different starting utterances. As Table 10 shows, all three starting utterances lead to a good correlation with human judgments. The model that starts with a question performs the best since raising a question is always a good way to start a conversation and make the bots start talking about it. We also find that when two bots are free to talk without human intervention, they prefer to steer the chat to their comfort zones (e.g., talking about their basic personal information) rather than stick to the ball started by CM. Hence, we decide to use the question in other experiments. We also test CM with a different number (e.g., 5, 10, 25, 50, 100, 150 and 200) of exchanges in a game. A game of no less than 100 exchanges reaches the best agreement between CM and human judgments. As a result, we use 100 exchanges in other experiment, which we believe is long enough to make the bots expose their flaws and show their strengths. Ranking is an inevitable part in sports competitions. Hence, in addition to TrueSkill ranking system, we have tried two other ranking methods commonly used in sports (Wikipedia):  win = 3, tie = 1, lose = 0  win = 2, tie = 1, lose = 0 With these sets of parameters,  g are both 0.62 while TrueSkill ranking setting correlates the best. TrueSkill ranking is capable of describing the ability of each bot in a more detailed way with their distribution than simply accumulating the points. It is commonly thought that we evaluators have less control over the bot-bot conversations than human-bot conversations as the automatic dialogue could veer into any direction. However, this does not mean that bot-bot chat logs are all that bad in quality, or less useful for evaluation. While going through bot-bot chat logs, we find that sometimes conversations between bots carry even more variety than that between human and bot. To demonstrate such serendipity, we present the average specificity score of the bot's utterances in Table 11 which indicates the variety of the use of words in dialogue. We can see that bots tend to generate longer responses while chatting with other bots. The diversity of words in bot-bot and human-bot conversations are quite close. More examples extracted from our collected bot-bot chat logs are shown in Appendix A. 7586 D-1 D-2 avg(D-1, D-2) Avg Len human-bot 0.44 0.89 0.67 12.8 bot-bot 0.49 0.83 0.66 15.7 Table 11: Average Distinct-1/2 and average lengths of bot utterances in different types of chat logs. Two major approaches are used for evaluating chatbots or dialogue systems. Some evaluate a single turn at a time by comparing the response with a ground truth utterance from a static script of real human dialogue or get a score by combining the response with the context. Others evaluate the interaction between human and bot or between bot and bot by some scoring metrics. We will present some typical systems from each of these approaches below and discuss their pros and cons. Most existing evaluation systems are based on static scripts. Traditional metrics such as BLEU (Papineni et al., 2002), ROUGE (Lin, 2004) and METEOR (Banerjee and Lavie, 2005) are widely used for evaluating text generation systems. More recently, automatic evaluation methods based on static scripts are gradually moving toward using pre-trained language models. Pang et al. (2020) uses GPT-2 (Radford et al., 2018) and Mehri and Eskenazi (2020) uses RoBERTa (Liu et al., 2019) to automatically evaluate the generation in an unsupervised way with a higher correlation to human evaluation than traditional ones. However, these static evaluation metrics which need fixed contexts are not flexible enough to assess chatbot's ability, in need of adapting to dynamic and changing contexts. We argue that interactive systems like ChatMatch are more promising as they test chatbots in a real conversation mode. Interactive evaluation systems attract increasing attention lately. Ghandeharioun et al. (2019) and Deriu and Cieliebak (2019) use dialogues between a bot and itself, which is called self-talk, to evaluate the bot in a more automatic manner. But it often leads to a lot of repeated chat context. Deriu et al. (2020) designed Spot The Bot , a framework that enables a group of bots to chat with each other and then asks humans to annotate if the bots talk more like a human or more like a bot. Prior to our work, there was still no interactive and automatic evaluation framework that works without any participation of human annotators. Our work fills this gap and moreover, it's very flexible since more complex metrics or algorithms can be plugged in as scoring functions. Our results also show that there is a strong correlation between automatic evaluation results and human judgments. In this work, we present a new automatic evaluation framework called ChatMatch. We first make the chatbots converse directly with each other. Then we use a three-level rule-based scoring framework to rank their performances which mimics the process of a double round-robin tournament. Our framework shows a good correlation with human judges, better than state-of-the-art automatic and semi-automatic chatbot evaluation frameworks. Another remarkable advantage of our framework is that it's totally automatic and time-saving which costs 2 min 57 secs on average to get the final ranking results among 7 chatbots, much faster than manual evaluation. We believe that this kind of automatic interactive evaluation framework opens up new opportunities for future research on dialogue system evaluation.
abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain objective abstain objective abstain abstain abstain result abstain abstain abstain objective objective objective objective objective objective objective objective objective objective objective objective abstain abstain abstain abstain abstain abstain abstain objective method objective abstain result result method method abstain abstain abstain abstain abstain abstain method abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain method method method abstain method abstain method abstain method other method other method other method method other other method method abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method method abstain method method abstain abstain method abstain method abstain other abstain abstain result abstain abstain abstain result abstain method abstain abstain abstain result abstain result abstain result other other other method other other other other other abstain other other other other abstain abstain abstain objective objective method result result objective