A good conversation requires balance  between simplicity and detail; staying on topic and changing it; asking questions and answering them. Although dialogue agents are commonly evaluated via human judgments of overall quality, the relationship between quality and these individual factors is less well-studied. In this work, we examine two controllable neural text generation methods, conditional training and weighted decoding, in order to control four important attributes for chitchat dialogue: repetition, specificity, response-relatedness and question-asking. We conduct a large-scale human evaluation to measure the effect of these control parameters on multi-turn interactive conversations on the PersonaChat task. We provide a detailed analysis of their relationship to high-level aspects of conversation, and show that by controlling combinations of these variables our models obtain clear improvements in human quality judgments. Neural generation models for dialogue, despite their ubiquity in current research, are still poorly understood. Well known problems, such as the genericness and repetitiveness of responses (Ser-ban et al., 2016a), remain without a de facto solution. Strikingly, the factors that determine human judgments of overall conversation quality are almost entirely unexplored. Most works have been limited to the next utterance prediction problem, whereas a multi-turn evaluation is necessary to evaluate the quality of a full conversation. In this work we both (i) conduct a large-scale study to identify the fine-grained factors governing human judgments of full conversations, and (ii) develop models that apply our findings in practice,  A.S. completed most of this work at Facebook (FAIR). leading to state-of-the-art performance. Specifi-cally, we identify and study eight aspects of conversation that can be measured by human judgments, while varying four types of low-level attributes that can be algorithmically controlled in neural models; see Figure 1. To control the low-level model attributes, we consider two simple but general algorithms: conditional training, in which the neural model is conditioned on additional control features, and weighted decoding, in which control features are added to the decoding scoring function at test time only. One major result of our findings is that existing work has ignored the importance of conversational flow, as standard models (i) repeat or contradict previous statements, (ii) fail to balance specificity with genericness, and (iii) fail to balance asking questions with other dialogue acts. Conducting experiments on the PersonaChat task (Zhang et al., 2018b), we obtain significantly higher engagingness scores than the baseline by optimizing control of repetition, specificity and question-asking over multiple turns. Using these findings, our best model matches the performance of the winning entry in the recent NeurIPS ConvAI2 competition (Dinan et al., 2019), which was trained on much more data but had no control (see Section 8.1). Our code, pretrained models, and full chatlogs, are available at https://parl.ai/projects/ controllable_dialogue . Dialogue Dialogue evaluation is relatively well understood in goal-oriented tasks, where automated approaches can be coded by measuring task completion (Bordes et al., 2017; El Asri et al., 2017; Hastie, 2012; Henderson et al., 2014; Wen et al., 2017). Task success combined with dialogue cost can be linked to human judgments like user satisfaction via the PARADISE framework (Walker et al., 1997). However in chitchat tasks, which we study in this work, automatic metrics and their relation to human ratings are less well-understood. While word-overlap metrics are effective for question-answering and machine translation, for dialogue they have little to no correlation with human judgments (Liu et al., 2016; Novikova et al., 2017)  this is due to the open-ended nature of dialogue. There are more recent attempts to find better automatic approaches, such as adversarial evaluation (Li et al., 2017b) and learning a scoring model (Lowe et al., 2017), but their value is still unclear. Nevertheless, a number of studies only use automatic metrics, with no human study at all (Lowe et al., 2015; Parthasarathi and Pineau, 2018; Ser-ban et al., 2016b). Other works do use human evaluations (Dinan et al., 2018; Li et al., 2016a,b; Venkatesh et al., 2017; Vinyals and Le, 2015; Zhang et al., 2018b), typically reporting just one type of judgment (either quality or appropriateness) via a Likert scale or pairwise comparison. Most of those works only consider single turn evaluations, often with a shortened dialogue history, rather than full multi-turn dialogue. A more comprehensive evaluation strategy has been studied within the scope of the Alexa prize (Venkatesh et al., 2017; Guo et al., 2018) by combining multiple automatic metrics designed to capture various conversational aspects (engagement, coherence, domain coverage, conversational depth and topical diversity). Though these aspects have some similarity to the aspects studied here, we also focus on lower-level aspects (e.g. avoiding repetition, fluency), to understand how they correspond to both our controllable attributes, and to overall quality judgments. Controllable neural text generation Researchers have proposed several approaches to control aspects of RNN-based natural language generation such as sentiment, length, speaker style and tense (Fan et al., 2018; Ficler and Goldberg, 2017; Ghazvininejad et al., 2017; Hu et al., 2017; Kikuchi et al., 2016; Peng et al., 2018; Wang et al., 2017). In particular, several works use control to tackle the same common sequence-to-sequence problems we address here (particularly genericness and unrelated output), in the context of single-turn response generation (Baheti et al., 2018; Li et al., 2016a, 2017a; Shen et al., 2017; Xing et al., 2017; Zhang et al., 2018a; Zhou et al., 2017). By contrast, we focus on developing controls for, and human evaluation of, multi -turn interactive dialogue  this includes a new method (described in Section 5) to control attributes at the dialogue level rather than the utterance level. In this work, we require a control method that is both general-purpose (one technique to simultaneously control many attributes) and easily tunable (the control setting is adjustable after train-ing). Given these constraints, we study two control methods: conditional training (variants of which have been described by Fan et al. (2018); Kikuchi et al. (2016); Peng et al. (2018)) and weighted decoding (described by Ghazvininejad et al. (2017) as a general technique, and by Baheti et al. (2018) to control response-relatedness). To our knowledge, this work is the first to systematically compare the effectiveness of two general-purpose control methods across several attributes. PersonaChat (Zhang et al., 2018b) is a chitchat dialogue task involving two participants (two humans or a human and a bot). Each participant is given a persona  a short collection of personal traits such as I'm left handed or My favorite season is spring  and are instructed to get to know each other by chatting naturally using their designated personas, for 68 turns. The training set contains 8939 conversations and 955 personas, collected via crowdworkers, plus 1000 conversations and 100 personas for validation, and a similar number in the hidden test set. The PersonaChat task was the subject of the NeurIPS 2018 ConvAI2 Challenge (Dinan et al., 2019), in which competitors were first evaluated with respect to automatic metrics (perplexity, hits@1 and F1 score), and then with respect to human judgment via the question How much did you enjoy talking to this user? on a scale of 14. Our baseline model is a 2-layer LSTM sequence-to-sequence model with attention. On any dialogue turn, the input x to the encoder is the entire dialogue history (separated using unique speaker-identifying tokens), with the model's own persona prepended. Conditioned on this input sequence x , the decoder generates a response y . Except when stated otherwise, all our models decode using beam search with beam size 20. We initialized the word embedding matrix with 300-dimensional GloVe embeddings (Pennington et al., 2014). Using the ParlAI framework (Miller et al., 2017), we pretrained the model on a dataset of 2.5 million Twitter message-response pairs, 1 then fine-tuned it on PersonaChat. On the PersonaChat validation set, the baseline model has a perplexity of 26.83 and F1 of 17.02, which would have placed us 4th out of 26 models in the ConvAI2 competition (Dinan et al., 2019). We attempt to improve over this baseline using control. Suppose we have a sequence-to-sequence model which gives P ( y | x ) =  t P ( y t | x, y 1 , . . . , y t  1 ) , the conditional probability of a response y (the model's next utterance) given input x (the context, which in our case includes the model's own persona and the dialogue history). Contrary to most previous work, which controls at the sentence level , we wish to control attributes of the output y at the dialogue level  meaning that a single control setting is used for a whole dialogue. For example, to control question-asking, we provide a control setting at the beginning of each dialogue (e.g. 20% questions or 70% questions ) rather than providing a control setting for each utterance (e.g. is a question or isn't a question ). With this approach, the sequence-to-sequence model is able to choose what value the controlled attribute should take for any particular utterance, but we are able to choose the overall distribution. We find that this approach works well  for example, the sequence-to-sequence model is 1 The Twitter dataset is provided in ParlAI; details can be found here: https://parl.ai/docs/tasks.html generally good at detecting when to ask a question. In particular, this is easier than the alternative: developing a separate process to decide, for each utterance, whether to ask a question. In this section, we describe the two methods  which we call Conditional Training (CT) and Weighted Decoding (WD)  that we use to control attributes of the output y at the dialogue level. Conditional Training (Fan et al., 2018; Kikuchi et al., 2016; Peng et al., 2018) is a method to learn a sequence-to-sequence model P ( y | x, z ) , where z is a discrete control variable . If the control attribute is naturally continuous (for example in our work, repetitiveness, specificity and response-relatedness), we use z to represent bucketed ranges. For a binary attribute like question-asking, z represents an overall probability (as explained in Section 5). To train a CT model, we first automatically annotate every ( x, y ) pair in the training set with the attribute we wish to control (for example, whether y contains a question mark). During training, for each example we determine the corresponding z value (for continuous attributes, this simply means sorting into the correct bucket; for question-asking, see Section 6.4). Next, the control variable z is represented via an embedding (each of the possible values of z has its own em-bedding). For all our experiments, the embedding is of length 10; this was determined via hyperpa-rameter tuning. There are several possible ways to condition the sequence-to-sequence model on z  for example, append z to the end of the input sequence, or use z as the START symbol for the decoder. We find it most effective to concatenate z to the decoder's input on every step. 2 Lastly, the CT model learns to produce y = y 1 , . . . , y T by optimizing the cross-entropy loss: loss CT =  1 TT (cid:88) t =1 log P ( y t | x, z, y 1 , . . . , y t  1 ) Our CT models are initialized with the parameters from the baseline sequence-to-sequence model P ( y | x ) (the new decoder parameters are initialized with small random values), then fine-tuned to optimize loss CT on the PersonaChat training set, until convergence of loss CT on the validation set. 2 To build a CT model P ( y | x, z 1 , . . . , z n ) conditioned on multiple controls { z 1 , . . . , z n } , we can simply concatenate multiple control embeddings to the decoder inputs. Weighted Decoding (Ghazvininejad et al., 2017) is a decoding method that increases or decreases the probability of words with certain features. The technique is applied only at test time, requiring no change to the training method. A limitation of WD is that the controllable attribute must be defined at the word-level; any desired utterance-level attribute must be redefined via word-level features. In weighted decoding, on the t th step of decoding, a partial hypothesis y <t = y 1 , . . . , y t  1 is expanded by computing the score for each possible next word w in the vocabulary: score ( w, y <t ; x ) = score ( y <t ; x ) + log PRNN ( w | y <t , x ) + (cid:88) i w i  f i ( w ; y <t , x ) . Here, log PRNN ( w | y <t , x ) is the log-probability of the word w calculated by the RNN, score ( y <t ; x ) is the accumulated score of the already-generated words in the hypothesis y <t , and f i ( w ; y <t , x ) are decoding features with associated weights w i . There can be multiple features f i (to control multiple attributes), and the weights w i are hyperpa-rameters to be chosen. A decoding feature f i ( w ; y <t , x ) assigns a real value to the word w , in the context of the text generated so far y <t and the context x . The feature can be continuous (e.g. the unigram probability of w ), discrete (e.g. the length of w in characters), or binary (e.g. whether w starts with the same letter as the last word in y <t ). A positive weight w i increases the probability of words w that score highly with respect to f i ; a negative weight decreases their probability. Note that weighted decoding and conditional training can be applied simultaneously (i.e. train a CT model then apply WD at test time)  a strategy we use in our experiments. In this section, we describe how we use conditional training and weighted decoding to control four attributes: repetition, specificity, response-relatedness and question-asking. We evaluate the effectiveness of both control methods via automatic metrics (i.e., measuring how well the attribute was controlled), and use our findings to select control methods and control settings to be explored further via human evaluation (Section 8). Our baseline model exhibits three types of repetition, which we call external repetition (self-repetition across utterances), internal repetition (self-repetition within utterances), and partner repetition (repeating the conversational partner). To control repetition with weighted decoding, 3 we define five n -gram based decoding features (see Appendix D). Three of these features ( extrep bigram , intrep bigram and partnerrep bigram ) identify repeating bigrams for the three repetition types. The other two features ( extrep unigram and intrep unigram ) identify repeating content words. By applying a negative weight to these features, we can reduce repetition. In particular, if the weight is  , our method is equivalent to n-gram blocking as described by Kulikov et al. (2018). We observe that repetition control is very important, thus all further control experiments include repetition control. Like many sequence-to-sequence models using beam search decoding, our baseline frequently asks generic questions such as What music do you like? and gives dull, unspecific responses, such as I like all kinds of music . We control specificity using Normalized Inverse Document Frequency (NIDF) as a measure of word rareness. 4 The Inverse Document Frequency of a word w is IDF ( w ) = log( R/c w ) where R is the number of responses in the dataset, and c w is the number of those responses that contain w . Normalized IDF (which ranges from 0 to 1) is NIDF ( w ) = IDF ( w )  min idf max idf  min idf (1) where min idf and max idf are the minimum and maximum IDFs, taken over all words in the vocabulary. To control specificity with weighted decoding, we use NIDF as a decoding feature. As shown in Table 1, this method produces reasonable outputs when the feature weight is within a certain range, but at the extremes the outputs are 3 We also tried controlling repetition with conditional training, defining z as the (bucketed) maximum ROUGE-L precision between the response y and the bot's previous utterances. However, this method was unsuccessful because there are not enough repetitive examples in the training data for the model to learn the control. Experimenting with data augmentation to solve this problem is an area for future work. 4 Note that our NIDF specificity features are similar to the NIRF and NIWF features used by Zhang et al. (2018a). nonsensical. The boundary for nonsensical output differs from example to example. To control specificity with conditional training, we define the specificity of an utterance y to be the mean NIDF of the words in y . Thus our control variable z is mean NIDF (discretized into 10 equal-sized buckets). As shown in Table 1, this method gives outputs with a narrower NIDF range, but overall produces less nonsensical outputs. In conversation, it's generally desirable to produce a response that is related to the partner's last utterance; for example if the partner says My grandfather died last month , it is appropriate to say I'm so sorry. Were you close to your grandfather? However, our baseline model frequently responds with unrelated utterances like Do you have any pets? To control response-relatedness with weighted decoding, we use the decoding feature resp rel : resp rel ( w ; y <t , x ) = cos sim ( word emb ( w ) , sent emb ( (cid:96) )) where word emb ( w ) is the GloVe embedding for the word w , sent emb ( (cid:96) ) is the sentence embedding for the partner's last utterance (cid:96) (note (cid:96) is part of the context x ), and cos sim is the cosine similarity between the two. In particular, the sentence embedding sent emb ( s ) for an utterance s is a Input: Do you go get coffee often Baseline Response: I do, when I am not playing the piano. weighted average of the GloVe embeddings of the words in s , with the first principal component projected out; for full details, see Arora et al. (2017). This method of controlling response-relatedness is similar to that described in (Baheti et al., 2018). We find that weighted decoding is effective to control the semantic relatedness of the model's response to the partner's last utterance (see Table 2). As before, we find that extreme weights lead to nonsensical output. To control response-relatedness with conditional training, we try defining the control variable z to be cos sim ( sent emb ( y ) , sent emb ( (cid:96) )) , the overall cosine similarity between the partner's last utterance (cid:96) and the model's response y (again, we discretize z ). However, we find this method ineffective  the CT model learns only a very weak connection between z and the semantic relatedness of the output (see Section 7 for more details). Considerate chitchat requires a reciprocal asking and answering of questions  asking too few or too many can appear self-centered or nosy. We control question-asking in order to study these trade-offs. To control question-asking with weighted decoding, we use the binary decoding feature is qn word ( w ) , which is equal to 1 if and only if the word w is in a pre-defined list of interrogative words ( how, what, when, where, which, who, whom, whose, why, ? ). We find this is a somewhat effective method to encourage or discourage questions, but with unintended side-effects: a negative weight can discourage valid non-question utterances that happen to contain interrogative words (such as I'm learning how to knit ) and a positive weight can result in degenerate utterances (such as 0 1 2 3 4 5 6 7 8 9 10 10 (boost) Question-Asking Control Level (CT) 0% 20% 40% 60% 80% 100% % U tt e r a n c e s c o n t a i n i n g ' ? ' Question-controlled CT Question-controlled CT w/ rep ctrl Target for question-controlled CT Beam search baseline Repetition-controlled baseline Gold data Figure 2: Controlling question-asking via conditional training. For conditional training, we regard an utterance y as containing a question if and only if y contains a question mark. We train our CT model on a control variable z with 11 possible values: { 0 , . . . , 10 } . As discussed in Section 5, we wish to control question-asking at the distributional, dialogue level, rather than at the binary, utterance level. Thus the setting z = i means that the model should produce, on average, utterances containing ?' with probability i/ 10 . During training we randomly assign examples to buckets such that each bucket i is trained on examples with the correct proportion of questions ( i/ 10 ), and all buckets have the same amount of training examples. We find that conditional training is effective to control question-asking  as shown in Figure 2, by increasing z from 0 to 10, we obtain a range of question-asking rates from 1.40% to 97.72%. However, when we introduce repetition control, question-asking is reduced  in particular, the z = 10 setting (which should produce 100% questions) now only produces 79.67% questions. The primary problem is the weighted decoding feature extrep bigram , which discourages bigrams that have appeared in previous utterances  this prevents the model from producing bigrams that commonly occur in many questions, such as do you and what is . To fix this, we introduce an extra setting z = 10 (boost) , in which we do not use the feature extrep bigram for weighted decoding during beam search, but we do use it to rerank the candidates after beam search. This setting, which allows the model to produce necessary question-asking bigrams, yields a 99.54% question-asking rate, at the cost of slightly increased external bigram repetition (see Appendix F). For controlling question-asking, conditional training is preferable to weighted decoding for two reasons. Firstly, it allows us to achieve (close to) 0% questions, 100% questions, or anything in between, without introducing the risk of degenerate output. Secondly, presence-of-a-question-mark captures the true attribute of interest (question-asking) more exactly and directly than presence of interrogative words. For these reasons, only the CT method is considered in the human evaluation. The previous section shows that conditional training and weighted decoding are both useful techniques, with different strengths and weaknesses. The primary disadvantage of conditional training is that it sometimes fails to learn the connection between the control variable z and the target output y . In practice, we find the model can learn simple attributes of the output (such as the presence of ?', and overall genericness), but not relationships between the input and output (such as semantic relatedness). By contrast, weighted decoding can force the desired feature to appear in the output by raising the weight arbitrarily high (though this may have unintended side-effects). The primary disadvantage of weighted decoding is that it risks going off-distribution when the weight is too strong. By contrast, conditional training produces mostly well-formed, in-distribution outputs. This highlights the importance of learned control  it is safer to learn to produce output that both satisfies the control variable and is appropriate, than to alter the decoding process to satisfy the control variable, potentially trading off appropriateness in the process. Other considerations include: (1) Convenience: conditional training requires retraining; weighted decoding doesn't, but is slower at test time. (2) Data availability: conditional training requires training examples of the controllable attribute, whereas weighted decoding can control any computable feature without requiring examples. (3) Attribute definition: conditional training can control sentence-level attributes, but they must be discrete. By contrast, weighted decoding requires word-level features, but they can be continuous. In order to study the effect of our controllable attributes, we conduct a large-scale human evaluation evaluation of 28 model configurations (see Appendix E), plus human-human conversations for comparison. Approach In our evaluation, a crowdworker chats with a model (or in the human-human case, another crowdworker) for six conversational turns, then answers eight multiple-choice questions which each capture different aspects of conversational quality: avoiding repetition, interestingness, making sense, fluency, listening, inquisitiveness, humanness and engagingness. The eight questions are Likert questions on a 1-4 scale, where higher is better. 5 To match the ConvAI2 Challenge, we also add a persona retrieval question, in which the crowdworker is asked to select which of two possible personas was the model's persona. For full details of the evaluation design, see Appendix B. Our evaluation is the same as the ConvAI2 Challenge evaluation, but more detailed  ConvAI2 includes only engagingness and persona retrieval. 6 As in the ConvAI2 challenge, each of our 28 model configurations was evaluated by over 100 crowdworkers, and the results were adjusted for annotator variance via a Bayesian calibration (Kulikov et al., 2018). In designing our evaluation, we aimed to capture the four aspects we expected to directly improve via control (avoiding repetition, interestingness, listening, inquisitiveness), two important error classes we thought would be affected by our controls (fluency, making sense), and two overall quality measures (engagingness, humanness). In this section we summarize the main findings of our human evaluation  whose full results can be found in Appendices G and H, with sample conversations in Appendix C. As Figure 3 shows, controlling for repetition, specificity and question-asking all lead to large 5 Exceptions: Avoiding repetition is a 1-3 scale, as we found this gave clearer instructions. Inquisitiveness has an optimal score of 3; 1 and 2 represent too little question-asking, and 4 represents too much. 6 There are three other minor differences between our evaluation and ConvAI2's: (1) We fix capitalization and spacing before showing the chatbot's utterances to crowdworkers, while ConvAI2 show the raw lowercase tokenized form. We found the latter interferes with fluency evaluation. (2) We conduct 6 dialogue turns, while ConvAI2 conducts 4-6. This was necessary to evaluate repetitiveness. (3) We use (publicly-available) validation set personas, while ConvAI2 uses (hidden) test set personas. This enables us to release our evaluation chatlogs. engagingness improvements over the greedy and beam-search baseline models. In particular, we find that controlling for multi-turn (self) repetition is important and should be incorporated alongside other attribute control methods. We found no improvement by controlling response-relatedness. To better understand these overall engagingness improvements, we consider the full set of human judgments, shown in Figure 4. We find that reducing repetition leads to improvements across all our aspects of conversational quality. Increasing specificity shows improvements in interestingness and listening ability over the repetition-controlled baseline, while increasing question-asking shows improvements in inquisitiveness and interestingness over the repetition-controlled baseline. Our most engaging model, which controls both repetition and question-asking  marked Ques-tion (CT)' in Figure 3 (left)  matches the engagingness of the winning entry in the ConvAI2 competition, as both models achieve a raw score 7 of 3 . 1 (Dinan et al., 2019). However, the ConvAI2 winner, Lost in Conversation, was trained on approximately 12  as much data as our model. Lost in Conversation is based on the OpenAI GPT Language Model (Radford et al., 2018), which is pretrained on the BookCorpus (Zhu et al., 2015), which contains approximately 985 million words, whereas our model is pretrained on the Twitter dataset (approximately 79 million words). Repetition (WD) We observe that self-repetition across utterances ( external repetition ) is by far the most severe form of repetition in our beam search baseline model. We evaluate several settings of the extrep bigram weighted decoding feature, and find that an aggressive repetition-reduction setting (reducing bigram repetition rate to below gold data levels) is rated best. We also find that blocking repeated content words improves the avoiding repetition score. See Appendices E, F and G for full details. As shown in Figure 3 (left) and Figure 4, our repetition-controlled model improves hugely 7 Although the same Bayesian calibration method was applied both in our study and in the ConvAI2 competition, calibrated scores are not comparable across the two; thus we compare raw scores (viewable in Table 7). over the beam search baseline in all metrics, and achieves close-to-human scores on all metrics except humanness. This striking result demonstrates that repetition is by far the biggest limiting quality factor for naive sequence-to-sequence dialogue agents. The result also emphasizes the importance of multi-turn dialogue evaluation to detect the problem. We refer to this model as the repetition-controlled baseline , and use it as a basis for all remaining experiments (i.e., we control specificity, response-relatedness and question-asking on top of these repetition-control settings). Specificity (WD, CT) For our weighted decoding models, the extreme settings (very generic and very specific) score poorly in engagingness due to the frequent presence of degenerate output  see Figure 3 (middle). We find that the weight = 4 setting (which is more specific than the repetition-controlled baseline and about as specific as the gold data) maximizes engagingness. As shown in Figure 3 (left) and Figure 4, this more-specific model is rated more interesting, engaging, and a better listener than the repetition-controlled baseline, but at the cost of reduced fluency and making sense. Our CT model with z = 7 (which has a similar NIDF level as WD with weight = 4 ) shows similar results, but the improvements are smaller. For further discussion on the interestingness of our specificity models, see Section 8.3. Response-relatedness (WD) We evaluated several control settings (weight =  10 , 5 , 10 , 13 ) and found that none scored better than weight = 0 (no response-relatedness control); see Appendix H. This is surprising  prior to running the human evaluation, we annotated 100 examples ourselves to determine the best control settings. While we identified a more responsive setting (weight = 5 ) as less likely than the uncontrolled model to ignore the user, crowdworkers rated it as a slightly worse listener than the uncontrolled model. One explanation for this discrepancy is that the more responsive model takes more risks, using more rare words (0.197 NIDF, up from 0.178), and thus receives a lower makes-sense score (3.41, down from 3.70). We hypothesize that, compared to us, the crowdworkers are less tolerant of slightly nonsensical output, and more tolerant of generic unrelated utterances. Question-asking (CT) As shown in Figure 3 (right), a question-asking rate of 65.7% ( z = 7 ) maximizes engagingness. This setting, which asks more questions than both the repetition-controlled baseline (50.0%) and the human-produced gold data (28.8%), brings us closest to human-level engagingness  see Figure 3 (left). Although we find that a rate of approximately 65.7% question-asking is the most engaging, a lower level (48.9%, or z = 4 ) is rated the best listener. Lastly, we find Model Win% Top 3 reasons for preferring model Specificity WD (weight = 6 ) 84.1% More information; Better flow; More descriptive Specificity WD (weight = 4 ) 75.5% More information; They describe their life in more detail; Funny Specificity CT ( z = 7 ) 56.2% More information; Better flow; Seems more interested Table 3: A/B tests comparing various specificity-controlled models to the repetition-controlled baseline on interestingness. that although asking too many questions is less engaging, most crowdworkers will not directly criticize a chatbot that asks questions on every turn  only 11.9% of crowdworkers judged the z = 10 (boost) setting, which asks 99.5% questions, as asking too many questions. 8 For full details of these scores, see Appendix F and H. For time and budget reasons, we did not evaluate any models controlling both question-asking and specificity. However, we expect it is possible to obtain further improvements by doing so. Though our more-specific models yielded significant improvements in engagingness, we were surprised that they did not yield clearer improvements in interestingness. To investigate further, we conducted an A/B interestingness evaluation of three specificity-controlled models, compared to the repetition-controlled baseline. Crowdworkers were shown two conversations (from the main human evaluation) and asked to choose which model was more interesting (see Figure 7 for details). We collected 500 samples per comparison, plus 200 additional human vs repetition-controlled baseline samples, which were used to filter for quality control. After discarding low-quality crowdworkers, we have roughly 300 evaluations per comparison, with an average Cohen's  = 0 . 6 . As shown in Table 3, all three models were rated significantly more interesting than the repetition-controlled baseline. This convincingly shows that producing utterances with more rare words is a valid strategy to improve interestingness. We have two explanations for why these interestingness differences did not materialize in our main evaluation. Firstly, interestingness is a particularly subjective metric (unlike more tangible metrics such as avoiding repetition and making sense)  this makes it hard to calibrate across crowdworkers. 8 Though this conclusion may hold true for the PersonaChat task  a synthetic chatting task that instructs participants to get to know each other  in real-life social conversations, incessant question-asking may be less tolerated. Secondly, we suspect that in our original evaluation, the crowdworkers may have evaluated the interestingness of the task rather than the chatbot . This could account for why subtle increases in conversational ability did not result in higher interestingness ratings  the PersonaChat task itself has a natural interestingness limit. What makes a good conversation? Through our evaluation, we showed that a good conversation is about balance  controlling for the right level of repetition, specificity and question-asking is important for overall quality. We also found that conversational aspects such as interestingness, listening, and inquisitiveness are all important  though optimizing these can introduce a trade-off against certain types of errors (such as repetitive, disfluent, or nonsensical output). Secondly, multiturn evaluation is essential to study what makes a good conversation  multiple turns are required to reveal issues such as repetition, consistency, and question-asking frequency. Lastly, what do we mean by good'? Although humanness and engagingness are both commonly used as overall quality metrics, the two are very different. While our models achieved close-to-human scores on engagingness, they failed to get close on humanness  showing that a chatbot need not be human-like to be enjoyable. This striking result also demonstrates the importance of measuring more than one quality metric when evaluating dialogue agents. Outlook Our work shows that neural generative systems have systemic problems when applied to open-ended dialogue, some of which (e.g. repetition) are only observable in the multi-turn setting. Furthermore, control of low-level attributes offers a practical way to correct these problems, yielding large improvements to overall quality  in our case, comparable to systems trained on much more data. Future work includes optimizing control settings automatically, and building more convincingly human-like chatbots.
abstain abstain method method result abstain abstain abstain abstain abstain abstain objective abstain result method result abstain abstain abstain result result other other other method other other other other other other method other method objective objective method objective abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain method method method method other abstain method abstain method abstain method method abstain method abstain method method method abstain abstain abstain other abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain method abstain method abstain abstain abstain method abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain method method other abstain method method abstain method abstain method abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain result abstain result result abstain abstain abstain abstain abstain result result result result abstain result abstain abstain abstain result result result abstain result abstain abstain abstain abstain result result abstain result result result abstain abstain abstain abstain result result result abstain result result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain result abstain result method abstain