To quantify how well natural language understanding models can capture consistency in a general conversation, we introduce the DialoguE COntradiction DEtection task (DE-CODE) and a new conversational dataset containing both human-human and human-bot contradictory dialogues. We show that: (i) our newly collected dataset is notably more effective at providing supervision for the dialogue contradiction detection task than existing NLI data including those aimed to cover the dialogue domain; (ii) Transformer models that explicitly hinge on utterance structures for dialogue contradiction detection are more robust and generalize well on both analysis and out-of-distribution dialogues than standard (un-structured) Transformers. We also show that our best contradiction detection model correlates well with human judgments and further provide evidence for its usage in both automatically evaluating and improving the consistency of state-of-the-art generative chatbots. Recent progress on neural approaches to natural language processing (Devlin et al., 2019; Brown et al., 2020), and the availability of large amounts of conversational data (Lowe et al., 2015; Smith et al., 2020) have triggered a resurgent interest on building intelligent open-domain chatbots. Newly developed end-to-end neural bots (Zhang et al., 2020; Adiwardana et al., 2020; Roller et al., 2020) are claimed to be superior to their predecessors (Worsnick, 2018; Zhou et al., 2020) using various human evaluation techniques (See et al., 2019; Li et al., 2019; Adiwardana et al., 2020) that aim to give a more accurate measure of what makes a good conversation. While the success is indisputable, there is still a long way to go before we  * Dolphins are mammals, not fish. arrive at human-like open-domain chatbots. For example, it has been shown that open-domain chatbots frequently generate annoying errors (Adiwar-dana et al., 2020; Roller et al., 2020) and a notorious one among these is the class of contradiction, or consistency errors. When interacting with chatbots, people carry over many of the same expectations as when interacting with humans (Nass and Moon, 2000). Self-contradictions by these bots (see Fig.1, bottom) are often jarring, immediately disrupt the conversational flow, and help support arguments about whether generative models could ever really understand what they are saying at all (Marcus, 2018). From a listener's perspective, such inconsistent bots fail to gain user trust and their long-term communication confidence. From a speaker's perspective, it violates the maxim of quality in Grice's cooperative principles (Grice, 1975) Do not say what you believe to be false. Hence, efforts on reducing contradicting or inconsistent conversations by open-domain chatbots are imperative. Prior works (Welleck et al., 2019) characterized the modeling of persona-related consistency as a natural language inference (NLI) problem (Dagan et al., 2005; Bowman et al., 2015), and constructed a dialog NLI dataset based on Persona-Chat (Zhang et al., 2018), but so far state-of-the-art chatbots (Roller et al., 2020) have not been able to make use of NLI techniques in improving dialogue consistency. Overall, the challenge remains that we are still unable to answer the simple yet important question how good are we at modeling consistency (including persona, logic, causality, etc.) in a general conversation? . The inability to measure this obscures to what degree building new modules or techniques can in turn help prevent contradicting responses during generation. Seeking to answer this question, we introduce the DialoguE COntradiction DEtection task (DE-CODE) 1 and collect a new conversational dataset containing human written dialogues where one of the speakers deliberately contradicts what they have previously said at a certain point during the conversation. We also collect an out-of-distribution (OOD) set of dialogues in human-bot interactive settings which contain human-labeled self-contradictions made by different chatbots. We then compare a set of state-of-the-art systems, including a standard unstructured approach and a proposed structured approach for utilizing NLI models to detect contradictions. In the unstructured approach, a Transformer NLI model directly takes in the concatenation of all utterances of the input dialogue for prediction, following the paradigm of NLU modeling. In the structured approach, utterances are paired separately before being fed into Transformer NLI models, explicitly taking account of the natural dialogue structure. dataset is notably more effective at providing supervision for the contradiction detection task than existing NLI data including those aimed at covering the dialogue domain; (2) the structured utterance-based approach for dialogue consistency modeling is more robust in our analysis and more transferable to OOD human-bot conversation than the unstructured approach. This finding challenges the mainstream unstructured approach of simply applying pre-trained Transformer models and expecting them to learn the structure, especially for OOD scenarios which are often the case when incorporating NLU modules into NLG systems, since intermediate in-domain data are scarce. Finally, with such improvements on the contradiction detection task, we show that our best resulting detector correlates well with human judgments and can be suitable for use as an automatic metric for checking dialogue consistency. We further provide evidence for its usage in improving the consistency of state-of-the-art generative chatbots. Several prior works on improving dialogue consistency have explored using direct modeling of the dialogue context in generation algorithms. The modeling can be implicit where the dialogue consistency-related information like style (Wang et al., 2017), topics, or personal facts are maintained in distributed embeddings (Li et al., 2016; Zhang et al., 2019a), neural long-term memories (Bang et al., 2015), hierarchical neural architecture (Serban et al., 2016), latent variables (Ser-ban et al., 2017), topical attention (Dziri et al., 2019a), or even self-learned feature vectors (Zhang et al., 2019b). Some works have grounded generation models on explicit user input (Qian et al., 2018), or designated personas (Zhang et al., 2018). Although, improvements on automatic generation metrics were often shown on guided response generation based on the consistency modeling, the issue of contradiction has never been resolved, nor have generally applicable methods to gauge the consistency improvements been developed. Further, simply scaling models has not made the problem go away, as is evident in the largest chatbots trained such as BlenderBot with up to 9.4B parameter Transformers (Roller et al., 2020). More similar to our work is utilizing NLI models in dialogue consistency. Dziri et al. (2019b) attempted to use entailment models trained on synthetic datasets for dialogue topic coherence evaluation. Particularly, Welleck et al. (2019) constructed the dialogue NLI dataset and (Li et al., 2020) utilized it to try to reduce inconsistency in generative models via unlikelihood training in a preliminary study that reports perplexity results, but did not measure actual generations or contradiction rates. We note that the dialogue NLI dataset is only semi-automatically generated, with limited coverage of only Persona-chat data (Zhang et al., 2018), whereas our DECODE is human-written and across multiple domains. Our task also involves logical and context-related reasoning beyond personal facts. We show that transfer of DECODE is subsequently more robust than dialogue NLI on both human-human and human-bot chats. We formalize dialogue contradiction detection as a supervised classification task. The input of the task is a list of utterances x = { u 0 , u 1 , u 2 , ..., u n } representing a dialogue or a dialogue snippet. The output is y , indicating whether the last utterance u n contradicts any previously conversed information contained in the dialogue { u 0 , u 1 , ..., u n  1 } , where y can be 0 or 1 corresponding to the noncontradiction and the contradiction label respectively. Preferably, the output should also include a set of indices I  { 0 , 1 , ..., n  1 } representing a subset of { u 0 , u 1 , ..., u n  1 } which contain information that is actually contradicted by the last utterance u n . The extra indices I output require models to pinpoint the evidence for the contradiction, providing an extra layer of explainability. Our goal is first to collect training and evaluation data for this task. We thus collect dialogues in which the last utterance contradicts some previous utterances in the dialogue history. To obtain such dialogues, we give annotators dialogue snippets from pre-selected dialogue corpora, and then ask them to continue the conversation by writing one or two utterances such that the last utterance by the last speaker contradicts the dialogue history. We also ask annotators to mark all the utterances in the dialogue history that are involved in the contradiction as supporting evidence. We ask annotators to write contradicting utterances based partly on existing dialogues rather than collecting new dialogue from scratch because the provided dialogues can often convey semantic-rich contexts from different domains and inspire annotators to write more diverse examples. We don't impose constraints on the annotation such that the annotator could have the flexibility to write more diverse contradictory responses that might not belong to pre-defined types (knowledge, emotion, persona, etc). Also note that we ask the annotator to write contradictory dialogues based on pre-selected human-human dialogue rather than collecting dialogues from human-bot interaction for the main dataset because we want the examples to be general and less bound to specific bots. 2 We crowdsource the continuation and annotation data with Amazon Mechanical Turk via ParlAI (Miller et al., 2017). To ensure data quality, we apply three techniques: (i) an onboarding test every annotator has to pass to contribute examples; (ii) each annotator can only create up to 20 examples; and (iii) all examples in the validation and test set are verified by asking 3 additional workers. More details about annotation are provided in Appendix. We collected 17,713 human-written contradicting dialogues in which 4,121 are verified by 3 annotators. The pre-selected dialogue source corpora are Wizard of Wikipedia (Dinan et al., 2019), EMPATHETICDIALOGUES (Rashkin et al., 2019), Blended Skill Talk (Smith et al., 2020), and ConvAI2 (Dinan et al., 2020), covering various conversational topics. To facilitate the evaluation of consistency modeling on the dialogue contradiction detection classification task, we sample an equal number of non-contradicting dialogues according to the same dialogue length distribution as the contradicting ones from the same dialogue corpus. Then, we make the splits such that the train split contains unverified examples, and dev and test splits only contain verified examples. Each split has balanced labels between contradiction and non-contradiction. The breakdown of each of the dataset sources is shown in Appendix. Auxiliary (Checklist) Test Sets. We further create two auxiliary checklist evaluation sets by transforming the contradiction examples in the original test in two ways such that the ground truth label is 2 Alongside the main dataset, another portion of the examples are collected via human-bot interaction and used as out-of-domain evaluation. either invariant or expected to flip. The two resultant sets serve as diagnostic tests on the behavior, generalization and transferability of our models. The transformations are described below:  Add Two Turns (A2T) We insert a pair of randomly sampled utterances into the dialogue such that the inserted utterances are between the two original contradicting utterances. This gives a new contradicting dialogue with a longer dialogue history. Remove Contradicting Turns (RCT) We remove all the turns (all pairs of utterances) 3 marked as supporting evidence for the contradiction in the dialogue except the last utterance. This results in a new non-contradiction dialogue. Human-Bot Test Set. Our main dataset involves human-written dialogues containing contradicting utterances based on human-human dialogues from existing corpora. In practice, to evaluate the response quality of a machine rather than a human in terms of its consistent responses, we care about how well a contradiction detector can perform in human-bot interactive conversations. To that end, we further collect human-bot dialogue data by employing crowdworkers to interact with a diverse set of open-domain bots. These include Poly-encoder (Humeau et al., 2019) based retrieval models, generative models (Roller et al., 2020), unlikelihood trained models (Li et al., 2020), retrieve-and-refine models (We-ston et al., 2018; Roller et al., 2020), models either pre-trained on a previously existing Reddit dataset 3 The dataset dialogues involve two speakers taking turns speaking. To maintain this structure, for each marked utterance, we remove a pair of utterances that represents a turn. This also helps remove information involved in the contradiction such that the new label should be non-contradiction. extracted and obtained by a third party that was hosted by pushshift.io (Baumgartner et al., 2020) or fine-tuned on the Blended Skill Talk (BST) dialogue tasks (Smith et al., 2020)  that is, all the dialogue models that are compared in the study in Roller et al. (2020). During the collection, if the bot generates an utterance that contradicts itself, we ask the worker to mark the utterance. In some of the dialogues, workers are explicitly instructed to goad the bots into making contradicting utterances. The final human-bot test set we derive contains 764 dialogues, half of which end with a contradicting utterance by the bot. All the dialogues in the set, with either contradiction or non-contradiction labels, are verified by 3 additional annotators, beside the human who actually talked to the bot. The auxiliary and human-bot test sets aim to test models' robustness and generalizability beyond the collected human-written test set (Ribeiro et al., 2020; Gardner et al., 2020), and give a more comprehensive analysis of the task. Table 1 summarizes the final overall dataset. Examples are provided for each dataset type in Fig. 1 and Appendix Table 5. To model the dialogue consistency task, we first employ some of the techniques used in NLI sequence-to-label modeling, where the input is a pair of textual sequences and the output is a label. The benefit of such modeling is that we can directly make use of existing NLI datasets during training. However, unlike previous work (Welleck et al., 2019) that directly utilized NLI models giving a 3-way output among entailment, contradiction, and neu-tral, we modify the model with a 2-way output between contradiction and non-contradiction (either entailment or neutral) labels, as our task is centered around the detection of inconsistency. More formally, we denote the model as  y pred = f  ( C , u ) , where  y pred is the prediction of the label y , i.e. whether the textual response u contradicts some textual context C = { u 0 , u 1 , ..., u n  1 } , and  are the parameters of the model. Unstructured Approach. In this approach, we simply concatenate all the previous utterances in the dialogue history to form a single textual context. Then, we apply f  to the context and the last utterance to infer the probability of contradiction:  y pred = f  ([ u 0 , u 1 , u 2 , ..., u n  1 ] , u n ) (1) When concatenating the utterances, we insert special tokens before each utterance to indicate the speaker of that utterance. This is aimed to provide a signal of the dialogue structure to the models. Still, this approach assumes that the model can use these features adequately to learn the underlying structure of the dialogue implicitly during training. Structured Utterance-based Approach. Since the reasoning crucially depends on the last utterance, in this method we first choose all the utterances by the last speaker to form a set S . We then pair every utterance in the set with the last utterance and feed them one by one into f UB . The final contradiction probability is the maximum over all the outputs:  y pred = max (cid:8) f UB ( u i , u n ) : u i  S (cid:9) (2) Additionally, the utterance-based approach is able to give a set of utterances as supporting evidence for a contradiction decision by choosing the pairs having contradiction probability higher than a threshold  e : I = (cid:8) i : f UB ( u i , u n ) >  e (cid:9) (3) This not only gives explanations for its prediction but can also help diagnose the model itself, e.g. we can measure metrics of the model's ability to provide these explanations by comparing them against gold supporting evidence annotations. One downside of this modeling approach is that it will not be able to capture reasoning between speakers. A case for that would be a pronoun by one speaker might refer to something initiated by the other speaker. Nevertheless, the utterance-based approach explicitly adds an inductive structure bias to learning and inference which we will see can aid its generalization capability. Thresholding. For both the unstructured and utterance-based approaches, the detection of contradiction is made by comparing  y pred with a threshold  and by default  is 0.5. We study four base pre-trained models variants for f  : BERT (Devlin et al., 2019), Electra (Clark et al., 2020), RoBERTa (Liu et al., 2019), and Model Training Data MT MT (Strict) HB SE BART (Lewis et al., 2020). They represent the start-of-the-art language representation models and have yielded successes in many NLU tasks. The input format of f  follows how these models handle sequence-pairs ( C and u ) for classification tasks with padding, separator and other special tokens such as position embeddings and segment features inserted at designated locations accordingly. We fine-tune f  on different combinations of NLI training data including SNLI (Bowman et al., 2015), MNLI (Williams et al., 2018), ANLI-R3 (Nie et al., 2020a) 4 , DNLI (Welleck et al., 2019), as well as our DECODE Main training set. We convert the 3-way labels of the examples in existing NLI datasets to 2-way, as described before, and  is optimized using cross-entropy loss. When training f UB in the utterance-based approach using the DECODE training set, the input sequences 4 ANLI data is collected in three rounds resulting in three subsets (R1, R2, R3). We only used training data in R3 since it contains some dialogue-related examples. are sampled utterance pairs from the DECODE dialogue. In other scenarios, f  or f UB are trained with data treated as in normal NLI training. The models are evaluated on the test sets described in subsection 3.3. For the utterance-based approach, which provides supporting evidence utterances (Equation 3), we report F1 on evidence retrieval. We also report a stricter score which evaluates whether both 2-way contradiction detection and supporting evidence retrieval exactly match with the ground truth on DECODE Main test. Our main results comparing various detectors on DECODE are shown in Table 2. We now describe our key observations. DECODE is notably more effective than other existing NLI data in providing supervision for contradiction detection in dialogue. We found that models trained on DECODE achieve higher accuracy than that of those trained on DNLI or ANLI-R3, on all evaluation sets, with large improvements, e.g. a 12-point jump from the same model training on ANLI-R3 and a 16-point jump from training on DNLI using utterance-based RoBERTa on the DECODE Main test set. Moreover, while training on All datasets (SNLI, MNLI, ANLI-R3, DNLI & DECODE) is effective, the removal of DECODE from the training data induces a consequential downgrade on the performance. Training on NLI data which does not cover the dialogue domain, e.g., SNLI+MNLI is even worse, only achieving 77.4% on DECODE Main (Test) vs. 93.19% for DECODE and cannot even reach the majority baseline on the Main (Test-Strict). Further, training on DECODE is also more helpful than DNLI or ANLI-R3 for supporting evidence retrieval. These findings indicate that existing NLI data has limited transferability to the dialogue contradiction detection task despite their coverage of the dialogue domain in addition to other domains and that our DECODE data provides a valuable resource for modeling dialogue consistency and developing data-driven approaches for contradiction detection. Different pre-training models that perform similarly on the in-domain test set can have very different performance on OOD human-bot dialogue. The last four rows of the table show the results of utterance-based RoBERTa, BERT, Elec-Utterance-based Unstructured 0.0 20.0 40.0 60.0 80.0 100.0 A cc u r a cy 93.2 96.8 84.7 70.0 91.5 97.5 78.4 34.4 DECODE Main (Test) Human-Bot A2T RCT Figure 2: Comparison between utterance-based and unstructured approaches of RoBERTa pre-trained, DECODE fine-tuned models on DECODE Main (Test), Human-bot, and auxiliary test sets. tra, and BART trained on DECODE. We can see that RoBERTa, Electra, and BART got similar in-domain accuracy on DECODE, around 93%-94%. RoBERTa stands out when comparing their performance on the human-bot test set with the highest score of 84.69% across the column and with better performance on supporting evidence retrieval as well. We speculate that this is due to the fact that RoBERTa pre-training data has a broader coverage than Electra and BART. We hope future work on dialogue contradiction detection could explore pretraining models on more dialogue-focused corpora. The unstructured approach gets higher accuracy on the in-domain test set. A direct comparison between unstructured RoBERTa and utterance-based RoBERTa trained on DECODE reveals that the unstructured approach more often than not gets a higher accuracy than its corresponding utterance-based approach when other experimental setups are kept identical. Noticeably, unstructured RoBERTa trained on all NLI data got a 97.46% score, whereas utterance-based yielded 94.19%. This seemingly indicates that training an unstructured model is able to yield a good representation of the consistency of the dialogue. However, analysis on the human-bot and auxiliary test sets shows that such high accuracy is an over-amplification of the model's real understanding ability, as we discuss next. The structured utterance-based approach is more robust, and more transferable. Figure 2 gives a comparison between utterance-based and unstructured RoBERTa on each of the evaluation sets. We can see that the utterance-based model is able to maintain satisfactory performance across all the sets whereas the unstructured model under-performs at the human-bot and RCT auxiliary test sets with a 34.4% accuracy on RCT compared to 78.4% for utterance-based, in stark contrast to the high performance of the unstructured method on the in-domain DECODE Main test set. This result indicates the unstructured approach overfits on superficial patterns in the DECODE Main training data which are still present due to RCT's construction process. 5 We also provide further analysis in Appendix E, including experiments showing that simply removing speaker utterances not uttered by the last speaker does not greatly improve the unstructured method. The fact that the utterance-based approach has good transferability to the OOD human-bot test set indicates that injecting the correct inductive structure bias is beneficial for modeling dialogue consistency. We believe this is an interesting result generally for research using Transformers, where there is currently a belief amongst some practitioners that they can just use a standard Transformer and it will learn all the structure correctly on its own. In our setting that is not the case, and we provide a method that can rectify that failing. In general, there is still much room for improvement. The results in Table 2 also demonstrate that the modeling of dialogue consistency is a demanding task. On the contradiction detection task, the best score achieved by the state-of-the-art pre-trained language models on DECODE (Test-Strict) is 80.86% and the best human-bot test score is 84.69%. Considering all the examples in the test sets are verified by at least 3 annotators, humans are able to swiftly identify such contradictions. This suggests there is a large ability gap between our best automatic detectors and humans. Closing this gap is an important challenge for the community. Model vs. Human Judgment. To further understand the detector predictions and how well they might align with human judgments, we consider the Human-Bot data again. We first divide all the utterances into two categories based on whether they are generated by a human or a bot. Then, the bot-generated utterances that have been marked by annotators as contradicting utterances are cat-5 Overfitting on superficial patterns is a typical issue and open problem in NLU modeling (Nie et al., 2020a). Utterance-based (DECODE) Utterance-based (DNLI) Unstructured (DECODE) F i r e R a t e 5.5 17.9 14.0 22.9 29.4 21.7 44.3 44.8 31.7 46.3 48.9 39.7 74.3 65.1 50.1 Human Bot @1 @2 @3 Figure 3: The fire rate (the percentage that it predicts contradiction) of RoBERTa models with different setups on utterances belonging to different categories. egorized into three sets based on the number of annotators that agree on the contradiction label. By design, the more annotators that agree on the contradiction label, the more plausible that it is a contradiction. We examine detector model fire rate on the utterances in the 5 different categories and results are shown in Figure 3. The fire rate of utterance-based RoBERTa trained on DECODE on human utterances is 5.5% contrasting to the 74.3% on 3-agreed contradicting utterances, whereas the fire rates of unstructured RoBERTa on different categories are more clustered together. This finding demonstrates that our models can discriminate between utterances with a distinct nature, and the model predictions are aligned with human judgments. Moreover, a strong discriminative detector could be a useful tool to stratify utterances. Using DECODE as an Automatic Metric. The results presented above indicate that the prediction of the detector can easily differentiate between the quality of utterances by humans and the utterances by bots. We further investigate whether it can differentiate the quality of the utterances by different bots and be used as an automatic metric checking generation consistency. We compare the average contradiction score of the detector with the contradiction rate by human judgments on the utterances generated by different classes of model (bots). The bots are the same set of models described in subsection 3.3 from which we collected our human-bot 0.04 0.06 0.08 0.1 0.12 Human Identified Contradiction Rate 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 A v g . dialogue examples. The trend in Figure 4 reveals that the scores are positively correlated with human judgments, with a Pearson correlation coefficient of 0.81. We would expect that improvement on the DECODE task will directly increase the correlation between the automatically produced detection score and human judgments, where use of such an automatic metric can ease the burden on laborious human evaluation of consistency. Given a contradiction detector, an obvious question other than using it as an automatic metric, is: can it be used to improve the consistency of dialogue generation models? We consider a very simple way to do that in the state-of-the-art generative model, BlenderBot (BST 2.7B) (Roller et al., 2020). During the decoding phase, for decoding methods that can output multiple hypotheses, we simply rerank the top scoring hypotheses using the contradiction detection classifier. We use our best performing classifier, our utterance-based RoBERTa model with DECODE fine-tuning, and consider three methods of decoding: beam search, topk sampling (Fan et al., 2018) and sample-and-rank (Adiwardana et al., 2020), and compare the standard and DECODE-reranked decoding methods to each other. For beam search we use the best found parameters from (Roller et al., 2020) which are beam size 10, minimum beam length 20 and beam blocking of 3-grams. For topk we use k = 40 . For Sample-and-Rank we use k =40 and 20 samples. We consider the same human-bot dialogue logs as before, but only between Blenderbot BST 2.7B and humans, selecting only contradicting Model + DECODE Human Decoding Strategy Contradict% Contradict% Standard generation Beam Search 69.7% 84.2% Topk ( k = 40 ) 42.1% 69.7% Sample-and-Rank 39.5% 55.3% DECODE Re-ranking Beam Search 46.1% 55.3% Topk ( k = 40 ) 2.6% 39.5% Table 3: Generation Re-ranking using DECODE vs. standard methods, reporting the contradiction % as flagged by our contradiction detection classifier (i.e., an automatic metric, DECODE Contradict%) in addition to human judgments (Human Contradict%). utterances. Table 3 presents the results. Automatic metric using DECODE. Using our same DECODE contradiction classifier as the automatic metric, as in subsection 5.2, we observe that by re-ranking the beam of beam search (size 10) we can improve the metric. Still, 46.1% of the time the detector flags generations as contradictions (vs. 69.7% without re-ranking). Upon observation of the outputs, this seems to be due to the beam of beam decoding not being diverse enough (Vijayakumar et al., 2016): when the top scoring utterance is flagged as contradicting, many of the other utterances in the beam are similar responses with slight rephrases, and are flagged contradicting as well. Topk sampling fares much better, where reranking in our test can very often find at least one from the k = 40 samples that does not flag the classifier, leaving only a 2.6% contradiction firing rate. We note we expect these numbers are over-optimistically low because the metric itself is being used to search (re-rank) and evaluate in this case. Human Judgments. The last column of Table 3 presents human judgments of the various model generations, judged using the same approach as before with human verifiers, and reporting the percentage of contradictions. We observe similar results to the automatic metric findings. DECODE re-ranking reduces the number of contradictions, particularly for Topk re-ranking vs. Topk : testing for significance with a Wilcoxon signed-rank test, we get p = 0 . 051 using two human verifiers and p = 0 . 023 for three verifiers. More detailed results and analysis can be found in Appendix G. 6 Conclusion We introduce the DialoguE COntradiction DEtection task (DECODE) and a new conversational dataset containing both human-human and human-bot contradictory dialogues. Training models on DECODE achieves better performance than other existing NLI data by a large margin. We further propose a structured utterance-based approach where utterances are paired before being fed into Transformer NLI models to tackle the dialogue contradiction detection task. We show the superiority of such an approach when transferring to out-of-distribution dialogues compared to a standard unstructured approach representative of mainstream NLU modeling. We further show that our best contradiction detector correlates with human judgments, and provide evidence for its usage in both automatic checking and improving the consistency of state-of-the-art generative chatbots. We thank the reviewers, and Jie Lei and Hao Tan for their helpful discussions. YN interned at Facebook. YN and MB were later sponsored by NSF-CAREER Award 1846185, DARPA MCS Grant N66001-19-2-4031, and DARPA YFA17-D17AP00022.
objective result objective abstain result abstain other abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain objective method objective abstain abstain objective abstain result result other other other other other method other other method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain method abstain method abstain abstain abstain method method method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain method method abstain method abstain abstain abstain method method result abstain abstain result abstain abstain abstain abstain abstain other abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain other abstain abstain abstain result abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain result result other other other