Implicit knowledge, such as common sense, is key to fluid human conversations. Current neural response generation (RG) models are trained to generate responses directly, omitting unstated implicit knowledge. In this paper, we present Think-Before-Speaking (TBS), a generative approach to first externalize implicit commonsense knowledge ( think ) and use this knowledge to generate responses ( speak ). We expect that externalizing implicit knowledge allows more efficient learning, produces more informative responses, and enables more explainable models. We analyze different choices to collect knowledge-aligned dialogues, represent implicit knowledge, and transition between knowledge and dialogues. Empirical results show TBS models outperform end-to-end and knowledge-augmented RG baselines on most automatic metrics and generate more informative, specific, and commonsense-following responses, as evaluated by human annotators. TBS also generates knowledge that makes sense and is relevant to the dialogue around 85% of the time 1 . Human communication strives to achieve common ground , consisting of mutual beliefs and common knowledge (Stalnaker, 1978; Clark and Schaefer, 1989). Such common ground depends not only on utterances, but also implicit knowledge. For example, in Figure 1, this common ground includes the relevant implicit background knowledge  rose is a type of flower . Integrating such common ground in utterances is an implicit process often referred to as knowledge grounding (Clark and Brennan, 1991). Recent state-of-the-art neural response generation (RG) models based on pre-trained language models (LM) mostly produce responses in an end-to-end manner (Vaswani et al., 2017; Zhang et al.,  Work done while Pei Zhou was an intern at Amazon Alexa AI 1 Code and data will be released after approval. 2020a; Lewis et al., 2020), i.e. , models are trained to take history and produce a response. Since implicit knowledge is unstated in dialogue history, RG models do not explicitly learn knowledge grounding and may generate uninformative and hallucinated responses (Serban et al., 2017; Welleck et al., 2019; Roller et al., 2021). Knowledge-grounded RG (Ghazvininejad et al., 2018; Dinan et al., 2019; Gopalakrishnan et al., 2019) addresses this issue, however, most approaches require a knowledge base (KB) to retrieve knowledge for RG (Zhou et al., 2018; Zhao et al., 2020; Eric et al., 2021), which may suffer from the limited knowledge coverage of the used KBs. Some work also casts knowledge as a latent factor in generation (Tuan et al., 2020; Xu et al., 2021), which makes it hard to examine the quality of knowledge generation and how exactly RG uses the implicit knowledge, posing interpretability concerns. We propose Think-Before-Speaking (TBS), an RG framework that trains the RG model to explicitly generate the implicit knowledge and use this knowledge to generate a response, inspired by inquiry-based discovery learning (Bruner, 1961). We argue that this decomposition brings three ma-jor benefits: 1) compared with end-to-end RG, generated knowledge augments and/or constrains RG to produce more informative responses; 2) compared with knowledge-retrieval models, explicitly generating intermediate groundings can potentially generalize to knowledge not included in KBs and synergize with the RG process; 3) explicitly generated implicit knowledge used in RG provides a faithful explanation of the response intent. This new RG paradigm poses three main challenges: (1) how to identify implicit commonsense knowledge associated with dialogue turns for training the knowledge generation module; (2) how to represent structured knowledge in natural language (NL) for neural generative models; and (3) how to integrate knowledge and dialogues while distinguishing implicit and explicit parts in responses. To collect knowledge associated with each dialogue instance for training the TBS generative model, we propose weak supervision procedures to automatically align knowledge with each dialogue turn, rather than manually collecting human-annotations, which is expensive and unscalable. This is achieved by using ConceptNet (Speer et al., 2017) as our knowledge base and different matching approaches to identify the implicit knowledge. We explore several ways to format knowledge originally represented as structured triples into natural language so that RG models can adapt to the knowl-edge+response generation task easily. We experiment with structured triples, triples converted to natural language, and a more colloquial question answering format. To ensure a smooth transition between knowledge and dialogues, we consider using special symbols or prompts as separators. To evaluate the TBS framework, we introduce new evaluation protocols to cover different aspects of the system, including response quality, knowledge quality, and how TBS models leverage generated knowledge. We conduct extensive human evaluations for different variants of our training procedure. Our experimental results show that our models produce more informative, specific, and responses that make more common sense compared to end-to-end RG models and other knowledge-augmented models such as knowledge-selection. Knowledge quality analysis shows that at least 85% of generated knowledge makes sense and is relevant, and the generated novel knowledge (not in ConceptNet) also has high quality. Furthermore, our TBS model even outperforms an RG model that takes in knowledge obtained using ground-truth responses, showing that explicitly generating implicit knowledge is a promising direction for response generation in open domain dialogue systems. Our TBS RG paradigm extends the traditional RG setting by incorporating an additional component of implicit knowledge in the generation process to externalize the knowledge grounding step in RG. We follow the common dialogue response generation setup (Weizenbaum, 1966; Ritter et al., 2011; Sordoni et al., 2015): given a dialogue history H (a sequence of dialogue utterances), generate an appropriate response R . Current neural RG models often frame this task as a conditional language modeling problem. Specifically, given a history ( H ) consisting of a sequence of n dialogue turns: X 1 , X 2 , ..., X n (each turn refers to an utterance containing a sequence of t i tokens: x i, 1 , x i, 2 , ..., x i,t i ) and a response ( R ) sentence Y comprised of a sequence of m tokens y 1 , y 2 , ..., y m , RG models aim to learn the conditional probability distribution by training on human dialogues: P  ( R | H ) = m (cid:89) i =1 P  ( y i | y <i , X 1 , ..., X n ) . To make the implicit knowledge grounding step explicit, we introduce a new component to RG  implicit knowledge that is conditioned on the dialogue history H . We use I to denote the implicit knowledge for brevity, which contains multiple natural language (NL) statements I = Z 1 , Z 2 , ... (each containing a sequence of tokens: z i, 1 , z i, 2 , ... ) expressing commonsense knowledge. For example, in Figure 1,  rose is a type of flower  and  rose is a symbol of love  are two NL statements expressing the implicit commonsense knowledge. To emulate realistic conversation scenario, we also fuse dialogue history H in traditional RG with implicit knowledge I for each turn and denote it with H  . i.e. H  = X 1 , I 1 , X 2 , I 2 ..., X n , where I i indicates the implicit knowledge statements for the i-th turn in the dialogue history. based learning (Bruner, 1961; Shwartz et al., 2020a), our TBS RG paradigm requires models to first generate implicit knowledge I conditioned on H  , i.e. P  ( I n | H  = X 1 , I 1 , X 2 , I 2 ..., X n ) . This section introduces our proposed TBS method to train a generative model that can both talk with itself to explicitly generate background commonsense knowledge ( P  ( I | H  ) ) and then generate response afterwards, P  ( R | H  , I ) . Figure 2 illustrates the process to train the TBS models. To pair each dialogue with appropriate implicit knowledge, we first define a matching process and use ConceptNet (Speer et al., 2017) as the implicit knowledge source (Section 3.1). Then, to construct training instances, we face two key method design choices: how to represent knowledge (3.2) and how to connect the knowledge with the dialogue (3.3). Finally, we train TBS RG models to learn P  ( I | H  ) and P  ( R | H  , I ) with the same parameters  . The following sections explain these components in details. To train TBS models we need dialogue datasets consisting of a dialogue history, a response, and the knowledge statement connecting them. We focus on two methods that create weakly-supervised knowledge labels for dialogues as they are more scalable and cost less than human annotations. Hard-Matching The hard-matching process first lemmatizes all the non-stop words in each utterance, then it identifies knowledge triples whose two concepts appear in an utterance and the next turn respectively. This is the same as the filtering process in Zhou et al. (2021a) and is closely related to distant supervision methods for relation extraction (Craven et al., 1999; Mintz et al., 2009). For more details, refer to Appendix A.1. Soft-Matching Using Embedding Similarity Hard-matching only captures the surface form and neglects many important semantic relations between words. We thus develop a soft-matching procedure using embedding similarity from SentenceBERT (Reimers and Gurevych, 2019) to measure semantic relations between dialogue turns and triples in ConceptNet. Specifically, we first extract candidate triples from ConceptNet with one concept appearing in the i th turn. Next, we form a query by concatenating the i th turn and the next ( i + 1) th turn response. Finally, we encode the query and all triple candidates using SentenceBERT and use cosine similarity to find the semantically closest triples as matched knowledge. More details are presented in Appendix A.1. Implicit commonsense knowledge I stored in ConceptNet is in the form of (subject s , relation r , object o ) triples, such as (rose, TypeOf, flower) , which is not compatible with RG models, which operate on NL sentences and may not include relation tokens in their trained vocabulary. Here we design two alternatives to represent the grounded knowledge and use the implicit knowledge in Figure 1 as a running example. Map Relations to Natural Language (NL) To convert ConceptNet triples into NL, we follow a common practice and map every relation r in the triple to its NL template, and fill in s and o in the template (Levy et al., 2017). We use the same mapping as that used in COMET (Bosselut et al., 2019), covering all standard types of relations in ConceptNet. For example, rose is a type of flower; rose is a symbol of love . Information-Seeking Question-Answer Pairs Another format to convert triples to NL sentences is through asking and answering information-seeking questions. Shwartz et al. (2020b) designed templates of information-seeking questions and answers to provide background knowledge for LMs. We adopt a similar strategy and design a template for each relation in ConceptNet. For example, What is a type of flower? Rose is a type of flower. Rose is a symbol of what? Rose is a symbol of love . The mappings we use for these two types of representations are shown in Appendix A.2. To help our RG models learn the TBS paradigm and generate outputs structured similarly, i.e., implicit knowledge first and then responses, we need to properly connect knowledge and dialogues in our data. Here we consider two alternatives for creating such a transition. Special symbols . Following the common practice of separating sequences in neural LMs (Rad-ford et al., 2018; Devlin et al., 2019), we use a 1239 (cid:46)(cid:81)(cid:82)(cid:90)(cid:79)(cid:72)(cid:71)(cid:74)(cid:72)(cid:16)(cid:36)(cid:79)(cid:76)(cid:74)(cid:81)(cid:72)(cid:71)(cid:3)(cid:39)(cid:76)(cid:68)(cid:79)(cid:82)(cid:74)(cid:88)(cid:72)(cid:86)(cid:3)(cid:11)(cid:43)(cid:15)(cid:3)(cid:44)(cid:15)(cid:3)(cid:53)(cid:12) (cid:53)(cid:72)(cid:86)(cid:83)(cid:82)(cid:81)(cid:86)(cid:72)(cid:3)(cid:11)(cid:53)(cid:12) (cid:39)(cid:76)(cid:68)(cid:79)(cid:82)(cid:74)(cid:88)(cid:72)(cid:3)(cid:43)(cid:76)(cid:86)(cid:87)(cid:82)(cid:85)(cid:92)(cid:3)(cid:11)(cid:43)(cid:12)(cid:3) (cid:44)(cid:80)(cid:83)(cid:79)(cid:76)(cid:70)(cid:76)(cid:87)(cid:3)(cid:46)(cid:81)(cid:82)(cid:90)(cid:79)(cid:72)(cid:71)(cid:74)(cid:72)(cid:3)(cid:11)(cid:44)(cid:12)(cid:3) (cid:94)(cid:195)(cid:348)(cid:224)(cid:262)(cid:265)(cid:294)(cid:256)(cid:728)(cid:132)(cid:334)(cid:300)(cid:224)(cid:234)(cid:340)(cid:340) (cid:44)(cid:3)(cid:81)(cid:72)(cid:72)(cid:71)(cid:3)(cid:87)(cid:82)(cid:3)(cid:69)(cid:88)(cid:92)(cid:3)(cid:86)(cid:82)(cid:80)(cid:72)(cid:3)(cid:288)(cid:82)(cid:90)(cid:72)(cid:85)(cid:86)(cid:3)(cid:73)(cid:82)(cid:85)(cid:3)(cid:80)(cid:92)(cid:3)(cid:90)(cid:76)(cid:73)(cid:72)(cid:17) (cid:55)(cid:85)(cid:76)(cid:83)(cid:79)(cid:72)(cid:86)(cid:3)(cid:70)(cid:82)(cid:81)(cid:89)(cid:72)(cid:85)(cid:87)(cid:72)(cid:71)(cid:3)(cid:87)(cid:82)(cid:3)(cid:49)(cid:47) (cid:53)(cid:82)(cid:86)(cid:72)(cid:3)(cid:76)(cid:86)(cid:3)(cid:68)(cid:3)(cid:87)(cid:92)(cid:83)(cid:72)(cid:3)(cid:82)(cid:73)(cid:3)(cid:263)(cid:82)(cid:90)(cid:72)(cid:85)(cid:30)(cid:53)(cid:82)(cid:86)(cid:72)(cid:3)(cid:76)(cid:86)(cid:3)(cid:68)(cid:3)(cid:86)(cid:92)(cid:80)(cid:69)(cid:82)(cid:79)(cid:3)(cid:82)(cid:73)(cid:3)(cid:79)(cid:82)(cid:89)(cid:72)(cid:30)(cid:17)(cid:17)(cid:17) (cid:51)(cid:72)(cid:85)(cid:75)(cid:68)(cid:83)(cid:86)(cid:3)(cid:92)(cid:82)(cid:88)(cid:322)(cid:71)(cid:3)(cid:69)(cid:72)(cid:3)(cid:76)(cid:81)(cid:87)(cid:72)(cid:85)(cid:72)(cid:86)(cid:87)(cid:72)(cid:71)(cid:3)(cid:76)(cid:81)(cid:3)(cid:85)(cid:72)(cid:71)(cid:3)(cid:85)(cid:82)(cid:86)(cid:72)(cid:86)(cid:17) (cid:52)(cid:88)(cid:72)(cid:86)(cid:87)(cid:76)(cid:82)(cid:81)(cid:3)(cid:36)(cid:81)(cid:86)(cid:90)(cid:72)(cid:85)(cid:76)(cid:81)(cid:74) (cid:58)(cid:75)(cid:68)(cid:87)(cid:3)(cid:76)(cid:86)(cid:3)(cid:68)(cid:3)(cid:87)(cid:92)(cid:83)(cid:72)(cid:3)(cid:82)(cid:73)(cid:3)(cid:263)(cid:82)(cid:90)(cid:72)(cid:85)(cid:34)(cid:3)(cid:53)(cid:82)(cid:86)(cid:72)(cid:3)(cid:76)(cid:86)(cid:3)(cid:68)(cid:3)(cid:87)(cid:92)(cid:83)(cid:72)(cid:3)(cid:82)(cid:73)(cid:3)(cid:263)(cid:82)(cid:90)(cid:72)(cid:85)(cid:30)(cid:17)(cid:17)(cid:17) (cid:39)(cid:76)(cid:68)(cid:79)(cid:82)(cid:74)(cid:88)(cid:72)(cid:39)(cid:68)(cid:87)(cid:68)(cid:86)(cid:72)(cid:87)(cid:86) (cid:46)(cid:81)(cid:82)(cid:90)(cid:79)(cid:72)(cid:71)(cid:74)(cid:72)(cid:3)(cid:41)(cid:82)(cid:85)(cid:80)(cid:68)(cid:87)(cid:86) (cid:38)(cid:82)(cid:81)(cid:81)(cid:72)(cid:70)(cid:87)(cid:76)(cid:81)(cid:74)(cid:3)(cid:43)(cid:15)(cid:3)(cid:44)(cid:15)(cid:3)(cid:68)(cid:81)(cid:71)(cid:3)(cid:53)(cid:3) (cid:16) (cid:54)(cid:83)(cid:72)(cid:70)(cid:76)(cid:68)(cid:79)(cid:3)(cid:54)(cid:92)(cid:80)(cid:69)(cid:82)(cid:79)(cid:86) (cid:16) (cid:49)(cid:47)(cid:3)(cid:51)(cid:85)(cid:82)(cid:80)(cid:83)(cid:87)(cid:86) (cid:41)(cid:76)(cid:81)(cid:68)(cid:79)(cid:3)(cid:87)(cid:85)(cid:68)(cid:76)(cid:81)(cid:76)(cid:81)(cid:74)(cid:3)(cid:76)(cid:81)(cid:86)(cid:87)(cid:68)(cid:81)(cid:70)(cid:72)(cid:29) (cid:323)(cid:44)(cid:3)(cid:81)(cid:72)(cid:72)(cid:71)(cid:3)(cid:87)(cid:82)(cid:3)(cid:69)(cid:88)(cid:92)(cid:3)(cid:86)(cid:82)(cid:80)(cid:72)(cid:3)(cid:288)(cid:82)(cid:90)(cid:72)(cid:85)(cid:86)(cid:3)(cid:73)(cid:82)(cid:85)(cid:3)(cid:80)(cid:92)(cid:3)(cid:90)(cid:76)(cid:73)(cid:72)(cid:17)(cid:3)(cid:31)(cid:76)(cid:80)(cid:83)(cid:79)(cid:76)(cid:70)(cid:76)(cid:87)(cid:33)(cid:3) (cid:85)(cid:82)(cid:86)(cid:72)(cid:3)(cid:76)(cid:86)(cid:3)(cid:68)(cid:3)(cid:87)(cid:92)(cid:83)(cid:72)(cid:3)(cid:82)(cid:73)(cid:3) (cid:263)(cid:82)(cid:90)(cid:72)(cid:85)(cid:30)(cid:3)(cid:85)(cid:82)(cid:86)(cid:72)(cid:3)(cid:76)(cid:86)(cid:3)(cid:68)(cid:3)(cid:86)(cid:92)(cid:80)(cid:69)(cid:82)(cid:79)(cid:3)(cid:82)(cid:73)(cid:3)(cid:79)(cid:82)(cid:89)(cid:72)(cid:30) (cid:3)(cid:31)(cid:18)(cid:76)(cid:80)(cid:83)(cid:79)(cid:76)(cid:70)(cid:76)(cid:87)(cid:33)(cid:3)(cid:51)(cid:72)(cid:85)(cid:75)(cid:68)(cid:83)(cid:86)(cid:3) (cid:92)(cid:82)(cid:88)(cid:322)(cid:71)(cid:3)(cid:69)(cid:72)(cid:3)(cid:76)(cid:81)(cid:87)(cid:72)(cid:85)(cid:72)(cid:86)(cid:87)(cid:72)(cid:71)(cid:3)(cid:76)(cid:81)(cid:3)(cid:85)(cid:72)(cid:71)(cid:3)(cid:85)(cid:82)(cid:86)(cid:72)(cid:86)(cid:17)(cid:324) (cid:55)(cid:85)(cid:68)(cid:76)(cid:81)(cid:3)(cid:53)(cid:42)(cid:3)(cid:80)(cid:82)(cid:71)(cid:72)(cid:79)(cid:86)(cid:3)(cid:87)(cid:82)(cid:3)(cid:74)(cid:72)(cid:81)(cid:72)(cid:85)(cid:68)(cid:87)(cid:72)(cid:3)(cid:88)(cid:81)(cid:71)(cid:72)(cid:85)(cid:79)(cid:76)(cid:81)(cid:72)(cid:71)(cid:3)(cid:44)(cid:14)(cid:53)(cid:3)(cid:83)(cid:68)(cid:85)(cid:87) Figure 2: Method illustration. special symbol to serve as the separator. We enclose the implicit knowledge I with special symbols  < implicit >  and  < /implicit >  and add it between H  and R , for example,  < speaker1 > I need to buy some flowers for my wife. < implicit > rose is a type of flower < /implicit > < speaker2 > Perhaps you'd be interested in red roses. Natural language prompts . More recent work has found that NL prompts help LMs to perform better on various downstream tasks, including natural language generation (NLG) (Brown et al., 2020; Liu et al., 2021; Zheng and Huang, 2021). Here we use the NL prompts to prompt RG models to generate implicit knowledge and responses. We use  The following background knowledge is helpful for generating the response:  to elicit knowledge and  Grounded on the background knowledge, what does the speaker probably say in the next response?  to elicit response. After constructing knowledge-aligned dialogues, each of our data instances is a sequence of tokens with three components: a dialogue history H  fused with potential implicit knowledge after each turn, implicit knowledge (empty or nonempty) I , and a response R . We split each instance d ( H  , R, I )  D to first train the model to generate just the knowledge I based on H  , P  ( I | H  ) , and then train it to generate R based on both I and H  , P  ( R | H  , I ) . Formally, we follow standard way of modeling P  in auto-regressive neural RG models and use Maximum Likelihood Estimation (MLE) to train our model to maximize P  ( I | H  ) (knowledge generation KG) by minimizing the conditional negative log-likelihood loss (NLL): LKG =  m (cid:88) i =1 log P  ( Z i | Z <i , X 1 , ..., X n ) , where Z i is the i-th statement in I . And to model P  ( R | H  , I ) we minimize: LRG =  m (cid:88) i =1 log P  ( y i | y <i , X 1 , I 1 ..., X n ) . We train one generative model on these losses in one-pass with splitted instances for KG and RG instead of multiple training phases. During inference, we only provide dialogue history as input and the model has to generate knowledge and responses. We consider dialogues from four datasets: Dai-lyDialog (Li et al., 2017), EmpatheticDia-logues (Rashkin et al., 2019), MuTual (Cui et al., 2020), and SocialIQA-prompted Commonsense-Dialogues (Zhou et al., 2021a). For training, we use the filtered version of the four datasets from Zhou et al. (2021a), which ensures each dialogue contains at least one commonsense knowledge triple from ConceptNet. In total, the training data contains 31k dialogues with 159k utterances. We reserve 10% of data as a development set for evaluating model training and selecting hyper-parameters. Table 1 shows the number of instances resulted from applying our hardand soft-matching procedures to our training data in order to construct knowledge-aligned dialogues. the response, we use the test data from the original data distribution of the 4 datasets mentioned above. The testing data consists of around 3k dialogues. We use DialoGPT-medium (Zhang et al., 2020a) as our base model, which is a commonly-used end-to-end RG model. We fine-tune DialoGPT using all of the 159K dialogue instances. We also use DialoGPT to serve as the backbone model and consider three variables in our TBS model configuration introduced from Sections 3.1 to 3.3: hard matching or soft -matching, special symbol as separator or NL prompt , and triple-convertedNL to represent knowledge or information seeking QA pairs. To justify our choice of using one model to do both KG and RG, we also compare with TBS-Two Model where we train separate models for knowledge generation (KG) and RG using the same training data. Our default model configuration is hard-symbol-NL . We also compare several knowledge-grounded RG baselines that retrieve external knowledge or generate knowledge with another model. For retrieval, we follow most common approaches in knowledge-selection (Zhao et al., 2017; Wolf et al., 2020; Eric et al., 2021) and train RoBERTa (Liu et al., 2019) to classify triples using our knowledge-aligned data (matched or not matched), and use it to label candidate triples during testing ( KS-RoBERTa ). For the generative model, we use COMET (Bosselut et al., 2019) as a commonsense knowledge generator ( KG-COMET ). Furthermore, we consider RG models that take the hard-matched or soft-matched knowledge obtained from the ground-truth response ( Hard-GT and Soft-GT ). Note that though there is noise in hard-matching or soft-matching procedure, this setting uses the next turn response and is likely to provide relevant knowledge. Implementation details for all the models are shown in Appendix B.1. 2005), ROUGE (Lin, 2004), CIDEr (Vedantam et al., 2015) and SkipThoughts (Kiros et al., 2015). We also use GRADE (Huang et al., 2020), a reference-free metric shown to have consistent correlation with human judgements (Yeh et al., 2021) to ensure the validity of experimental results. Human Evaluation We conduct extensive human evaluation using 300 randomly sampled instances from unseen test dialogues described above. For response quality , we conduct pairwise comparison where we present a dialogue history and two responses made by two different models and ask them to choose one or select  not sure  based on different criteria (Zhou et al., 2018; Zhang et al., 2020b) 2 . We evaluate on six dimensions: which response is more grammatical , coherent , engaging , informative , specific , and makes common sense (Zhang et al., 2020b; Roller et al., 2021). More details of the instructions for annotators on each dimension with examples are included in Appendix B.2. For knowledge quality , we evaluate the generated knowledge in isolation ( does this knowledge make sense ) and in conjunction with the context for relevance. We perform majority voting per instance using three annotators from Amazon Mechnical Turk (AMT). We use Fleiss' Kappa (  ) (Fleiss, 1971) to measure agreement among the annotators. By evaluating our TBS model variants with other baselines, we aim to address the following questions: 1) do TBS models produce better responses than standard end-to-end RG models? 2) compared with other approaches to retrieve or generate additional knowledge, is TBS more helpful for RG? 3) do TBS RG models generate knowledge that makes sense and is relevant to the dialogue context? 4) do TBS models faithfully leverage the generated knowledge? Model variant analysis To find the best-performing configuration of our TBS method, we consider alternatives as discussed in Sections 3.1 to 3.3, and conduct 4 pairwise comparisons: soft vs. 2 We choose to conduct pairwise comparison since multiple previous work has shown that it produces a more reliable evaluation than directly asking humans to score the response, which is a highly subjective task (Amidei et al., 2019; CallisonBurch et al., 2007; Celikyilmaz et al., 2020) hard , prompt vs. symbol , and QA vs. relation-converted NL format . From Table 2, we find that using soft-matching to create knowledge-aligned dialogue dataset produces more grammatical responses and responses that make more common sense, with  =0.64-0.73, indicating substantial agreement according to one interpretation from Landis and Koch (1977). Using QA to represent knowledge makes the responses more grammatical, coherent, commonsensical, and also achieves the best performance on average on six dimensions. We also compare results that combine these alternatives, e.g., soft-symbol-QA (due to space constraints, results are shown in Appendix C.1), however, we do not observe significant improvements after combining these alternatives and our best configuration in terms of average improvement is still hard-symbol-QA . We thus use hard-symbol-QA as our final configuration and refer to it as TBS throughout this section. Does TBS produce better responses vs. end-to-end RG? By comparing TBS and end-to-end DialoGPT-ft model in Table 3 and Figure 3, we find that TBS models produce better-quality responses using both automatic and human evaluations. Specifically, even though hard-matching only annotates about 33% of the training instances, TBS outperforms end-to-end RG model significantly on most automatic metrics. From human evaluation (  =0.62-0.69), we find our TBS model performs on par with DialoGPT trained on more data in grammar, coherence, and engagingness, and achieves statistically-significant (p < 0.05) improvement on informativeness, specificity, and the common sense aspects of generated responses 3 . We argue that by providing weakly-supervised knowledge labels and TBS training, RG models require less data and can generate quality responses with improvement in the informativeness, specificity, and common sense aspects of the responses. Is TBS knowledge generation better than other knowledge-augmented RG? We compare TBS models with other knowledge-augmented baselines that retrieve knowledge from ConceptNet using embedding scores (KS-SBERT) or a trained selector (KS-RoBERTa), or generate from another model (KG-COMET). From Table 3, we find that these models perform similarly to the end-to-end DialoGPT model and are outperformed by TBS models on most automatic metrics. Figure 3 shows that while TBS methods have significant improvements on all dimensions against knowledge-selection baselines, COMET as a knowledge generator has smaller gaps on informativeness, specificity, and common sense, but is outperformed significantly on grammar, coherence, and engagingness. Next we compare against the setup where we feed the model the knowledge that is derived using the ground-truth response (Hard/Soft-GT), i.e. , the provided knowledge is obtained using concepts appearing in the ground-truth response. From Table 3, we surprisingly find that even though our 3 We also conducted direct scoring in human evaluations and observed significant improvement (on average 7.3 out of 10 for TBS vs. 5.9 for DialoGPT-ft), but since it results in lower agreement (  =0.49), we focus on comparative evaluation. proposed TBS model has no access to response-leaking knowledge labels and is trained on much less data, the TBS RG model still achieves statistically significant improvement on GRADE and BLEU-4. And from human evaluation results in Figure 4, TBS model significantly improves the specificity and common sense aspect of responses while stays on par on other evaluation dimensions compared with the hard-GT model and improves even more compared with soft-GT. We find that one potential explanation is that only around 55% of Hard-GT knowledge is labeled as used in response whereas it is 77% in our TBS model (see Section 5.3). This is also related to how the RG model leverages the knowledge in training. Further analysis is needed to understand the effect of knowledge and the relationship between knowledge and responses. We then examine how well TBS RG models learn to generate knowledge on unseen dialogues. We use human evaluation and focus on three dimensions: does the model generate novel knowledge that does not appear in ConceptNet? does the gen-Model Novel Makes Sense Relevant KS-SBERT 0% 91.7%* 85.0% KS-RoBERTa 0% 77.7%* 76.3% KG-COMET 63.3% 68.3%/63.2% 67.5%/68.9% TBS-two-model 46.3% 89.0%/85.6% 90.7%/90.2% TBS-one-model 44% 86.3%/85.9% 85.7%/86.5% Table 4: Human evaluation on knowledge quality . erated knowledge statement make sense as a standalone fact? and is the generated knowledge relevant to the dialogue context? For the first question we directly query from ConceptNet and show percentages. For the latter two we follow Section 4.3 and show the percentages that MTurkers think the knowledge makes sense and is relevant from the 300 sampled test instances (the same used in response quality). We test our TBS model, the two-model variant, and other knowledge-augmented baselines introduced in Section 4.2. makes sense and is relevant Table 4 shows that TBS models can generate implicit knowledge that makes sense and is relevant to the context for around 85% of the time as judged by human annotators (  =0.73-0.80). Compared with knowledge-selection models that retrieve knowledge from ConceptNet, TBS generates knowledge that is similar in terms of common sense and has better relevance to the dialogue history. Compared with COMET that also generates knowledge, we find TBS models generate more knowledge that follows common sense and is relevant to the dialogue. Comparing two-model and one-model TBS, we find that two-model generates more knowledge that makes sense and is relevant, although its response quality is poorer (Table 3 and Figure 3). This might be due 1243 Grammatical Coherent Engaging Informative Specific Common Sense Evaluation Dimensions 30 35 40 45 50 55 60 P r e f e r e n c e P e r c e n t a g e s 50.0* 55.0* 53.3* 43.7 44.7 53.0* 38.0* 38.3* 40.3* 39.6 46.0 43.3* Effects on Noisy Knowledge Input Models TBSTBS-Noisy Knowledge Figure 5: Effects of noisy knowledge on response quality. to model synergies when learning both knowledge generation and response generation. Model generates novel knowledge We find a significant portion of novel knowledge generated from the COMET and TBS models that is not present in the training data. Furthermore, the quality of the generated novel knowledge is similar to that of knowledge existing in ConceptNet. COMET generates more new knowledge but the quality (both common sense and relevance) is significantly lower than TBS models. We include some examples of novel knowledge generated in Appendix C. In general we find that the new knowledge is complimentary to ConceptNet, not just a paraphrased version of existing triples (since in those cases the model will directly generate the ConceptNet triple). This shows a promising sign that TBS RG models can potentially generate good-quality novel knowledge labels for unseen dialogues. Most responses are knowledge grounded To examine how TBS methods leverage knowledge for RG, we also present annotators a history, generated knowledge, and generated response, and ask them whether the knowledge is used in response . We find that around 77% of generated knowledge is used in the generated response, i.e. , the response is grounded in the knowledge generated from TBS. Noisy knowledge heavily impacts quality To better showcase the connection between knowledge and response, we examine how knowledge quality generated from TBS methods can affect response quality. During inference, we randomly sample noisy knowledge from another dialogue, feed it to the model to generate a response conditioned on irrelevant knowledge, and compare the response quality with response generated from TBS knowledge. Fig 5 shows that there is a statistically significant (p  0.05) drop in response quality in four dimensions. This indicates that the quality of knowledge input heavily influences response quality and that TBS models generate better responses because of its decent knowledge quality. Qualitative examples and limitations We show several qualitative examples from different models and human responses in Table 5. We find that TBS generates relevant knowledge and responses grounded properly in that knowledge, whereas KS/KG models retrieve noisy knowledge and Hard-GT generates response not grounded in knowledge. Here we present a summary of error patterns of TBS models and discuss potential directions to improve. More examples can be found in Table 6. First, our matching procedures do not concern multi-hop triples that might be needed for complex reasoning chains. Second, ConceptNet mostly contains taxonomic and lexical knowledge ( RelatedTo, IsA, etc ), limiting the diversity of generated knowledge from TBS models. We plan to explore other knowledge resources such as ATOMIC2020 (Hwang et al., 2021) in the future. Third, currently the model always generates implicit knowledge. In future work, we are interested in training RG models that understand when implicit knowledge is needed based on the dialogue context. Open-Domain Dialogue Generation Recent work focused on fine-tuning large pre-trained transformer models (Radford et al., 2019; Zhang et al., 2020a; Roller et al., 2021) on massive dialogue data. Knowledge-augmented RG has been studied extensively to alleviate the issue of generic or hallucinated responses (Serban et al., 2017; Welleck et al., 2019; Roller et al., 2021). Most work retrieves relevant knowledge from knowledge candidates (wikipedia or KBs) and generates responses after incorporating additional knowledge in dialogue context (Ghazvininejad et al., 2018; Zhou et al., 2018; Wu et al., 2020). More recent work also explored other ways of constructing knowledge, such as by considering knowledge as a latent variable (Tuan et al., 2020; Li et al., 2020) and generating it implicitly. Our TBS framework differs from these two lines of work in that it explicitly generates knowledge in text and uses one generative model for both knowledge generation and RG. Understanding (NLU) Although explicit knowledge generation (KG) for RG has not been explored, similar methods have been proposed for NLU tasks such as question answering (Shwartz et al., 2020b). Previous work has also explicitly generated rationales that can be seen as helpful additional knowledge (Rajani et al., 2019). TBS differs from such work in that we consider a generative task and use the same generative model to do both KG and RG. Inspired by how humans contribute to the common ground during communication, We propose to train RG models that explicitly generate implicit knowledge and then respond (TBS). This brings us three main benefits compared with prior end-to-end RG models: 1) more informative and coherent responses by augmenting with knowledge; 2) generated knowledge provides faithful explanations of RG model's inner-workings; 3) models do not rely on external knowledge bases in response generation time. We first identify implicit knowledge in dialogues, explore different knowledge representation and transition choices, and demonstrate promising results compared with end-to-end and knowledge-grounded RG models from extensive evaluations. We find strong and promising results for TBS RG model compared with end-to-end RG. In particular, TBS can produce good quality and novel knowledge, outperform end-to-end RG models despite training on less data, and even produce better responses than RG models that take ground-truth knowledge. We hope our findings encourage more future studies on making RG models better emulate human communication process and produce better-quality responses. Our work aims to train RG models that explicitly generate implicit knowledge before responding. Sheng et al. (2021) have found biases in DialoGPT (our base model) responses and Mehrabi et al. (2021) have found representational harms in common sense resources. We acknowledge that the 1245 generated responses from our models might contain biases. All of the dialogue datasets and models are in English, which benefits English speakers more. We have conducted human evaluation using Amazon Mechanical Turks. We pay turkers around $15 per hour, well above the highest state minimum wage and engage in constructive discussions if they have concerns about the process. We also give each annotation instance enough time so that we do not pressure annotators.
abstain abstain objective method method abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain objective other abstain objective result objective method method objective method objective abstain result abstain method abstain other abstain method abstain abstain abstain abstain abstain abstain abstain method method abstain method method abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain method abstain method abstain abstain method method method abstain method abstain method method method method abstain method method method abstain method method method method method method method method method abstain abstain abstain abstain method method method abstain method method method abstain abstain abstain abstain result result abstain result abstain result abstain abstain result abstain result abstain abstain abstain result abstain result result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result result other abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result result result abstain abstain abstain abstain abstain abstain other other other other objective abstain abstain abstain objective objective objective result abstain method abstain abstain method abstain method abstain abstain