We propose knowledge internalization (KI), which aims to complement the lexical knowledge into neural dialog models. Instead of further conditioning the knowledge-grounded dialog (KGD) models on externally retrieved knowledge, we seek to integrate knowledge about each input token internally into the model's parameters. To tackle the challenge due to the large scale of lexical knowledge, we adopt the contrastive learning approach and create an effective token-level lexical knowledge retriever that requires only weak supervision mined from Wikipedia. We demonstrate the effectiveness and general applicability of our approach on various datasets and diversified model structures. Vacuous responses (Li et al., 2016; Ghazvininejad et al., 2018), such as, I don't know , are commonly observed in end-to-end neural dialog models (Shang et al., 2015; Sordoni et al., 2015). This is mostly because these models ignore the knowledge that resides in people's minds during a conversation. To bridge the gap, many existing works (Moghe et al., 2018; Dinan et al., 2018) have attempted to condition the dialog model on external knowledge, either a sentence or a paragraph, retrieved based on the utterance and/or previous context. This curates datasets with utterance-response-knowledge triples (see Fig 1(a)). These knowledge-grounded dialog (KGD) models, despite demonstrated effectiveness, suffer from two major problems. First, equipping models with sentence-level knowledge alone will limit responses' informativeness and diversity. As shown in Fig 1(a), with the knowledge retrieved giving the utterance, a KGD model can relate J.K Rowling to Khalsa Aid . However, retrieval based solely on sentence embeddings  The majority of this work was done while the first author was interning at Tencent AI Lab. will result in ignorance of lexical knowledge associated with individual tokens. In this example, the knowledge about J.K Rowling , COVID-19 , donates , and India , is ignored during the retrieval, due to the semantic gaps between those lexical knowledge sentences (see Fig 1(b)) and the utterance. This makes it rather difficult (if not impossible) for the model to generate responses carrying relevant information as shown in Fig 1(b). Second, retrieving knowledge for open-domain dialogs during inference incurs heavier computation, often involving similarity search over tens of millions of passages (Petroni et al., 2021). Existing systems (Zhao et al., 2020; Zheng et al., 2020) alleviate this problem relying on pre-selecting a 7945 small candidate set based on TF-IDF (Schtze et al., 2008), in sacrifice of the diversity and the accuracy of the retriever. Directly conditioning the dialog model on the retrieved text, these models are easily effected by the quality of the constructed candidate set and are thus prone to errors (Dinan et al., 2018; Kim et al., 2020; Zhao et al., 2020). In this work, we propose to complement the lexical knowledge into neural dialog models by K nowledge I nternalization ( KI ), a training approach based on contrastive learning (Hadsell et al., 2006). The central idea of KI is to integrate more fine-grained lexical knowledge about each input token internally into model parameters (e.g., word embeddings), rather than further conditioning the model on externally retrieved knowledge (e.g., directly copy and/or modify tokens from external knowledge when decoding). Our research contributions include:  a novel training objective (KI; 3.2) that infuses lexical semantics into word representations. With the knowledge internalized into the contextualized representation of every token, a dialog model can generate informative and diverse responses without engaging an external knowledge retrieval module during inference time, thus making the inference more efficient (6.1);  an effective token-level lexical knowledge retriever (4) trained with weak supervision to contextually align tokens in dialog corpora to their related and possibly different knowledge (Ap-pendix C). a demonstration of the effectiveness and general applicability of KI with extensive experiments on diversified dialog models and on three benchmark datasets: DailyDialog (Li et al., 2017), Wizard of Wikipedia (Dinan et al., 2018), and Commonsense Reddit Dataset (Zhou et al., 2018). To address the vacuous responses problem in neural dialog models, researchers propose to ground dialogs on real world knowledge and construct new corpora that contain utterance-response-knowledge triples. Specifically, responses are grounded to external knowledge derived from different knowledge sources (Zhou et al., 2018; Liu et al., 2018; Wu et al., 2019; Dinan et al., 2018; Moghe et al., 2018; Ghazvininejad et al., 2018; Mostafazadeh et al., 2017; Meng et al., 2020; Zhang et al., 2020). Among different sources, textual knowledge (Di-nan et al., 2018; Parthasarathi and Pineau, 2018; Qin et al., 2019) receives the most attention as it is easy to obtain and scale. However, the construction of knowledge-grounded datasets is costly and time-consuming. To build a more practical system without assuming a given knowledge, recent studies enhance KGD models with an extra knowledge selection component (Dinan et al., 2018; Kim et al., 2020; Zheng et al., 2020; Zhao et al., 2020). Most existing KGD models can be viewed as models with externalized knowledge , where knowledge is explicitly used as part of the model input. The principle behind these models is to copy words and/or modify sentences from external knowledge when generating responses (Wu et al., 2020; Zhu et al., 2017; Zhao et al., 2019). Our KI, on the other hand, does not explicitly present knowledge to dialog models for reading and/or copying. Instead, we inject and store external knowledge into models' parameters and encourage models to elicit the encoded knowledge during generation. The idea of knowledge internalization has also been explored in language modeling. Factual knowledge (Zhang et al., 2019; Sun et al., 2020; Liu et al., 2020), visual knowledge (Tan and Bansal, 2020) and syntactic knowledge (Kuncoro et al., 2020) have been injected into language models (LMs) and shown great promise in improving the performance of downstream tasks. KI differs from those knowledge-enhanced LMs in two aspects: (i) KI can be trained end-to-end with dialog models, while applying LMs on dialog generation often requires multiple rounds of pre-train and fine-tune. (ii) KI is lightweight that barely introduces extra parameters to the dialog model while applying LMs usually introduces hundreds of millions of extra parameters. In this section, we illustrate how to train a dialog model with knowledge internalization. To infuse more fine-grained lexical knowledge to a neural dialog model, we assume a dialog corpus where each token is aligned with relevant knowledge (we will discuss the construction of such a corpus in 4). In particular, for an input sentence X in the corpus, we assume each token x i  X is associated with a corresponding descriptive sentence K i . Given an utterance-response pair ( X, Y ) , where X = { x 1 , x 2 , . . . , x n } and Y = { y 1 , y 2 , . . . , y m } , neural dialog models generally minimize the negative log-likelihood loss: where P ( y i ) = P ( y i | y <i , X ) is the probability of generating the i -th response token y i given the utterance X and other tokens generated in previous steps y <i = { y 1 , y 2 , . . . , y i  1 } . P ( y i ) is generally modeled by a sequence-to-sequence model (Sutskever et al., 2014), which consists of an encoder and a decoder. The encoder represents X as a sequence of hidden vectors H ( X ) = h 1 , h 2 , ..., h n , where each h i is a low-dimensional representation of the token x i . The decoder generates y i based on H ( X ) and y <i , often with the attention mechanism (Bahdanau et al., 2014). Given a dialog corpus with token-level knowledge as discussed above, we now introduce a new training task: knowledge internalization ( KI ). In KI, we seek to boost dialog models by internalizing lexical knowledge into each token's representation. In particular, each token x i and its associated knowledge K i are first mapped into a low-dimensional space. We then adopt contrastive learning to shorten the distance between x i and K i in the space while enlarging that between x i and other irrelevant knowledge. Note that for each x i  X , dialog models' encoder can embed it into a contextualized representation h i . Therefore, we only need an extra knowledge encoder to represent K i as g ( K i ) (details will be given in  4.2). After h i and g ( K i ) are computed, we calculate the similarity between x i and K i by the inner product: s ( x i , K i ) = f 1 ( h i ) (cid:62) f 2 ( g ( K i )) , (2) where f 1 and f 2 are the functions that map the h i and g ( K i ) into the same vector space and normalize them. For each ( x i , K i ) pair, we randomly sample an in-batch unrelated knowledge K  i associated with other input sentences, where K  i (cid:54) = K i , to construct a negative sample pair ( x i , K  i ) in contrastive learning. Finally, the objective function of KI is defined by the contrastive loss between positive and negative sample pairs: LKI ( X )= n (cid:88) i =1 max (cid:8) 0 , m  s ( x i , K i )+ s ( x i , K  i ) (cid:9) , (3) where m denotes the margin. We now illustrate how to deploy KI on a neural dialog model. We use a sequence-to-sequence dialog model based on Transformer (Vaswani et al., 2017) as an example. The original model is trained to minimize the negative log-likelihood loss of response tokens, i.e., LNLL ( X, Y ) (see Eq. 1). We can conveniently integrate KI into the model by reusing the contextualized representations generated by the model's encoder. The training objective of a knowledge-internalized dialog model can then be formulated as: L = LNLL ( X, Y ) +  LKI ( X ) (4) where  is a hyperparameter. Note that the token-level knowledge is only required during training to compute LKI ( X ) . At the inference time, those relevant knowledge is no longer required as they have been internalized into model by KI, making inference more efficient. In this section, we present how to train an effective retriever to mine knowledge for each token in the dialog corpora. Given a dialog utterance X = { x 1 , x 2 , . . . , x n } , the trained retriever will retrieve a relevant knowledge K i for each token x i in X . The constructed token-knowledge alignments can then be used to train a knowledge-internalized neural dialog model as in  3. 4.1 Training Data Collection To train such a retriever, we need a corpus with token-level knowledge annotated. However, to our best knowledge, no human annotated data exist and it is prohibitively expensive to build one. We therefore seek to train the retriever with distant supervision. A straight-forward solution is to align the noun words in an utterance to certain knowledge graph triples using entity linking tools (Shen et al., 2014). The problem of that is it can only cover 7947 about 15% words in human conversations (Biber et al., 2000). To address this issue, we propose to mine token-knowledge distant annotations from Wikipedia. In each Wiki article, the first sentence S = { s 1 , s 2 , ..., s n } is mostly declarative that gives a high-level summary on the topic of the article. Thus this sentence can used as a lexical knowledge item, denoted as K (note that K and S refer to the same sentence here). Inspired by Tan and Bansal (2020), we then further associate every token in the sentence with this knowledge item. These constructed alignments (e.g., ( s i , K ) ) can then be used to train a token-level knowledge retriever. The core of the retriever's training is to learn a scoring function r ( s i | S, K ) to measure the relevance between a token s i and a knowledge item K , giving s i 'context S . Similar as Eq. 2, we implement the scoring function r ( s i | S, K ) as the inner product between s i 'contextualized token representation f ( h i ) and the knowledge representation f ( g ( K )) . Here, we use a pre-trained BERT (Devlin et al., 2019) model to obtain h i ; we apply another pre-trained BERT model to encode knowledge K and further generate g ( K ) with an average-pooling operator. Two BERT models will be fine-tuned with the retriever. Our training objective is to maximize the relevance score of aligned token-knowledge pairs while minimizing that of unaligned pairs. We also adopt the hinge loss similar as in Eq 3 by replacing x i in the dialog corpus to s i in the constructed token-knowledge pairs. Once the retriever is trained, we can use it to mine token-level lexical knowledge required in KI. We first construct a candidate knowledge base K that consists of 6.4 million knowledge items (first sentence) extracted from Wikipedia articles. Given a dialog utterance X = { x 1 , x 2 , . . . , x n } , we retrieve a lexical knowledge K i for each token x i by searching for the knowledge item that has the largest relevance score with x i . To improve the retrieval results, we further employ two useful strategies: (i) Stopword Masking , where we discard knowledge associated with stopwords; (ii) Exact Matching , where if an utterance token exactly matches the title of a Wikipedia article, we will directly return the first sentence of this article as the retrieval result. The retrieval process has two properties that can significantly improve dialog corpora's knowledge coverage. First, the retrieval is contextualized such that a token can be aligned to different knowledge items when it occurs in different contexts. Second, the retrieval is at token-level that enables us to associate each dialog sentence with multiple knowledge items (See Appendix C). In this section, we present the datasets and metrics used for evaluation. Datasets We use three datasets from various domains (statistics in Appendix A). The first one is DailyDialog (Li et al., 2017), a multi-turn dialog benchmark that contains daily dialogs recorded as utterance-response pairs. However, there is no knowledge associated with the dialogs in DailyDialog, making it difficult to evaluate the informativeness of generated responses. To fully illustrate the strength of KI, we further consider two knowledge-grounded datasets: (i) Wizard of Wikipedia (WoW) (Dinan et al., 2018), a multi-turn dataset that contains utterance-response-knowledge triples. For each dialog, a sentence retrieved from Wikipedia is selected to guide response generation. WoW contains two test sets: Test Seen/Unseen, where the latter includes topics that never appear in Train and Valid set. (ii) Commonsense Reddit Dataset (CRD) (Zhou et al., 2018): a weakly knowledge-grounded single-turn dataset. Each dialog in the dataset is paired with at least one triple automatically extracted from ConceptNet (Speer et al., 2017). Metrics We conduct both automatic evaluation and human annotations. For automatic evaluation, we evaluate the generated responses from three perspectives 1 :  Appropriateness : we employ Perplexity (PPL), corpus-level BLEU-4 (Papineni et al., 2002) and ROUGE-l (Lin, 2004). Diversity : the ratio of distinct uni/bi-grams in all generated texts, i.e., Distinct-1/2 (Li et al., 2016). 1 For PPL and %safe, smaller is better, while for all other metrics, larger is better. Informativeness : For WoW, we consider wikiF1 (Dinan et al., 2018), the overlapping F1 between the generated response and the grounded knowledge. For CRD, we calculate entity score (Ent.) (Zhou et al., 2018), the average number of entities per response. To further measure the likelihood of generating safe responses, we define %safe : the percentage of responses that contains I'm not sure or I don't know. 2 We also report the accuracy of knowledge selection (ACC) following Zheng et al. (2020). We further perform human annotations by randomly sampling 200/200/300/300 examples from WoW Test Seen/WoW Test Unseen/ CRD/DailyDialog, respectively. We recruit 5 annotators from a commercial annotation company to rate each response on a scale of 1-5 for its appropriateness (Zhang et al., 2020; Zheng et al., 2020) and informativeness (Young et al., 2018; Zhu et al., 2019). The former measures whether the topic of the response fits that of the utterance, while the latter evaluates whether a response provides new information. A response is scored 1 if it is not appropriate/informative at all, 3 if part of the response is appropriate/informative, 5 if it is highly related to utterance and context or it can provide rich information to deepen the discussion. 2 and 4 are for decision dilemmas. it with three sets of baselines: 2. We then investigate whether KI can complement or even further improve the state-of-the-art KGD model's performance. All model structures and training setups are given in Appendix B. 2 Upon manual inspection, we find that these two are the most common safe responses generated. 1. We first investigate the effectiveness and general applicability of KI by applying KI on conventional dialog models that are randomly initialized and trained with utterance-response pairs only. 3. As discussed in 2, although LMs differ from KI in many aspects, they also capture knowledge in their parameters. We thus compare KI with LMs to investigate its effectiveness in encouraging informative and appropriate responses. We first deploy KI on two representative neural dialog models that do not directly condition on any external knowledge: (i) Seq2Seq : a LSTM-based (Hochreiter and Schmidhuber, 1997) sequence-to-sequence model with the attention mechanism (Bahdanau et al., 2014); (ii) Transformer (Vaswani et al., 2017): an encoder-decoder architecture relying solely on the attention mechanisms. Effectiveness As shown in Table 1's Setup 1 (rows 1-8), dialog models with KI consistently outperform their counterparts without KI on almost all the metrics across the datasets used. We want to point out the advantage of KI from two perspectives: (1) Promoting informativeness. We first observe that applying KI can significantly improve the wikiF1 and Ent. scores. Unlike KGD models that can generate informative responses by explicitly copying words from given knowledge, models discussed here are not provided with any external knowledge during testing (thus copy mechanism is not applicable for them). This suggests that the improvement in informativeness should be attributed to the effectiveness of KI in injecting knowledge into models' parameters. The Info. scores from human evaluation in Table 2 can also substantiate our findings. (2) Promoting diversity and reducing occurrence of safe response. Compared with the plain models, models with KI can significantly improve the Distinc-1/2 scores on all the test sets (some-times doubled, even tripled). We also see a significant reduction of safe responses by the gap in %safe scores. Those improvements are powered by the rich lexical knowledge used in KI (see Appendix C). Efficiency Besides the improvements in responses' quality, KI is also very efficient during inference. We report the decoding speed of Transformer and Transformer+KI in Table 3. As we can see, KI does not incur any extra computation during inference. We then apply KI on DiffKS (Zheng et al., 2020) a state-of-the-art model that uses a knowledge-aware decoder to generate a response based on utterance and the knowledge retrieved from a set of candidates. In the empirical study, DiffKS has outperformed many KGD models like CCM (Zhou et al., 2018) 4 and ITDD (Li et al., 2019). We enhance DiffKS by applying KI on its context encoder. The rest of the model remains unchanged. Table 1 Rows 9-10 show that DiffKS with KI improves ACC over the plain DiffKS model. The reason is that with the injection of token-level knowledge, DiffKS can better understand the utterance, which leads to more accurate knowledge selection and thus less noisy external knowledge. As a result, we observe clear gains on overlapping-based metrics (BLEU and ROUGE). These results emphasize the importance of more fine-grained knowledge in KGD. Human evaluation results (Table 2) also suggest that KI can help KGD models in generating more informative and appropriate responses. We follow previous practice (Rothe et al., 2020) to replace the Transformer's encoder with LMs and keep the decoder the same as the Transformer above. 5 We consider two baselines: (i) Bert2Rnd : Initializing Transformer's encoder with a pre-trained BERT, which has been shown capturing rich factual knowledge during pre-training (Petroni et al., 2019; Jiang et al., 2020). (ii) Ernie2Rnd : Initializing the encoder with ERNIE 2.0 (Sun et al., 2020), a knowledge-enhanced BERT which is pre-trained with novel objectives that injecting lexical, 4 Comparison with CCM is in Appendix D 5 We keep the hidden state dimension of decoder consistent with the LMs to enable encoder-decoder attention. syntactic, and semantic knowledge into its parameters (Zhang et al., 2019). From Table 4, we see that parameters of LM-based models (Bert2Rnd and Ernie2Rnd) are more than three times than that of the Transformer baseline. But they do not seem to help improve informativeness (based on wikiF1, BLEU-4, and Info.) of responses. This indicates that although pre-trained LMs can encode knowledge in their parameters, eliciting the encoded knowledge for response generation is difficult when we only have utterance-response pairs for training. Another reason might be that previously learned knowledge is forgotten due to catastrophic forgetting (McCloskey and Cohen, 1989). Comparing with knowledge-enhanced LMs, KI is more lightweight and more effective. In addition, we observe that introducing LMs can significantly improve responses' diversity as KI does. However, according to Appr. metric and upon manual examination, we find that although the generated responses are diverse, they are often inconsistent with the context or hallucinating non-existing facts (e.g., "Yea, Canada is the largest country in the US."). These are known issues for LMs as discussed in Dou et al. (2021); Shuster et al. (2021); Chen et al. (2020). We also apply KI on Bert2Rnd/Ernie2Rnd, but we do not observe significant improvements as when applied on randomly initialized models. This could be due to the fact that we implement KI using knowledge from Wikipedia, which is already part of LMs' training corpora. We leave it as future work to investigate how to use KI to elicit knowledge from LMs better (e.g., use adapters (Xu et al., 7950 Model DailyDialog CRD WoW Seen WoW Unseen Model WoW Seen WoW Unseen Appr. In this section, we perform an in-depth analysis to understand the effectiveness of KI. We investigate the working principle of KI by visualizing the token embeddings learned on WoW. We use principal component analysis (PCA) to map embeddings into a two-dimensional space as shown in Fig 2. Since there is no co-occurrence of British and Rowling in WoW, their embeddings learned by Transformer are distant (see Fig 2(a)). However, their embeddings learned by Transformer+KI (see Fig 2(b)) are much closer. This is because KI injects lexical knowledge (i.e., a British author) into the embedding of Rowling. Specifically, the Euclidean distances between British and Rowling are 0.37 for Transformer and 0.22 for Transformer+KI, respectively. This observation sheds light on the working principle of KI: the contrastive learning objective shortens the embedding distance between a token and tokens from its lexical knowledge. Thus when decoding, if a token is predicted (e.g. Rowl-ing), its relevant knowledge tokens (e.g., British) are likely to receive high probabilities and be selected in the following steps (see the J.K Rowling example in Fig 1(b). Firstly, we experiment with a model variant (de-noted as Random), which randomly assign knowledge to each utterance token. Results in Table 5 (Row 2) validate the effectiveness of the proposed token-knowledge retriever. To further show the advantage of token-level knowledge, we consider a model variant in which we degenerate token-level KI to sentence-level by assigning all utterance tokens to a same lexical knowledge (we denote it as Sentence-level knowledge ). Given the lexical knowledge retrieved for each token in an utterance, the sentence-level knowledge is chosen as the most-frequent one among all token-level knowledge. The results are summarized in Table 5 (Row 3). Note that token-level knowledge results in better performance than sentence-level knowledge. This shows that fine-grained information is useful in promoting more informative and diverse responses. Lastly, we dive deep into the lexical knowledge retrieved to investigate which type of knowledge is most helpful in response generation. We classify a retrieved knowledge into two types: factual knowledge, which describes a real-world subject (e.g., knowledge about J.K Rowling), and is often associated with noun words in the utterance; linguistic knowledge, which explains the meaning of certain words (e.g., knowledge about donate, see Fig 1(b), and is often associated with words except nouns.We use part-of-speech (POS) tags to classify tokens and their associated knowledge. We consider two model variants that only use fac-tual/linguistic knowledge in KI respectively, denoted as factual and linguistic . In Fig 3, we compare these two model variants to a vanilla model without KI (denoted as base ), and a full model that uses both knowledge (denoted as both ). We find that injecting factual knowledge brings significant improvements on BLEU-4 and ROUGE-l. We also observe similar, albeit smaller improvements when equipping with linguistic knowledge. More interestingly, these two types of knowledge can complement one another to further improve the model 7951 0.6 0.4 0.2 0.0 0.2 0.4 0.6 0.4 0.2 0.0 0.2 0.4 rowling british author harry potter performance. This emphasizes the need to consider non-factual knowledge in KGD, which is usually ignored in previous study. To understand what causes the difference between using factual and linguistic knowledge, we compute Knowledge Coverage : the percentage of ground truth response tokens that have been recalled in the retrieved knowledge. As we can see from Fig 3(c), factual knowledge is more helpful because people tend to respond based on knowledge related to subjects (usually nouns) appearing in the dialog. We show an example case in Appendix E to demonstrate how KI improves dialog generation and what the limitation is. We propose knowledge internalization (KI), which aims to incorporate the lexical knowledge into neural dialog models. Models with KI can generate informative and diverse responses without explicitly conditioning on external knowledge. To provide the fine-grained knowledge needed in KI, we also build an effective token-level lexical knowledge retriever that contextually align tokens in a sentence to their related knowledge. We show the effectiveness and general applicability of KI by evaluating KI on various datasets and diversified model structures. This project is supported by the Tencent AI Lab Rhino-Bird Focused Research Program, the Shanghai Committee of Science and Technology, China (Grant No. 21DZ1100100). This research is also partly supported by the HKU-TCL Joint Research Center for Artificial Intelligence fund (Project 200009430).
objective method method objective abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain other other other other other other other method method other other other other other method method method abstain abstain abstain abstain abstain abstain method abstain method abstain abstain method method abstain abstain method abstain method abstain abstain abstain method abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain method method abstain abstain method abstain abstain abstain abstain abstain method method abstain abstain method method method method method method abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain result abstain result abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain method method method method method method method abstain abstain abstain method method abstain method objective abstain method result other other