Knowledge-driven conversation approaches have achieved remarkable research attention recently. However, generating an informative response with multiple relevant knowledge without losing fluency and coherence is still one of the main challenges. To address this issue, this paper proposes a method that uses recurrent knowledge interaction among response decoding steps to incorporate appropriate knowledge. Furthermore, we introduce a knowledge copy mechanism using a knowledge-aware pointer network to copy words from external knowledge according to knowledge attention distribution. Our joint neural conversation model which integrates recurrent Knowledge-Interaction and knowledge C opy (KIC) performs well on generating informative responses. Experiments demonstrate that our model with fewer parameters yields significant improvements over competitive baselines on two datasets Wizard-of-Wikipedia(average Bleu +87%; abs.:0.034) and DuConv(average Bleu +20%; abs.:0.047) with different knowledge formats (textual & structured) and different languages (English & Chinese). Dialogue systems have attracted much research attention in recent years. Various end-to-end neural generative models based on the sequence-to-sequence framework (Sutskever et al., 2014) have been applied to the open-domain conversation and achieved impressive success in generating fluent dialog responses (Shang et al., 2015; Vinyals and Le, 2015; Serban et al., 2016). However, many neural generative approaches from the last few years confined within utterances and responses, suffering from generating uninformative and inappropriate responses. To make responses more meaningful and expressive, several works on the dialogue system exploiting external knowledge. Knowledge-driven methods focus on generating more informative and meaningful responses via incorporating structured knowledge consists of triplets (Zhu et al., 2017; Zhou et al., 2018; Young et al., 2018; Liu et al., 2018) or unstructured knowledge like documents (Long et al., 2017; Parthasarathi and Pineau, 2018; Ghazvininejad et al., 2018; Ye et al., 2019). Knowledge-based dialogue generation mainly has two methods: a pipeline way that deals with knowledge selection and generation successively (Lian et al., 2019), and a joint way that integrates knowledge selection into the generation process, for example, several works use Memory Network architectures (Sukhbaatar et al., 2015) to integrate the knowledge selection and generation jointly (Dinan et al., 2018; Dodge et al., 2015; Parthasarathi and Pineau, 2018; Madotto et al., 2018; Ghazvininejad et al., 2018). The pipeline approaches separate knowledge selection from generation, resulting in an insufficient fusion between knowledge and generator. When integrating various knowledge, pipeline approaches lack flexibility. The joint method with the memory module usually uses knowledge information statically. The confidence of knowledge attention decreasing at decoding steps, which has the potential to produce inappropriate collocation of knowledge words. To generate informative dialogue response that integrates various relevant knowledge without losing fluency and coherence, this paper presents an effective knowledge-based neural conversation model that enhances the incorporation between knowledge selection and generation to produce more informative and meaningful responses. Our model integrates the knowledge into the generator by using a recurrent knowledge interaction that dynamically updates the attentions of knowledge selection via decoder state and the updated knowledge attention assists in decoding the next state, which maintains the confidence of knowledge attention during the decoding process, it helps the decoder to fetch the latest knowledge information into the current decoding state. The generated words ameliorate the knowledge selection that refines the next word generation, and such repeated interaction between knowledge and generator is verified to be an effective way to integrate multiple knowledge coherently that to generate an informative and meaningful response when knowledge is fully taken account of. Although recurrent knowledge interaction better solves the problem of selecting appropriate knowledge for generating the informative response, the preferable integration of knowledge into conversation generation still confronts an issue, i.e., it is more likely that the description words from external knowledge generated for the dialog response have a high probability of being an oov(out-of-vocabulary), which is a common challenge in natural language processing. A neural generative model with pointer networks has been shown to have the ability to handle oov problems (Vinyals et al., 2015; Gu et al., 2016). Very few researches on copyable generative models pay attention to handle external knowledge, while in knowledge-driven conversation, the description words from knowledge are usually an important component of dialog response. Thus, we leverage a knowledge-aware pointer network upon recurrent knowledge interactive decoder, which integrates the Seq2seq model and pointer networks containing two pointers that refer to utterance attention distribution and knowledge attention distribution. We show that generating responses using the knowledge copy resolves the oov and the knowledge incompleteness problems. In summary, our main contributions are: (i) We propose a recurrent knowledge interaction, which chooses knowledge dynamically among decoding steps, integrating multiple knowledge into the response coherently. (ii) We use a knowledge-aware pointer network to do knowledge copy, which solves oov problem and keeps knowledge integrity, especially for long-text knowledge. (iii) The integration of recurrent knowledge interaction and knowledge copy results in more informative, coherent and fluent responses. (iv) Our comprehensive experiments show that our model is general for different knowledge formats (textual & structured) and different languages (English & Chinese). Furthermore, the results significantly outperform competitive baselines with fewer model parameters. Given a dataset D = { ( X i , Y i , K i ) } Ni =1 , where N is the size of the dataset, a dialog response Y = { y 1 , y 2 , . . . , y n } is produced by the conversation history utterance X = { x 1 , x 2 , . . . , x m } , using also the relative knowledge set K = { k 1 , k 2 , . . . , k s } . Here, m and n are the numbers of tokens in the conversation history X and response Y respectively, and s denotes the size of relevant knowledge candidates collection K . The relevant knowledge candidates collection K is assumed to be already provided and the size of candidates set is limited. Each relevant knowledge element in candidate collection could be a passage or a triplet, denoted as k = {  1 ,  2 , . . . ,  l } , where l is the number of the tokens in the knowledge element. As illustrated in Figure 1, the model KIC proposed in this work is based on an architecture involving an encoder-decoder framework (Sutskever et al., 2014) and a pointer network (Vinyals et al., 2015; See et al., 2017). Our model is comprised of four major components: (i) an LSTM based utterance encoder; (ii) a general knowledge encoder suitable for both structural and documental knowledge; (iii) a recurrent knowledge interactive decoder; (iv) a knowledge-aware pointer network. The utterance encoder uses a bi-directional LSTM (Schuster and Paliwal, 1997) to encode the utterance inputs by concatenating all tokens in the dialogue history X and obtain the bi-directional hidden state of each x i in utterance, denoted as H = { h 1 , h 2 , . . . , h m } . Combining two-directional hidden states, we have the hidden state h  t as h  t = [  LST M ( x t , h t  1 );  LST M ( x t , h t +1 )] . As illustrated in Model Description, the knowledge input is a collection of multiple knowledge candidates K . The relevant knowledge k i can be a passage or a triplet. This paper provides a universal encoding method for both textual and structured knowledge. The relevant knowledge is represented as a sequence of tokens, which are encoded by a transformer encoder (Vaswani et al., 2017), i.e., z t = T ransformer (  t ) . Static attention a ki is Figure 1: The architecture of KIC. used to encode knowledge Z = { z 1 , z 2 , . . . , z l } to obtain the overall representation K rep for the relevant knowledge as a ki = softmax ( V Tz tanh( W z z i )) (2) K rep = l (cid:88) i =1 a ki z i , (3) where V Tz and W z are learnable parameters. So far we have the knowledge representations for the knowledge candidate collection C repk . The decoder is mainly comprised of a single layer LSTM (Hochreiter and Schmidhuber, 1997) to generate dialogue response incorporating the knowledge representations in collection C repk . As shown in Figure 1, in each step t, the decoder updates its state s t +1 by utilizing the last decode state s t , current decode-input U td and knowledge context C tk . The current decode-input is computed by the em-beddings of the previous word e ( y t ) and utterance context vector C tu . We provide the procedure as e ti = v Te tanh( W h h i + W us s t + b ua ) (4) u t = softmax ( e t ) (5) C tu = m (cid:88) i =1 u ti h i (6) U td = V u [ e ( y t ) , C tu ] + b u , (7) where V u , b u , v e , W h , W us , b ua are learnable parameters. Instead of modeling knowledge selection independently, or statically incorporating the representation of knowledge into the generator, this paper proposes an interactive method to exploit knowledge in response generation recurrently. The knowledge attention d t updates as the decoding proceeds to consistently retrieve the information of the knowledge related to the current decoding step so that it helps decode the next state correctly, which writes as  ti = v Tk tanh( W k K repi + W ks s t + b ak ) (8) d t = softmax (  t ) (9) C tk = s (cid:88) i d ti K repi , (10) where v k , W k , W ks , b ak are learnable parameters. A knowledge gate g t is employed to determine how much knowledge and decode-input is used in the generation, which is defined as g t = sigmoid ( V g [ U td , C tk ] + b g ) , (11) where V g and b g are learnable parameters. As the steps proceed recurrently, the knowledge gate can dynamically update itself as well. Hence, the decoder updates its state as: s t +1 = LST M ( s t , ( g t U td + (1  g t ) C tk )) (12) 2.4 Knowledge-Aware Pointer Networks Pointer networks using a copy mechanism are widely used in generative models to deal with oov problem. This paper employs a novel knowledge-aware pointer network. Specifically, we expand the scope of the original pointer networks by exploiting the attention distribution of knowledge representation. Besides, the proposed knowledge-aware pointer network shares extended vocabulary between utterance and knowledge that is beneficial to decode oov words. As two pointers respectively refer to the attention distributions of utterance and knowledge, each word generation is determined by the soft switch of utterance u gen and the soft switch of knowledge k gen , which are defined as u gen =  ( w Tuc C tu + w Tus s t + w Tu U td + b up ) (13) k gen =  ( w Tkc C tk + w Tks s t + w Tg U tg + b kp ) , (14) where w Tuc , w T us , w Tu , b up , w T kc , w Tks , w T g , b kp are learnable parameters. The U tg here is defined as U tg = V g [ e ( y t ) , C tk ] + b g , (15) where V g , b g are learnable parameters. Therefore, the final probability of the vocabulary w is P final ( w ) = ( u gen + k gen ) P v ( w )+  (1  u gen ) (cid:88) i u ti +  (1  k gen ) (cid:88) i d ti , (16) P v ( w ) = softmax ( V 2 ( V 1 [ s t , C tu , C tk ] + b 1 ) + b 2 ) , (17) where V 1 , V 2 , b 1 , b 2 ,  and  are learnable parameters under constrain  +  = 1 . Note that if the word is an oov word and does not appear in utterance, P v ( w ) is zero and we copy words from knowledge instead of dialogue history. We use two recently released datasets Wizard-of-Wikipedia and DuConv, whose knowledge formats are sentences and triplets respectively. Wizard-of-Wikipedia (Dinan et al., 2018): an open-domain chit-chat dataset between agent wizard and apprentice. Wizard is a knowledge expert who can access any information retrieval system recalling paragraphs from Wikipedia relevant to the dialogue, which unobserved by the agent apprentice who plays a role as a curious learner. The dataset contains 22311 dialogues with 201999 turns, 166787/17715/17497 used for train/valid/test, and the test set is split into two subsets, Test Seen(8715) and Test Unseen(8782). Test Seen has 533 overlapping topics with the training set; Test Unseen contains 58 topics never seen before in train or validation. We do not use the ground-truth knowledge information provided in this dataset because the ability of knowledge selection during generation is a crucial part of our model. DuConv (Wu et al., 2019b): a proactive conversation dataset with 29858 dialogs and 270399 utterances. The model mainly plays the role of a leading player assigned with an explicit goal, a knowledge path comprised of two topics, and is provided with knowledge related to these two topics. The knowledge in this dataset is a format of the triplet(subject, property, object), which totally contains about 144k entities and 45 properties. We implement our model both on datasets Wizard-of-Wikipedia and DuConv, and compare our approach with a variety of recently competitive baselines in these datasets, respectively. In Wizard-of-Wikipedia, we compare the approaches as follows:  Seq2Seq: an attention-based Seq2Seq without access to external knowledge which is widely used in open-domain dialogue. (Vinyals and Le, 2015)  MemNet(hard/soft): a knowledge grounded generation model, where knowledge candidates are selected with semantic similar-ity(hard); / knowledge candidates are stored into the memory units for generation (soft). (Ghazvininejad et al., 2018)  PostKS(concat/fusion): a hard knowledge grounded model with a GRU decoder where knowledge is concatenated (concat); / a soft model use HGFU to incorporated knowledges with a GRU decoder.(Lian et al., 2019)  KIC: Our joint neural conversation model named knowledge-aware pointer networks and recurrent knowledge interaction hybrid generator. While in dataset DuConv, a Chinese dialogue dataset with structured knowledge, we compare to the baselines referred in (Wu et al., 2019b) that consists of retrieval-based models as well as generation-based models. We adopt an automatic evaluation with several common metrics proposed by (Wu et al., 2019b; Lian et al., 2019) and use their available automatic evaluation tool to calculate the experimental results to keep the same standards. Metrics include Bleu1/2/3, F1, DISTINCT1/2 automatically measure the fluency, coherence, relevance, diversity, etc. Metric F1 evaluates the performance at the character level, which mainly uses in Chinese dataset DuConv. Our method incorporates generation with knowledge via soft fusion that does not select knowledge explicitly, therefore we just measure the results of the whole dialog while not evaluate performances of knowledge selection independently. Besides, we provide 3 annotators to evaluate the results on a human level. The annotators evaluate the quality of dialog response generated on fluency, informativeness, and coherence. The score ranges from 0 to 2 to reflect the fluency, informativeness, and coherence of results from bad to good. For example, of coherence , score 2 means the response with good coherence without illogical expression and continues the dialogue history reasonably; score 1 means the result is acceptable but with a slight flaw; score 0 means the statement of result illogically or the result improper to the dialog context. We implement our model over Tensorflow frame-work(Abadi et al., 2016). And our implementation of point networks is inspired by the public code provided by (See et al., 2017). The utterance sequence concats the tokens of dialog history and separated knowledge. And the utterance encoder has a single-layer bidirectional LSTM structure with 256 hidden states while the response decoder has a single-layer unidirectional LSTM structure with the same dimensional hidden states. And the knowledge encoder has a 2-layer transformer structure. We use a vocabulary of 50k words with 128 dimensional random initialized embed-dings instead of using pre-trained word embed-dings. We train our model using Adagrad (Duchi et al., 2011) optimizer with a mini-batch size of 128 and learning rate 0.1 at most 130k iterations(70k iterations on Wizard-of-Wikipedia) on a GPU-P100 machine. The overall parameters are about 44 million and the model size is about 175MB, which decreases about 38% against the overall best baseline PostKS(parameters:71 million, model size: 285M) 3.5 Results and Analysis 3.5.1 Automatic Evaluation As the experimental results on Wizard-of-Wikipedia with automatic evaluation summarized in Table 1, our approach outperforms all competitive baseline referred to recently working (Lian et al., 2019), and achieves significant improvements over most of the automatic metrics both on Seen and Unseen Test sets. The Bleu-1 enhances slightly in Test Seen while improving obviously in Test Unseen. Bleu-2 and Bleu-3 both yield considerable increments not only in Test Seen but in Test Unseen as well, for example, the Bleu-3 improves about 126% (absolute improvement: 0.043) in Test Seen and about 234%(absolute improvement: 0.047) in Test Unseen. The superior performance on metrics Bleu means the dialog response generated by model KIC is closer to the ground-truth response and with preferable fluency. As all Figure 2: Bleu improvements on Wizard-of-Wikipedia. Bleu metrics are shown in Figure 2, we can find that the improvement of result increasing with the augment of Bleu's grams, which means the dialog response produced via model KIC is more in line with the real distribution of ground-truth response in the phrase level, and the better improvement on higher gram's Bleu reflects the model have preferable readability and fluency. Generally, the ground-truth responses in datasets make up with the expressions from knowledge which conduces to the informativeness of response. As the recurrent knowledge interaction module in model KIC provides a mechanism to interact with the knowledge when decoding words of dialog response step by step. Moreover, the knowledge-aware pointer Models Test Seen Test Unseen Bleu-1/2/3 DISTINCT-1/2 Bleu-1/2/3 DISTINCT-1/2 Seq2Seq 0.169/0.066/0.032 0.036/0.112 0.150/0.054/0.026 0.020/0.063 MemNet(hard) 0.159/0.062/0.029 0.043/0.138 0.142/0.042/0.015 0.029/0.088 MemNet(soft) 0.168/0.067/0.034 0.037/0.115 0.148/0.048/0.023 0.026/0.081 PostKS(concat) 0.167/0.066/0.032 0.056/0.209 0.144/0.043/0.016 0.040/0.151 PostKS(fusion) 0.172/0.069/0.034 0.056/0.213 0.147/0.046/0.021 0.040/0.156 KIC(ours) 0.173/0.105/0.077 0.138/0.363 0.165/0.095/0.068 0.072/0.174 Table 1: Automatic Evaluation on Wizard-of-Wikipedia. network in KIC allows copying words from the expression of knowledge while decoding. Therefore, the dialog response generated by KIC contains relatively complete phrases of knowledge that as knowledge-informativeness as the ground-truth response. In addition, the improvements of metrics Bleu increase from Test Seen to Test Unseen, that is to say, the KIC with an advantage in case of unseen knowledge guided dialogue, which shows that our model is superior to address the dialogues with topics never seen before in train or validation. Besides, the metrics DISTINCT also achieves impressive results and prior than most of the baselines, about average 77% over the most competitive method PostKS. The metrics DISTINCT mainly reflects the diversity of generated words, whose improvements indicating that the dialogue response produced by KIC could present more information. In addition to experiments on Wizard-of-Wikipedia, we also conduct experiments on DuConv to further verify the effectiveness of our model on structured knowledge incorporated conversation. As the dataset DuConv released most recently that we compare our model to the baselines mentioned in the (Wu et al., 2019b) which are first applied to the DuConv including both retrieval-based and generation-based methods. The results presented in Table 2 show that our model obtains the highest results in most of the metrics with obvious improvement over retrieval and generation methods. Concretely, the F1, average Bleu, average DISTINCT, and ppl are over the best results of baseline norm generation about 6.6%, 20.5%, 115.8%, and 5.5%. Similar to Wizard-of-Wikipedia, the impressive augments of metrics demonstrate that the model has the capacity of producing appropriate responses with fluency, coherence, and diversity. In human evaluation, according to the dialogue history and the related knowledge, the annotators evaluate the quality dialog responses in terms of fluency and coherence. The score ranges from 0 to 2; the score is as higher as the responses are more fluent, informative, and coherent to the dialog context and integrate more knowledge. Manual evaluation results are summarized in Table 3, the model achieves high scores both in Wizard-of-Wikipedia and DuConv, meaning that the responses generated by KIC also with good fluency, informativeness, Models F1 Bleu-1 Bleu-2 DISTINCT1 DISTINCT2 Parameters Part1: seq2seq w/o klg. and coherence in human view, close to the superior performance of automatic evaluation. We conduct further ablation experiments to dissect our model. Based on the Seq2Seq framework, we aggrandize it with each key component of model KIC progressively and the results are summarized in Table 4 and Table 5. We first incorporate knowledge into Seq2Seq architecture with dot attention of knowledge and use a gate to control the utilization of knowledge during generation, and the results achieve considerable improvement with the help of knowledge. And then, we apply knowledge-aware pointer networks over the model illustrated in last step to introduce a copy mechanism, which increases effect significantly demonstrates the facilitation of knowledge-aware copy mechanism to produce dialogue response with important words adopted from utterance and knowledge. In the end, we replace the knowledge dot attention by dynamic attention updated with decode state recurrently, which is the whole KIC model proposed in this paper, and the experimental results show that such amelioration also achieves an impressive enhancement. The dynamic update of knowledge attention during decoding effectively integrates multiple knowledge into the response that improves the informativeness. The performances of the model are gradually improved with the addition of components, meaning that each key component of the model KIC plays a crucial role. Additionally, with the considerable improvement at each progressive step, the model size and the parameters just increase slightly, which means the model KIC has a good cost performance. As shown in Figure 3, we present the responses generated by our proposed model KIC and the model PostKS(fusion), which achieves overall best performance among competitive baselines. Given utterance and knowledge candidates, our model is better than PostKS(fusion) to produce context-coherence responses incorporating appropriate multiple knowledge with complete descriptions. The model KIC prefers to integrate more knowledge into dialogue response, riching the informative without losing fluency. Furthermore, our model has an additional capability of handling oov problem, which can generate responses with infrequent but important words (which are oov words most of the time) from the knowledge context, like the Alfred Hitchcock Presents in Figure 3. We also compare to the result of the model with static knowledge attention, whose result mismatches between the award and the representative work Alfred Hitchcock Presents. The static knowledge attention calculated before decoding, the information and confidence losing with the decoding step by step, leading to mispairing the expression of multiple knowledge. While the recurrent knowledge interaction helps the decoder to fetch the closest knowledge information into the current decoding Figure 3: Case study of DuConv. state, which superior to learn the coherent collocation of multiple knowledge. Some more cases of Wizard-of-Wikipedia and DuConv will present in the appendix section. Conversation with knowledge incorporation has received considerable interest recently and is demonstrated to be an effective way to enhance performance. There are two main methods in knowledge-based conversation, retrieval-based approches(Wu et al., 2016; Tian et al., 2019) and generation-based approaches. The generation-based method which achieves more research attention focuses on generating more informative and meaningful responses via incorporate generation with structured knowledge (Zhu et al., 2017; Liu et al., 2018; Young et al., 2018; Zhou et al., 2018) or documental knowl-edge(Ghazvininejad et al., 2018; Long et al., 2017). Several works integrate knowledge and generation in the pipeline way, which deal with knowledge selection and generation separately. Pipeline approaches pay more attention to knowledge selection, such as using posterior knowledge distribution to facilitate knowledge selection (Lian et al., 2019; Wu et al., 2019b) or used context-aware knowledge pre-selection to guide select knowledge (Zhang et al., 2019). While various works entirety integration the knowledge with generation in an end-to-end way, which usually manage knowledge via external memory module. (Parthasarathi and Pineau, 2018) introduced a bag-of-words memory network and (Dodge et al., 2015) performed dialogue discussion with long-term memory. (Dinan et al., 2018) used a memory network to retrieve knowledge and combined with transformer architectures to generate responses. The pipeline approaches lack of flexibility as constricted by the separated knowledge selection, and the generation could not exploit knowledge sufficiently. The end-to-end approaches with memory module attention to knowledge statically, when integrating multiple knowledge into a response are easier to be confused. Whereas we provide a recurrent knowledge interactive generator that sufficiently fusing the knowledge into generation to produce more informative dialogue responses. Our work is also inspired by several works of text generation using copy mechanisms. (Vinyals et al., 2015) used attention as a pointer to generate words from the input resource by index-based copy. (Gu et al., 2016) incorporated copying into seq2seq learning to handle unknown words. (See et al., 2017) introduced a hybrid pointer-generator that can copy words from the source text while retaining the ability to produce novel words. In task-oriented dialogue, the pointer networks were also used to improve copy accuracy and mitigate the common out-of-vocabulary problem (Madotto et al., 2018; Wu et al., 2019a). Different from these works, we extend a pointer network referring to attention distribution of knowledge candidates that can copy words from knowledge resources and generate dialogue responses under the guidance of more complete description from knowledge. We propose a knowledge grounded conversational model with a recurrent knowledge interactive generator that effectively exploits multiple relevant knowledge to produce appropriate responses. Meanwhile, the knowledge-aware pointer networks we designed allow copying important words, usually oov words, from knowledge. Experimental results demonstrate that our model is powerful to generate much more informative and coherent responses than the competitive baseline models. In future work, we plan to analyze each turn of dialogue with reinforcement learning architecture, and to enhance the diversity of the whole dialogue by avoiding knowledge reuse.
abstain abstain objective method method objective abstain abstain abstain abstain other other abstain abstain abstain abstain method method abstain abstain abstain abstain abstain result objective objective objective abstain result abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain other abstain abstain abstain abstain abstain abstain method abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain method method abstain abstain abstain result result abstain abstain abstain abstain result abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain other other other other other other other other other other method method other other other other objective objective method objective method