Recently, many studies are emerging towards building a retrieval-based dialogue system that is able to effectively leverage background knowledge (e.g., documents) when conversing with humans. However, it is non-trivial to collect large-scale dialogues that are naturally grounded on the background documents, which hinders the effective and adequate training of knowledge selection and response matching. To overcome the challenge, we consider decomposing the training of the knowledge-grounded response selection into three tasks including: 1) query-passage matching task; 2) query-dialogue history matching task; 3) multi-turn response matching task, and joint learning all these tasks in a unified pre-trained language model. The former two tasks could help the model in knowledge selection and comprehension, while the last task is designed for matching the proper response with the given query and background knowledge (dialogue history). By this means, the model can be learned to select relevant knowledge and distinguish proper response, with the help of ad-hoc retrieval corpora and a large number of ungrounded multi-turn dialogues. Experimental results on two benchmarks of knowledge-grounded response selection indicate that our model can achieve comparable performance with several existing methods that rely on crowd-sourced data for training. Along with the very recent prosperity of artificial intelligence empowered conversation systems in the spotlight, many studies have been focused on building human-computer dialogue systems (Wen et al., 2017; Zhang et al., 2020) with either retrieval-based methods (Wang et al., 2013; Wu et al., 2017; Whang et al., 2020) or generation-based methods (Li et al., 2016; Serban et al., 2016; Zhang et al., 2020), which both predict the response with only the given context. In fact, unlike a person who may associate the conversation with the background knowledge in his or her mind, the machine can only capture limited information from the query message itself. As a result, it is difficult for a machine to properly comprehend the query, and to predict a proper response to make it more engaging. To bridge the gap of the knowledge between the human and the machine, researchers have begun to simulating this motivation by grounding dialogue agents with background knowledge (Zhang et al., 2018; Dinan et al., 2019; Li et al., 2020), and lots of impressive results have been obtained. In this paper, we consider the response selection problem in knowledge-grounded conversion and specify the background knowledge as unstructured documents that are common sources in practice. The task is that given a conversation context and a set of knowledge entries, one is required 1): to select proper knowledge and grasp a good comprehension of the selected document materials (knowledge selection); 2): to distinguish the true response from a candidate pool that is relevant and consistent with both the conversation context and the background documents (knowledge matching). While there exists a number of knowledge documents on the Web, it is non-trivial to collect large-scale dialogues that are naturally grounded on the documents for training a neural response selection model, which hinders the effective and adequate training of knowledge selection and response matching. Although some benchmarks built upon crowd-sourcing have been released by recent works (Zhang et al., 2018; Dinan et al., 2019), the relatively small training size makes it hard for the dialogue models to generalize on other domains or topics (Zhao et al., 2020). Thus, in this work, we focus on a more challenging and practical scenario, learning a knowledge-grounded conversation agent without any knowledge-grounded dialogue data, which is known as zero-resource settings. Since knowledge-grounded dialogues are unavailable in training, it raises greater challenges for learning the grounded response selection model. Fortunately, there exists a large number of unstructured knowledge (e.g., web pages or wiki articles), passage search datasets (e.g., query-passage pairs coming from ad-hoc retrieval tasks) (Khattab and Zaharia, 2020) and multi-turn dialogues (e.g., context-response pairs collected from Reddit) (Hen-derson et al., 2019), which might be beneficial to the learning of knowledge comprehension, knowledge selection and response prediction respectively. Besides, in multi-turn dialogues, the background knowledge and conversation history (excluding the latest query) are symmetric in terms of the information they convey, and we assume that the dialogue history can be regarded as another format of background knowledge for response prediction. Based on the above intuition, in this paper, we consider decomposing the training of the grounded response selection task into several sub-tasks, and joint learning all those tasks in a unified model. To take advantage of the recent breakthrough on pretraining for natural language tasks, we build the grounded response matching models on the basis of a pre-trained language model (PLMs) (Devlin et al., 2019; Yang et al., 2019), which are trained with large-scale unstructured documents from the web. On this basis, we further train the PLMs with query-passage matching task, query-dialogue history matching task, and multi-turn response matching task jointly. The former two tasks could help the model not only in knowledge selection but also in knowledge (and dialogue history) comprehension, while the last task is designed for matching the proper response with the given query and background knowledge (dialogue history). By this means, the model can be learned to select relevant knowledge and distinguish proper responses, with the help of a large number of ungrounded dialogues and ad-hoc retrieval corpora. During the testing stage, we first utilize the trained model to select proper knowledge, and then feed the query, dialogue history, selected knowledge, and the response candidate into our model to calculate the final matching degree. Particularly, we design two strategies to compute the final matching score. In the first strategy, we directly concatenate the selected knowledge and dialogue history as a long sequence of background knowledge and feed into the model. In the second strategy, we first compute the matching degree between each query-knowledge and the response candidates, and then integrate all matching scores. We conduct experiments with benchmarks of knowledge-grounded dialogue that are constructed by crowd-sourcing, such as the Wizard-of-Wikipedia Corpus (Dinan et al., 2019) and the CMU DoG Corpus (Zhou et al., 2018a). Evaluation results indicate that our model achieves comparable performance on knowledge selection and response selection with several existing models trained on crowd-sourced benchmarks. Our contributions are summarized as follows:  To the best of our knowledge, this is the first exploration of knowledge-grounded response selection under the zero-resource setting. We propose decomposing the training of the grounded response selection models into several sub-tasks, so as to empower the model through these tasks in knowledge selection and response matching. We achieve a comparable performance of response selection with several existing models learned from crowd-sourced training sets. Early studies of retrieval-based dialogue focus on single-turn response selection where the input of a matching model is a message-response pair (Wang et al., 2013; Ji et al., 2014; Wang et al., 2015). Recently, researchers pay more attention to multiturn context-response matching and usually adopt the representation-matching-aggregation paradigm to build the model. Representative methods include the dual-LSTM model (Lowe et al., 2015), the sequential matching network (SMN) (Wu et al., 2017), the deep attention matching network (DAM) (Zhou et al., 2018b), interaction-over-interaction network (IoI) (Tao et al., 2019) and multi-hop selector network (MSN) (Yuan et al., 2019). More recently, pre-trained language models (Devlin et al., 2019; Yang et al., 2019) have shown significant benefits for various NLP tasks, and some researchers have tried to apply them on multi-turn response selection. Vig and Ramea (2019) exploit BERT to represent each utterance-response pair and fuse these representations to calculate the matching score; Whang et al. (2020) and Xu et al. (2020) treat the context as a long sequence and conduct context-response matching with BERT. Besides, Gu et al. (2020a) integrate speaker embeddings into BERT to improve the utterance representation in multi-turn dialogue. To bridge the gap of the knowledge between the human and the machine, researchers have investigated into grounding dialogue agents with unstructured background knowledge (Ghazvininejad et al., 2018; Zhang et al., 2018; Dinan et al., 2019). For example, Zhang et al. (2018) build a persona-based conversation data set that employs the interlocu-tor's profile as the background knowledge; Zhou et al. (2018a) publish a data where conversations are grounded in articles about popular movies; Dinan et al. (2019) release another document-grounded data with Wiki articles covering a wide range of topics. Meanwhile, several retrieval-based knowledge-grounded dialogue models are proposed, such as document-grounded matching network (DGMN) (Zhao et al., 2019) and dually interactive matching network (DIM) (Gu et al., 2019) which let the dialogue context and all knowledge entries interact with the response candidate respectively via the cross-attention mechanism. Gu et al. (2020b) further propose to pre-filter the context and the knowledge and then use the filtered context and knowledge to perform the matching with the response. Besides, with the help of gold knowledge index annotated by human wizards, Dinan et al. (2019) consider joint learning the knowledge selection and response matching in a multi-task manner or training a two-stage model. In this section, we first formalize the knowledge-grounded response matching problem and then introduce our method from preliminary to response matching with PLMs to details of three pre-training tasks. We first describe a standard knowledge-grounded response selection task such as Wizard-of-Wikipedia. Suppose that we have a knowledge-grounded dialogue data set D = { k i , c i , r i , y i } Ni =1 where k i = { p 1 , p 2 , . . . , p l k } represents a collection of knowledge with p j the j -th knowledge entry (a.k.a., passage) and l k is the number of entries; c i = { u 1 , u 2 , . . . , u l c } denotes multi-turn dialogue context with u j the j -th turn and l c is the number of dialogue turns. It should be noted that in this paper we denote the latest turn u l c as dialogue query q i , and dialogue context except for query is denoted as h i = c i / { q i } . r i stands for a candidate response. y i = 1 indicates that r i is a proper response for c i and k i , otherwise y i = 0 . N is the number of samples in data set. The goal knowledge-grounded dialogue is to learn a matching model g ( k, c, r ) from D , and thus for any new ( k, c, r ) , g ( k, c, r ) returns the matching degree between r and ( k, c ) . Finally, one can collect the matching scores of a series of candidate responses and conduct response ranking. Zero-resource grounded response selection then is formally defined as follows. There is a standard multi-turn dialogue dataset D c = { q i , h i , r i } Ni =1 and an ad-hoc retrieval dataset D p = { q i , p i , z i } Mi =1 where q i is a query and p i stands a candidate passage, z i = 1 indicates that p i is a relevant passage for q i , otherwise z i = 0 . Our goal is to learn a model g ( k, h, q, r ) from D c and D p , and thus for any new input ( k, h, q, r ) , our model can select proper knowledge  k from k and calculate the matching degree between r and ( k, q, h ) . Pre-trained language models have been widely used in many NLP tasks due to the strong ability of language representation and understanding. In this work, we consider building a knowledge-grounded response matching model with BERT. Specifically, given a query q , a dialogue history h = { u 1 , u 2 , ..., u n h } where u i is the i -th turn in the history, a response candidate r = { r 1 , r 2 , ..., r l r } with l r words, we concatenate all sequences as a single consecutive tokens sequence with special tokens, which can be represented as x = { [ CLS ] , u 1 , [ SEP ] , . . . , [ SEP ] , u l h , [ SEP ] , q, [ SEP ] , r, [ SEP ] } . [ CLS ] and [ SEP ] are classification symbol and segment separation symbol respectively. For each token in x , BERT uses a summation of three kinds of embeddings, including WordPiece embedding (Wu et al., 2016), segment embedding, and position embedding. Then, the embedding sequence of x is fed into BERT, giving us the contextualized embedding sequence { E [CLS] , E 2 , . . . , E l x } . E [CLS] is an aggregated representation vector that contains the Input  Dialogue History or Knowledge Response Pre-trained Language Model (BERT)  ,, OutputLayer MLP Token Embeddings Position Embeddings Segment Embeddings     Response Matching Task Query Query-Dialogue History Matching Task Query-Passage Matching Task  [Background Knowledge] [Response] [Query]        !"#  $ !  #%&  $ "  #%&  $ #  #%&  '  #%&  ( ! semantic interaction information between the query, history, and response candidate. Finaly, E [CLS] is fed into a non-linear layer to calculate the final matching score, which is formulated as: g ( h, q, r ) =  ( W 2  tanh( W 1 E [ CLS ] + b 1 ) + b 2 ) (1) where W { 1 , 2 } and b { 1 , 2 } is training parameters for response selection task,  is a sigmoid function. In knowledge-grounded dialogue, each dialogue is associated with a large collection of knowledge entries k = { p 1 , p 2 , . . . , p l k } 1 . The model is required to select m ( m  1) knowledge entries based on semantic relevance between the query and each knowledge, and then performs the response matching with the query, dialogue history and the highly-relevant knowledge. Specifically, we denote  k = ( p 1 , . . . ,  p m ) as the selected knowledge entries, and feed the input sequence x = { [ CLS ] ,  p 1 , [ SEP ] , . . . , [ SEP ] ,  p m , [ SEP ] , u 1 , [ SEP ] , . . . , [ SEP ] , u l h , [ SEP ] , q, [ SEP ] , r, [ SEP ] } to BERT. The final matching score g ( k, h, q, r ) can be computed based on [ CLS ] representation. On the basis of BERT, we further jointly train it with three tasks including 1) query-passage matching task ; 2) query-dialogue history matching task ; 3) multi-turn response matching task . The former two tasks could help the model in knowledge selection and knowledge (and dialogue history) comprehension, while the last task is designed for matching the proper response with the given query and background knowledge (dialogue 1 The scale of the knowledge referenced by each dialogue usually exceeds the limitation of input length in PLMs. history). By this means, the model can be learned to select relevant knowledge and distinguish the proper response, with the help of a large number of ungrounded dialogues and ad-hoc retrieval corpora. Although there exist a huge amount of conversation data on social media, it is hard to collect sufficient dialogues that are naturally grounded on knowledge documents. Existing studies (Dinan et al., 2019) usually extract the relevant knowledge before the response matching or jointly train the knowledge retrieval and response selection in a multi-task manner. However, both methods need in-domain knowledge-grounded dialogue data (with gold knowledge label) to train, making the model hard to generalize to a new domain. Fortunately, the ad-hoc retrieval task (Harman, 2005; Khattab and Zaharia, 2020) in the information retrieval area provides a potential solution to simulate the process of knowledge seeking. To take advantage of the parallel data in the ad-hoc retrieval task, we consider incorporating the query-passage matching task, so as to help the knowledge selection and knowledge comprehension for our task. Given a query-passage pair ( q, p ) , we first concatenate the query q and the passage p as a single consecutive token sequence with special tokens separating them, which is formulated as: S qp = { [CLS] , w p 1 , . . . , w pn p , [SEP] , w q 1 , . . . , w qn q } (2) where w pi , w qj denotes the i -th and j -th token of knowledge entry p and query q respectively. For each token in S qpi , token , segment and position embeddings are summated and fed into BERT. It is worth noting that here we set the segment embedding of the knowledge to be the same as the dialogue history. Finally, we feed the output representation of [ CLS ] E qp[CLS] into a MLP to obtain the final query-passage matching score g ( q, p ) . The loss function of each training sample for query-passage matching task is defined by L p ( q, p + , p  1 , . . . , p  n p ) =  log( e g ( q,p + ) e g ( q,p + ) + (cid:80)  p j =1 e g ( q,p  j ) ) (3) where p + stands for the positive passage for q , p  j is the j -th negative passage and  p is the number of negative passage. In multi-turn dialogues, the conversation history (excluding the latest query) is a piece of supplementary information for the current query and can be regarded as another format of background knowledge during the response matching. Besides, due to the natural sequential relationship between dialogue turns, the dialogue query usually shows a strong semantic relevance with the previous turns in the dialogue history. Inspired by such characteristics, we design a query-dialogue history matching task with the multi-turn dialogue context, so as to enhance the capability of the model to comprehend the dialogue history with the given dialogue query and to rank relevant passages with these pseudo query-passage pairs. Specifically, we first concatenate the dialogue history into a long sequence. The task requires the model to predict whether a query q = { w q 1 , . . . , w qn q } and a dialogue history sequence h = { w h 1 , . . . , w hn h } are consecutive and relevant. We concatenate two sequences into a single consecutive sequence with [ SEP ] tokens, S qh = { [CLS] , w h 1 , . . . , w hn h , [SEP] , w q 1 , . . . , w qn q } (4) For each word in S qh , token , segment and position embeddings are summated and fed into BERT. Finally, we feed E qh[CLS] into a MLP to obtain the final query-history matching score g ( q, h ) . The loss function of each training sample for query-history matching task is defined by L h ( q, h + , h  1 , . . . , h  n h ) =  log( e g ( q,h + ) e g ( q,h + ) + (cid:80)  h j =1 e g ( q,h  j ) ) (5) where h + stands for the true dialogue history for q , h  j is the j -th negative dialogue history randomly sampled from the training set and  h is the number of sampled dialogue history. The above two tasks are designed for empowering the model to knowledge or history comprehension and knowledge selection. In this task, we aim at training the model to match reasonable responses based on dialogue history and query. Since we treat the dialogue history as a special form of background knowledge and they share the same segment embeddings in the PLMs, our model can acquire the ability to identify the proper response with either dialogue history or the background knowledge through the multi-turn response matching task. Specifically, we format the multi-turn dialogues as query-history-response triples and requires the model to predict whether a response candidate r = { w r 1 , . . . , w rn r } is appropriate for a given query q = { w q 1 , . . . , w qn q } and a concatenated dialogue history sequence h = { w h 1 , . . . , w hn h } . Concretely, we concatenate three input sequences into a single consecutive tokens sequence with [ SEP ] tokens, S hqr = { [CLS] , w h 1 , . . . , w hn h , [SEP] , w q 1 , . . . , w qn q , [SEP] , w r 1 , . . . , w rn r } (6) Similarly, we feed an embedding sequence of which each entry is a summation of token , segment and position embeddings into BERT. Finally, we feed E hqr[CLS] into a MLP to obtain the final response matching score g ( h, q, r ) . L r ( h, q, r + , r  1 , . . . , r   r ) =  log( e g ( h,q,r + ) e g ( h,q,r + ) + (cid:80) n r i = j e g ( h,q,r  j ) ) (7) where r + is the true response for a given q and h , r  j is the j -th negative response candidate randomly sampled from the training set and  r is the number of negative response candidate. L final = L p + L h + L r After learning model from D c and D p , we first rank { p i } n k i =1 according to g ( q, k i ) and then select top m knowledge entries { p 1 , . . . , p m } for the subsequent response matching process. Here we design two strategies to compute the final matching score g ( k, h, q, r ) . In the first strategy, we directly concatenate the selected knowledge and dialogue history as a long sequence of background knowledge and feed into the model to obtain the final matching score, which is formulated as, g ( k, h, q, r ) = g ( p 1  . . .  p m  c, q, r ) (9) where  denotes the concatenation operation. In the second strategy, we treat each selected knowledge entry and the dialogue history equally as the background knowledge, and compute the matching degree between each query, background knowledge, and the response candidates with the trained model. Consequently, the matching score is defined as an integration of a set of knowledge-grounded response matching scores, formulated as, g ( k, h, q, r ) = g ( h, q, r )+ max i  (0 ,m ) g ( p i , q, r ) (10) where m is the number of selected knowledge entries. We name our model with the two strategies as PTKGC cat and PTKGC sep respectively. We compare the two learning strategies through empirical studies, as will be reported in the next section. Training Set. We adopt MS MARCO passage ranking dataset (Nguyen et al., 2016) built on Bing's search for query-passage matching task. The dataset contains 8 . 8 M passages from Web pages gathered from Bing's results to real-world queries and each passage contains an average of 55 words. Each query is associated with sparse relevance judgments of one (or very few) passage marked as relevant. The training set contains about 500 k pairs of query and relevant passage, and another 400M pairs of query and passages that have not been marked as relevant, from which the negatives are sampled in our task. For the query-dialogue history matching task and multi-turn response matching task, we use the multi-turn dialogue corpus constructed from the Reddit (Dziri et al., 2018). The dataset contains more than 15 million dialogues and each dialogue has at least 3 utterances. After the pre-processing, we randomly sample 2 . 28 M/ 20 K dialogues as the training/validation set. For each dialogue session, we regard the last turn as the response, the last but one as the query, and the rest as the positive dialogue history. The negative dialogue histories are randomly sampled from the whole dialogue set. On average, each dialogue contains 4 . 3 utterances, and the average length of the utterances is 42 . 5 . Test Set. We tested our proposed method on the Wizard-of-Wikipedia (WoW) (Dinan et al., 2019) and CMU DoG (Zhou et al., 2018a). Both datasets contain multi-turn dialogues grounded on a set of background knowledge and are built with crowd-sourcing on Amazon Mechanical Turk. In WoW, the given knowledge collection is obtained from Wikipedia and covers a wide range of topics or domains, while in CMU DoG, the underlying knowledge focuses on the movie domain. Unlike CMU DoG where the golden knowledge index for each turn is unknown, the golden knowledge index for each turn is provided in WoW. Two configurations (e.g., test-seen and test-unseen) are provided in WoW. Following existing works (Dinan et al., 2019; Zhao et al., 2019), positive responses are true responses from humans and negative ones are randomly sampled. The ratio between positive and negative responses is 1 : 99 for WoW and 1 : 19 for CMU DoG. More details of the two benchmarks are shown in Appendix A.1. Evaluation Metrics. Following previous works on knowledge-grounded response selection (Gu et al., 2020b; Zhao et al., 2019), we also employ recall n at k R n @k (where n = 100 for WoW and n = 20 for CMU DoG and k = { 1 , 2 , 5 } ) as the evaluation metrics. Our model is implemented by PyTorch (Paszke et al., 2019). Without loss of generality, we select English uncased BERT base (110M) as the matching model. During the training, the maximum lengths of the knowledge (a.k.a., passage), the dialogue history, the query, and the response candidate were set to 128 , 120 60 , and 40 . Intuitively, the last tokens in the dialogue history and the previous Models Test Seen Test Unseen R@1 R@2 R@5 R@1 R@2 R@5 IR Baseline 17.8 -14.2 -BoW MemNet 71.3 -33.1 -Two-stage Transformer 84.2 -63.1 -Transformer MemNet 87.4 -69.8 -DIM (Gu et al., 2019) 83.1 91.1 95.7 60.3 77.8 92.3 FIRE (Gu et al., 2020b) 88.3 95.3 97.7 68.3 84.5 95.1 PTKGC cat 85.7 94.6 98.2 65.5 82.0 94.7 PTKGC sep 89.5 96.7 98.9 69.6 85.8 96.3 Table 1: Evaluation results on the test set of WoW. tokens in the query and response candidate are more important, so we cut off the previous tokens for the context but do the cut-off in the reverse direction for the query and response candidate if the sequences are longer than the maximum length. We set a batch size of 32 for multi-turn response matching and query-dialogue history matching, and 8 for query-document matching in order to train these tasks jointly under the circumstance of training examples inequality. We set  p = 6 ,  h = 1 and  r = 12 for the query-passage matching, the query-dialogue history matching and the multiturn response matching respectively. Particularly, the negative dialogue histories are sampled from other training instances in a batch. The model is optimized using Adam optimizer with a learning rate set as 5 e  6 . The learning rate is scheduled by warmup and linear decay. A dropout rate of 0 . 1 is applied for all linear transformation layers. The gradient clipping threshold is set as 10 . 0 . Early stopping on the corresponding validation data is adopted as a regularization strategy. During the testing, we vary the number of selected knowledge-entries m  { 1 , . . . , 15 } and set m = 2 for PTKGC cat and set m = 14 for PTKGC sep because they achieve the best performance. Since the characteristics of the two data sets are different (only WoW provides the golden knowledge label), we compare the proposed model with the baselines on both data sets individually. Baselines on WoW. 1) IR Baseline (Dinan et al., 2019) uses simple word overlap for response selection; 2) BoW MemNet (Dinan et al., 2019) is a memory network where knowledge entries are embedded via bag-of-words representation, and the model learns the knowledge selection and response matching jointly; 3) Transformer MemNet (Dinan et al., 2019) is an extension of BoW MemNet, Models R@1 R@2 R@5 Starspace (Wu et al., 2018) 50.7 64.5 80.3 BoW MemNet (Zhang et al., 2018) 51.6 65.8 81.4 KV Profile Memory (Zhang et al., 2018) 56.1 69.9 82.4 Transformer MemNet (Mazare et al., 2018) 60.3 74.4 87.4 DGMN (Zhao et al., 2019) 65.6 78.3 91.2 DIM (Gu et al., 2019) 78.7 89.0 97.1 FIRE (Gu et al., 2020b) 81.8 90.8 97.4 PTKGC cat 61.6 73.5 86.1 PTKGC sep 66.1 77.8 88.7 Table 2: Evaluation results on the test set of CMU DoG. and the dialogue history, response candidate and knowledge entries are encoded with Transformer encoder (Vaswani et al., 2017) pre-trained on a large data set. 4) Two-stage Transformer (Dinan et al., 2019) trains two separately models for knowledge selection and response retrieval respectively. A best-performing model on the knowledge selection task is used for the dialogue retrieval task. Baselines on CMU DoG 1) Starspace (Wu et al., 2018) selects the response by the cosine similarity between a concatenated sequence of dialogue context, knowledge, and the response candidate represented by StarSpace (Wu et al., 2018); 2) BoW MemNet (Zhang et al., 2018) is a memory network with the bag-of-words representation of knowledge entries as the memory items; 3) KV Profile Memory (Zhang et al., 2018) is a key-value memory network grounded on knowledge profiles; 4) Transformer MemNet (Mazare et al., 2018) is similar to BoW MemNet and all utterances are encoded with a pre-trained Transformer; 5) DGMN (Zhao et al., 2019) lets the dialogue context and all knowledge entries interact with the response candidate respectively via the cross-attention; 6) DIM (Gu et al., 2019) is similar to DGMN and all utterance are encoded with BiLSTMs; 7) FIRE (Gu et al., 2020b) first filters the context and knowledge and then use the filtered context and knowledge to perform the iterative response matching process. Performance of Response Selection. Table 1 and Table 2 report the evaluation results of response selection on WoW and CMU DoG where PTKGC cat and PTKGC sep represent the final matching score computed with the first strategy (Equation 9) and the second strategy (Equation 10) respectively. We can see that PTKGC sep is Models Wizard of Wikipedia CMU DoG Test Seen Test Unseen R@1 R@2 R@5 R@1 R@2 R@5 R@1 R@2 R@5 PTKGC sep 89.5 96.7 98.9 69.6 85.8 96.3 66.1 77.8 88.7 PTKGC sep (q) 70.6 79.7 86.8 55.9 70.8 83.4 47.3 58.8 75.0 PTKGC sep (q+h) 84.9 93.9 97.8 64.9 81.7 94.3 59.5 72.3 86.1 PTKGC sep (q+k) 89.5 96.4 98.6 67.0 84.0 96.0 62.7 73.8 84.8 PTKGC sep , m = 1 85.6 94.4 97.9 66.7 82.8 94.3 60.4 72.5 86.0 PTKGC sep , m = 1 L p 84.7 93.5 97.5 63.4 80.5 94.0 58.7 70.8 85.6 PTKGC sep , m = 1 L h 84.9 93.7 97.6 65.5 81.7 94.1 59.4 71.4 85.3 Table 3: Ablation study. consistently better than PTKGC cat over all metrics on two data sets, demonstrating that individually representing each knowledge-query-response triple with BERT can lead to a more optimal matching signal than representing a single long sequence. Our explanation to the phenomenon is that there is information loss when a long sequence composed of the knowledge and dialogue history passes through the deep architecture of BERT. Thus, the earlier different knowledge entries and dialogue history are fused together, the more information of dialogue history or background knowledge will be lost in matching. Particularly, on the WoW, in terms of R@1, our PTKGC sep achieves a comparable performance with the existing state-of-the-art models that are learned from the crowd-sourced training set, indicating that the model can effectively learn how to leverage external knowledge feed for response selection through the proposed pre-training approach. Notably, we can observe that our PTKGC sep performs worse than DIM and FIRE on the CMU DoG. Our explanation to the phenomenon is that the dialogue and knowledge in CMU DoG focus on the movie domain while our train data including ad-hoc retrieval corpora and multi-turn dialogues come from the open domain. Thus, our model may not select proper knowledge entries and can not well recognize the semantics clues for response matching due to the domain shift. Despite this, PTKGC sep can still show better performance than several existing models, such as Transformer MemNet and DGMN, though PTKGC sep does not access any training examples in the benchmarks. Performance of Knowledge Selection. We also assess the ability of models to predict the knowledge selected by human wizards in WoW data. The results are shown in Table 4. We can find that the performance of our method is comparable with various supervised methods trained on the gold knowledge index. In particular, on the test-seen, our model is slightly worse than Transformer (w/ pretrain), while on the test-unseen, our model achieves slightly better results. The results demonstrate the advantages of our pretraining tasks and the good generalization ability of our model. Ablation Study. We conduct a comprehensive ablation study to investigate the impact of different inputs and different tasks. First, we remove the dialogue history, knowledge, and both of them from the model, which is denoted as PTKGC sep (q+k), PTKGC sep (q+h) and PTKGC sep (q) respectively. According to the results of the first four rows in Table 3, we can find that both the dialogue history and knowledge are crucial for response selection as removing anyone will generally cause a performance drop on the two data. Besides, the background knowledge is more critical for response selection as removing the background knowledge causes more significant performance degradation than removing the dialogue history. as PTKGC sep -X, where X  {L p , L h } meaning query-passage matching task and query-dialogue history matching task respectively. Table 4 shows the ablation results of knowledge selection. We can find that both tasks are useful in the learning of knowledge selection, and query-passage matching plays a dominant role since the performance of knowledge selection drops dramatically when the task is removed from the pre-training process. The last two rows in Table 3 show the ablation results of response selection. We report the ablation results when only 1 knowledge is provided since the knowledge recalls for different ablated models and the full model are very close when m is large ( m = 14 ). We can see that both tasks are helpful and the performance of response selection drops more when removing the query-passage matching task. Particularly, L p plays a more important role and the performance on test-unseen of WoW drops more obvious when removing each training task. To further investigate the impact of our pretraining tasks on the performance of the multiturn response selection (without considering the grounded knowledge), we conduct an ablation study and the results are shown in Table 5. We can observe that the performance of the response matching model (no grounded knowledge) drops obviously when removing one of the pretraining tasks or both tasks. Particularly, the query-passage matching task contributes more to the response selection. The impact of the number of selected knowledge. We further study how the number of selected knowledge ( m ) influences the performance of PTKGC sep . Figure 2 shows how the performance of our model changes with respect to different numbers of selected knowledge. We observe that the performance increases monotonically until the knowledge number reaches a certain value, and then stable when the number keeps increasing. The results are rational because more knowledge entries can provide more useful 0.85 0.86 0.87 0.88 0.89 0.90 0.856 0.864 0.869 0.875 0.877 0.882 0.885 0.887 0.889 0.891 0.892 0.893 0.894 0.895 0.895 SeenUnseen 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 The number of selected knowledge ( m ) 0.65 0.66 0.67 0.68 0.69 0.70 0.667 0.672 0.675 0.682 0.682 0.681 0.682 0.682 0.685 0.687 0.688 0.690 0.692 0.696 0.696 R 100 @ 1 Figure 2: The performance of response selection across different number of selected knowledge. information for response matching, but when the knowledge becomes enough, the noise will be brought to matching. In this paper, we study response matching in knowledge-grounded conversations under a zero-resource setting. In particular, we propose decomposing the training of the knowledge-grounded response selection into three tasks and joint train all tasks in a unified pre-trained language model. Our model can be learned to select relevant knowledge and distinguish proper response, with the help of ad-hoc retrieval corpora and amount of multiturn dialogues. Experimental results on two benchmarks indicate that our model achieves a comparable performance with several existing methods trained on crowd-sourced data. In the future, we would like to explore the ability of our proposed method in retrieval-augmented dialogues. We would like to thank the anonymous reviewers for their constructive comments. This work was supported by the National Key Research and Development Program of China (No. 2020YFB1406702), the National Science Foundation of China (NSFC No. 61876196) and Beijing Outstanding Young Scientist Program (No. BJJWZYJH012019100020098). Rui Yan is the corresponding author, and is supported as a young fellow at Beijing Academy of Artificial Intelligence (BAAI).
abstain abstain method abstain abstain result abstain abstain abstain abstain abstain method abstain abstain abstain method abstain other method method method method abstain abstain objective method objective objective method result objective objective result other other other other other other other other other other other abstain abstain other abstain abstain abstain abstain abstain abstain abstain other abstain abstain method other abstain abstain method other abstain other abstain abstain other abstain method abstain abstain abstain abstain abstain abstain method other abstain method abstain other abstain abstain method abstain abstain abstain abstain other abstain abstain method method method abstain abstain abstain abstain abstain method abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain other abstain abstain abstain other abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain result abstain result result abstain abstain result abstain abstain abstain result result other abstain method objective abstain result objective other other other