2 State Key Laboratory of Computer Sciences, Institute of Software, Chinese Academy of Sciences 3 University of Chinese Academy of Sciences, Beijing, China Abstract In this paper, we focus on the problem of keyword and document matching by considering different relevance levels. In our recommen003 dation system, different people follow differ004 ent hot keywords with interest. We need to 005 attach documents to each keyword and then 006 distribute the documents to people who fol007 low these keywords. The ideal documents 008 should have the same topic with the keyword, 009 which we call topic-aware relevance. In other 010 words, topic-aware relevance documents are 011 better than partially-relevance ones in this ap012 plication. However, previous tasks never de013 fine topic-aware relevance clearly. To tackle 014 this problem, we define a three-level relevance 015 in keyword-document matching task: topic016 aware relevance, partially-relevance and irrel017 evance. To capture the relevance between 018 the short keyword and the document at above019 mentioned three levels, we should not only 020 combine the latent topic of the document with 021 its deep neural representation, but also model 022 complex interactions between the keyword 023 and the document. To this end, we propose 024 a Two-stage Interaction and Topic-Aware text 025 matching model (TITA). In terms of topic026 aware, we introduce neural topic model to 027 analyze the topic of the document and then 028 use it to further encode the document. In 029 terms of two-stage interaction, we propose 030 two successive stages to model complex in031 teractions between the keyword and the docu032 ment. Extensive experiments reveal that TITA 033 outperforms other well-designed baselines and 034 shows excellent performance in our recom035 mendation system. The keyword-document matching is mostly like the query-document matching task. The query039 document matching task, aiming to calculate rele040 vance score between a query and a document, has 041 been extensively studied over the past few years. It 042 is widely applicable in many real scenarios: (1) in 043 the information retrieval systems (Guo et al., 2016), 044 query-document matching is an important feature 045 in the ranking models. (2) as for the task of ques-046 tion answering (Yang et al., 2016), query-document 047 matching method can be used to find document can-048 didates or to help predict the answer span. (3) it 049 is also widely applied to recommendation systems 050 (Jiang et al., 2019). In many scenarios, we need to distinguish dif-052 ferent keyword-document (query-document) rele-053 vance levels. For instance, in our recommendation 054 system, we need to attach documents to some hot 055 keywords and then distribute the documents to the 056 people who follow the keywords. In this circum-057 stance, the document and the keyword should better 058 have the same topic, which we call topic-aware rel-059 evance. As shown in Table 1, for the hot keyword 060 cherry blossoms, the document (labeled 2) should 061 be the ideal document which should be attached be-062 cause it has the same topic with the keyword while 063 the document (labeled 1) should be a secondary 064 choice, because only several words or phrases in 065 this document match the keyword but the topics of 066 the document mismatch the keyword. To tackle this problem, we define a three-068 level relevance: topic-aware relevance, partially-069 relevance and irrelevance. The topic-aware rele-070 vance means the keyword and the document have 071 the same topic while the partially-relevance means 072 only part of the document matches with the key-073 word. Our task is more challenging than previous 074 query-document matching tasks. To capture the rel-075 evance between the keyword and the document at 076 above-mentioned three levels, we should not only 077 combine the latent topic of the document with its 078 deep representation, but also model complex inter-079 actions between the keyword and the document. Previous neural query-document matching mod-081 els (similar as keyword-document matching) can be divided into two categories according to their model architectures (Guo et al., 2016). One is the 084 Keyword : cherry blossoms Original Keyword :   Label : 0 Irrelevance Case Translated Document : There was a flower shop which has opened for a few months. representation-based models, in which representa085 tions for a query and a document are built indepen086 dently. In other words, there are no word-level or 087 phrase-level interactions between the query and the 088 document. For instance, the well-known DSSM 089 (Huang et al., 2013) has been verified effective in 090 query-document matching tasks. However, these 091 representation-based series cannot model complex 092 interactive signals between a query and a document 093 effectively. The other one we call interaction-based 094 models, in which word or phrase-level information 095 fusion occurs. It has been verified more effective 096 to directly learn interactions than individual repre097 sentations. Examples include ARC II (Hu et al., 098 2014), MatchPyramid (Pang et al., 2016). Recently, 099 interaction-based methods are widely used in many 100 NLP tasks, like BIDAF (Seo et al., 2016) and R101 NET (Wang et al., 2017). More recently, BERT (Devlin et al., 2018) has made great influence in the field of NLP. It has 104 achieved state-of-the art results in many NLP ap105 plications. The pre-trained language models can be 106 applied directly to this keyword-document match-107 ing task. However, these above-mentioned types of keyword-document (query-document) matching models can be improved to be applied to our rec-111 ommendation system in the following aspects: (1) They do not analyse the topic of the document. It is 113 expected that topic model can be used to solve this 114 problem. (2) Previous interaction-based models 115 can still be improved to capture complex matching 116 signals between a query and a document. To this 117 end, we propose the TITA model. By topic-aware, 118 we introduce neural topic model (Miao et al., 2017) 119 to analyze the latent topic representation of the 120 document and then use this latent topic to further 121 encode the document. By two-stage interaction, we 122 propose a two-stage interaction to model complex 123 interactions between a query and a document. Our research contributions can be summarized as follows. propose the TITA model to improve them. 129 Our model has two advantages: (1) it encodes 130 the latent topic embedding into the deep neu131 ral representation of the document, which can 132 aid the prediction of the topic-aware relevance. 133 (2) it can model more complex interactions be134 tween a keyword and a document through a 135 two-stage keyword-document interaction. 136  We perform extensive experiments on our 137 keyword-document matching dataset. The re138 sults reveal that the proposed TITA model 139 outperforms the well-designed baselines. 140  From a real recommendation system, we 141 define a three-level relevance in keyword142 document matching task and construct a new 143 dataset. 144  Our model is applied in our recommendation 145 system and improves the click-through rate by 146 4.35%. Depending on the model architectures, text matching models can be divided into two categories: representation-based and interaction-based. The 151 former ones first transform every piece of text to a 152 representation with neural networks, such as Deep 153 Semantic Similarity Model(DSSM) (Huang et al., 154 2013), Convolutional Deep Semantic Similarity 155 Model(CDSSM) (Shen et al., 2014), LSTM-RNN 156 (Palangi et al., 2016), Bi-LSTM, etc. Conversely, 157 the latter models focus on modeling the interac158 tion between a query and a document, such as 159 Arc-II(Hu et al., 2014), MatchPyramid (Pang et al., 160 2016), BIDAF(Seo et al., 2016) and RNET(Wang 161 et al., 2017). Representation-based methods generate distributed representations from input texts through neural networks. There are a number of works em165 ploying these methods, which differ mainly in the 166 procedure to construct the representations and the 167 way of calculating a matching score. Huang et al. 168 (2013) propose DSSM, which is the first one to 169 apply a neural network. In DSSM, each piece of 170 the query or the document is represented through 171 a multilayer perceptron and then a matching score 172 is calculated by the cosine similarity. Compared 173 to traditional text matching models, DSSM shows 174 significant improvements. Compared with representation-based methods, the interaction-based methods aim to capture di- rect matching features: the degree and the struc-178 ture of matching. The interaction-based model, 179 which means query-document interaction occurs 180 before matching, can somewhat solve the above-181 mentioned problem in the representation-based 182 models. It has been verified more effective to di-183 rectly learn interactions than individual represen-184 tations. Hu et al. (2014) propose ARC-II, which 185 first represents the query and the document by the 186 knowledge of each other, and adjusts the sliding 187 windows in the first convolution layer to focus on 188 adjacent word vectors. Inspired by the success of 189 convolutional neural network in image recognition, 190 Pang et al. (2016) propose MatchPyramid to model 191 text matching as the problem of image recogni-192 tion. Leveraging the attention mechanism, Seo 193 et al. (2016) and Wang et al. (2017) introduce at-194 tention mechanism to improve the matching degree 195 of the query and the document. Recently BERT (Devlin et al., 2018) has caused a stir in the field of NLP. It has achieved state-198 of-the-art results in many NLP applications. The 199 pre-trained language model series can be applied 200 directly to this keyword-document matching task. Topic models aim to discover the topics as well as the topic representations of documents in the document collection. It learns latent topics from 204 documents in an unsupervised manner. Topics are 205 captured as latent variables that have a word prob-206 ability distribution. Topic models have a long tra-207 dition in this scenario area as well, such as biblio-208 metrics, translations and recommendations. Hall et al. (2008) describe the flow of topics between papers. Zhao and Xing (2006) enable 211 word alignment process to leverage topical contents 212 of document-pairs. Jiang et al. (2015) use topic 213 model to enrich users' information for effective 214 inference. In this section, we describe details of the TITA model. As depicted in Figure 1, our TITA model 218 has three major components: (1) a two-stage 219 keyword-document interaction, see Part A; (2) a 220 neural topic model, see Part B; (3) a joint train-221 ing mechanism, see Part C. First, we introduce 222 the task definition. Then, we elaborate the two-223 stage keyword-document interaction and neural 224 topic model in the TITA model respectively. Fi-225 nally, a joint training mechanism is introduced to 226 incorporate latent topics to the deep representation 227 Figure 1: The architecture of the TITA model, which consists of three major components: (1) a two-stage keyword-document interaction, which combines the multi-head attention in BERT and a successive cross representation layer to link the keyword and the document; (2) a neural topic model, which calculate a latent topic of the document to further enrich the document representation; (3) a joint training mechanism to train the model in a joint process. of the document and train the model in a joint pro228 cess. Notably, we conduct experiments using both 229 Bi-LSTM and BERT as text encoders. Here, we 230 only describe the proposed methods with BERT as 231 the encoder for simplicity. In our keyword-document matching task, we explicitly model the relevance between a keyword and a document as a relevance level prediction task. 236 The input of the task is a keyword Q and a docu237 ment D . The output r Q,D  { 0 , 1 , 2 } indicates the 238 keyword-document relevance levels. The keyword-document matching model is desired to capture the rich interactions between the keyword and the document in the matching process. 244 As show in Table 1, the keyword cherry blossoms 245 and the topic-aware relevance document have many 246 correlating signals, e.g., the phrase cherry blos247 soms in the keyword and the phrase flowering 248 period in the document. The two-stage keyword-document interaction in the TITA model is to fuse the information of the document and the keyword. In the first-stage in252 teraction, we employ BERT (Devlin et al., 2018) 253 as the encoder to simultaneously model the se254 quential information of the keyword and the doc255 ument along with their interactive relationship 256 by the multi-head self-attention mechanism. In 257 the second-stage interaction, we perform a cross-258 attention between the representations of the key-259 word and the document to further capture their 260 interactive relationship. First-stage Interaction As shown in Figure 1, in the first-stage interaction, we concatenate the key-263 word and the document by a separator [SEP] as input and then feed them into BERT. The input con-265 sists of the keyword characters c Q = { c Qm } Mm =1 and 266 the document characters c D = { c Dn } Nn =1 , where M , 267 N indicate the length of the keyword characters and 268 the document characters respectively. The states in 269 the last hidden layer of BERT can be regarded as 270 the encoding of the document, i.e., e D . e D = BERT ([ c Q ; [SEP] ; c D ]) (1) where e D = { e Dn } Nn =1  RN  d . In each hidden 273 layer of BERT, the multi-head self-attention mech-274 anism is performed as the following equations: 275 Attention (  Q,  K,  V ) = softmax (  Q  KT  d k )  V (2) 276 MultiHead (  Q,  K,  V ) = Concat ( hd 1 , ..., hd h ) WO (3) 277 hd i = Attention (  QW  Qi ,  KW  Ki ,  V W  Vi ) (4) 278 where  Q ,  K and  V are the output hidden states 279 of the former layer. W  Qi , W  Ki and W  Vi are the 280 parameters corresponding to each head. WO is 281 the output projection parameter. Second-stage Interaction Note that in the firststage interaction, the query and the document characters are concatenated as input. The model learns 286 keyword-keyword, keyword-document, document287 document interactions simultaneously through 288 self-attention mechanism in transformer blocks 289 of BERT. In our keyword-document matching 290 task, keyword-document interaction is more im291 portant than document-document and keyword292 keyword interactions. Therefore, we introduce the 293 second-stage interaction layer to conduct keyword294 document contextualization independently. Firstly, 295 we obtain the representation of the keyword e Q by 296 the BERT encoder. where e Q = { e Q m } Mm =1 . Then, we compute a simi299 larity matrix using the keyword embedding and the 300 document embedding. S = ( s mn )  RM  N s mn = (cid:10) e Qm , e Dn (cid:11) v T  R where (cid:104) e Qm , e Dn (cid:105) represents a element-wise multiplication, v  R d is a trainable weight vector. In this 305 similarity matrix, the value s mn indicates the link 306 between the m -th character embedding in the key307 word and the representation of the n -th character in 308 the document. Then, we apply this similarity ma309 trix to further encode the keyword by calculating 310 attention over the document: 311 u Q = (cid:8) u Qm (cid:9) M m =1 (8) 312 u Q m = N (cid:88) n =1 a mn e Dn  R d (9) 313 a m = softmax ( s m )  RN (10) 314 where s m = { s mn } Nn =1 and a m means which 315 characters in the document should be attended re316 garding the m -th character of the keyword. We 317 then add the original keyword representation e Q 318 with u Q to get the keyword embedding: 319 u Q = u Q + e Q (11) 320 Similarly, we use this similarity matrix to get the 321 document representation u D  RN  d . As show in Table 1, the topic-aware relevance case and the partially-relevance case both have some words relevant to cherry blossoms. But the topic 326 of the topic-aware relevance document is more re-327 lated with the keyword cherry bollosoms. By 328 contrast, the topic of the partial-relevance doc-329 ument is more likely to be a document about 330 a restaurant, which is not related to the key-331 word cherry blossom. Following this direction, 332 analysing the topic of the document is a way to pro-333 mote keyword-document matching models. Specif-334 ically, we introduce neural topic model to produce 335 the latent topic and then use it to update the up-336 stream representation of the document. As shown in Figure 1, the input of the neural topic model is a word sequence of the document w D . The bag-of-words (BOW) representation of 340 the document is x D  R | V w | , where | V w | is the 341 size of the word vocabulary. Assume that the latent 342 variable  represents the topic distribution in the 343 document w D . The probabilistic topic models, like 344 LDA(Blei et al., 2003), apply the Dirichlet distribu-345 tion as the prior of the latent variable   Dir (  ) , 346 where  is the parameter of the Dirichlet distribu-347 tion. By contrast, in the neural topic model, Gaus-348 sian Softmax Construction (Miao et al., 2017) is 349 applied using a neural network to parameterise the 350 topic distribution   GGSM (  0 ,  20 ) : 351 x  N (  0 ,  20 ) (12) 352  = softmax ( WT 1 x ) (13) 353 where W 1 is a trainable parameter. 0 and  0 are 354 the parameters of the prior Gaussian distribution N . 355 Assuming there are K topics, if z n  { 1 , ..., K } 356 is the topic assignment for the observed word w D n , 357 then: 358 z n  Multi (  ) (14) 359  z n  R | V w | is a topic distribution over the 360 words in the vocabulary given z n . The topic distri-361 bution can be calculated by the similarity between 362 the topic and the words in the vocabulary: 363  z n = softmax ( v T t z n ) (15) 364 where t  R d  K is the topic vector which is a 365 parameter of the neural topic model,  v  R d | V w | 366 is the word vector. K is the total topic number. 367 Then, the generative probability of each word w n 368 can be calculated by: 369 p ( w n |  z n ) = Multi (  z n ) (16) 370 The neural topic model is implemented by an 371 Auto-Encoding Variational Bayes (AEVB) algo372 rithm (Kingma and Welling, 2013). The encoder 373 is used to approximate the true posterior of the 374 latent variable p (  | x ) . Specifically, the encoder 375 takes the BOW (Bag-of-Words) representation of 376 the document as the input and generates the poste377 rior Gaussian Softmax Construction parameters  378 and  2 through neural networks. In practice, the la379 tent variable  is sampled by the reparameterization 380 trick. = f 1 ( x D ) , log  = f 2 ( x D )   GGSM ( ,  2 ) where f  (  ) is a multi-layer perceptron. The de384 coder is responsible for reconstructing the docu385 ment by maximizing the log likelihood of the input 386 document. The latent variable z n can be integrated 387 out as follows. Finally, the variational lower bound of the neural topic model is obtained by combining the reconstruction error term and the KL divergence term. 393 The parameters of neural topic model can be trained 394 by maximizing this function. where p (cid:48) (  | D ) means the variational posterior distribution of document D , approximating the true posterior p (  | D ) . It's expected that introducing topic model can benefit the model in the prediction of the abovementioned three levels. In this subsection, we de403 sign a joint training mechanism to incorporate the 404 latent topic representation to further encode the 405 document and train the model in a joint process. 406 As described above, u D is the document repre-407 sentation after the two-stage keyword-document 408 interaction. RK | V w | is the topic distribu-409 tion over the vocabulary, where  ij means that the 410 weight between the i -th topic and the j -th word. 411 We are inspired from an end2end memory net-412 work(Sukhbaatar et al., 2015), which is used to 413 memorize multiple sentences in question answer-414 ing task. Similarly, in TITA, we intend to embed 415 the topic-word weight into the deep representation 416 of the document. 417 As depicted in Figure 1 part C, the input of mem-418 ory network is  and the deep document repre-419 sentation after the two-stage keyword-document 420 interaction u D . is memorized in the memory of 421 the network, where  k means the representation of 422 the k -th topic over the vocabulary of size | V w | . The TITA model has two memory hops as shown in the Figure 1. In the following, we describe the 425 model in a single memory hop operation for sim-426 plicity. One hop has two major components: the 427 input memory and the output memory. In the in-428 put memory representation, a matching score is 429 calculated taking  and u D as input: 430 p k = softmax (  k V u D ) (22) 431 where V  R | V w | d is a trainable weight vector. In 432 the output memory representation part, we compute 433 the slot output vector using the output memory and 434 the matching score: 435 o D = K (cid:88) k =1 ( p k c k ) (23) 436 437 o D = W o ( o D + u D ) (24) 438 where c  RK  d is a trainable output memory. 439 We compute two relevance vectors r 1 and r 2 . One 440 takes u D and u Q as input, while the other one using 441 u Q and o D . We merge the two relevance vectors 442 and then apply softmax function to get the final 443 relevance level: 444 r 1 = WR 1 [ u D ; u Q ] + b R 1 (25) 445 r 2 = WR 2 [ o D ; u Q ] + b R 2 (26) 446 r Q,D = softmax ( WR [ r 1 ; r 2 ] + b R ) (27) 447 where [; ] is vector concatenation operation and 448 W  , b  are all trainable variables. The TITA model integrates three different parts as shown in Figure 1: a two-stage keyworddocument interaction, a neural topic model and a joint training mechanism. In the training pro453 cess, the neural topic model and the joint model are 454 trained alternatively to a convergent status. We first 455 train the neural topic model for  epochs to get a 456 topic distribution over vocabulary, i.e.,  . Then in 457 the joint training process, the model takes the out458 put of the two-stage keyword-document interaction 459 and the output of the neural topic model to conduct 460 training the model parameters for classification. In this section, we conduct experiments on our keyword-document matching dataset from our recommendation system and the results demonstrate the superiority of the TITA model compared to the baselines. We apply accuracy as the evaluation metric. In 468 this paper, we care mostly about rigidly distinguish469 ing the three keyword-document matching levels. 470 We believe that documents of different matching 471 levels have different usages. For instance, in our 472 online recommendation system, the goal of our 473 model is to recall the topic-aware relevance docu474 ments and there is no need to rank documents of 475 each keyword. Our keyword-document matching dataset is in Chinese, derived from our recommendation system. 479 The domains mainly lie in food (e.g., beef and 480 western food), sports (e.g., football and jogging), 481 entertainment (e.g., photography and comedy) and 482 so on. For all the 8901 keywords, we get 10 doc483 uments for each keyword by users' behavior in 484 our recommendation system, e.g., click-through. 485 As for how to choose 10 documents for each key486 word in the baseline online recommendation sys487 tem, for a certain keyword, hundreds of documents 488 are recalled for different users, in which the topic489 aware relevance documents tend to have high click490 through rate while irrelevance ones tend to have 491 low click-through rate. For each keyword, we se492 lect 6 documents which have high click-through 493 rate as well as 4 documents with low click-through 494 rate. According to our analysis, this setting tends 495 to generate similar ratios of three-level relevance 496 documents for all the keywords. As a result, each 497 keyword has 10 corresponding documents. Each 498 keyword-document pair is manually annotated at 499 different relevance levels. As shown in Table 1, 500 relevance level-2 means the document and the key-501 word have the same topic, while relevance level-0 502 means the keyword and the document are irrele-503 vant. Relevance level-1 is an intermediate rele-504 vance level, which means only a small portion of 505 the document describes some useful information of 506 the keyword. To make the ratios of level-2, level-1, 507 level-0 cases nearly the same, we randomly delete 508 some documents. As a result, we have 8,901 key-509 words and 66,019 corresponding documents. Fi-510 nally, the dataset is randomly split into 50% for 511 training, 25% for validation and 25% for testing. In the experiment, we set the cutoff length of the document sequence as 512 characters and the cutoff length of the keyword as 16 characters in Chinese. 516 The size of the character vocabulary V c is 21128. 517 The size of the word vocabulary for neural topic 518 model V w is 5000, which contains top frequent 519 words after deleting stop words. We use pre-trained 520 embeddings by BERT to initialize the character 521 embeddings. We directly use BERT base model 522 released by Google with the hidden size of 768. In 523 the neural topic model, we set the number of topics 524 # K = 50. We use all the documents in the training 525 set to train the neural topic model for 50 epochs. 526 The topic embedding size d is set to 384 and we set 527 the word embedding to the same size. The padding 528 is masked to avoid affecting the gradient. We use 529 the optimization algorithm Adam (Kingma and Ba, 530 2014) with learning rate 5e-5 and batch size as 32. 531 As for the parameters of Adam,  1 and  2 are set 532 to 0.9 and 0.999 respectively. As described in the Introduction Section, the keyword-document matching models can be di-536 vided into two categories: representation-based and interaction-based matching model. As shown in 538 Table 2, many strong baselines are included in the 539 performance comparison. Table 2 shows that the TITA model outperforms all the models evaluated by accuracy in this keyword-543 document matching task. From this table, we have 544 the other observations: (1) The TITA model is more 545 competent in this task. It outperforms ARC-II by 546 7.06% and outperforms BERT by 5.38%, which 547 Figure 2: The architecture of online deployment of the TITA model, which consists of two major components: an offline data processor module and an online data usage module. strongly proves that topic model and two-stage in548 teraction can benefit this task. (2) Most interaction549 based models behave better than representation550 based ones. (3) Pre-trained word embeddings can 551 also aid this task. To further examine the effectiveness of the neural topic model and the two-stage keyworddocument interaction, we make a detailed ablation analysis as shown in Table 3. Bi-LSTM : The TITA model is based on BiLSTM, which encodes a query and a document independently before matching. + Neural Topic Model : Bi-LSTM plus neu-560 ral topic model outperforms the Bi-LSTM baseline by a large scale (i.e., 2.02%), which indicates that the keyword-document match-563 ing task can benefit from the latent topic rep-564 resentation of the document. + First-stage Keyword-document Interac-566 tion : After adding the first-stage keyword-567 document interaction, the model behaves bet-568 ter. It proves that concatenating the query and 569 document to conduct interaction is effective. + Second-stage Keyword-document Inter-571 action : We add the second-stage interaction to make further improvement. We infer that 573 the cross attention is more capable in captur-574 ing interactions between a keyword and a doc-575 ument. Replace Bi-LSTM with BERT : We apply BERT to initialize the word representation, whose parameters are to be finetuned. We can 579 observe that the model performs even better 580 than the former one, which reveals that the 581 pre-trained word representations are useful in 582 the keyword-document matching task. Because the model is heavy and the total numbers of keywords are limited (8901 in total), we gener-586 ate data in offline, as shown in Figure 2. In offline 587 data processor, we first use BM25 to retrieve and 588 rank billions of document candidates and keep the 589 top10000 candidates for TITA model to further 590 conduct query-document relation prediction. After 591 that we can get a ranked list of topic matching doc-592 uments and partially relevance documents for all 593 keywords, which will be stored in a KV database. 594 In the online data usage, we recall documents of all 595 the keywords, which the user follows, for further 596 re-ranking in our recommendation system. As for the online gains, we attached more than one million topic-matching documents for the 8901 keywords. These documents are all distributed 600 in our recommendation system with the number 601 of views about 1 . 9 e 6 /day . We improve the click602 through rate by 4.35% (from 6.52% to 10.87%), 603 which is a great improvement. We define a new keyword-document matching task with three relevance levels from a real recommendation system, to address the problem that different scenarios require documents of different relevance levels. Further, we propose a TITA model to dis610 tinguish different relevance levels, which can cap611 ture latent topics of a document and hold complex 612 keyword-document interactions at the same time. 613 Extensive experiments reveal the superiority of our 614 model compared to other strong baselines. Ab615 lation test shows that the model can improve the 616 keyword-document matching in the same way as 617 we think. Moreover, our model shows excellent per618 formance in our recommendation system, in which 619 it improves the click-through rate by 4.35%.
abstain method method abstain abstain abstain abstain method method objective method objective result abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain objective method objective objective abstain method abstain method objective objective result other other other other other other other other other other other other other other other other other other other other other other other other other abstain abstain abstain other abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain other abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain method method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain method abstain abstain abstain abstain objective objective result result result