In this paper, we present a novel integrated approach for keyphrase generation (KG). Unlike previous works which are purely extractive or generative, we first propose a new multitask learning framework that jointly learns an extractive model and a generative model. Besides extracting keyphrases, the output of the extractive model is also employed to rectify the copy probability distribution of the generative model, such that the generative model can better identify important contents from the given document. Moreover, we retrieve similar documents with the given document from training data and use their associated keyphrases as external knowledge for the generative model to produce more accurate keyphrases. For further exploiting the power of extraction and retrieval, we propose a neural-based merging module to combine and re-rank the predicted keyphrases from the enhanced generative model, the extractive model, and the retrieved keyphrases. Experiments on the five KG benchmarks demonstrate that our integrated approach outperforms the state-of-the-art methods. Keyphrases are short text pieces that can quickly express the key ideas of a given document. The keyphrase generation task aims at automatically generating a set of keyphrases given a document. As shown in the upper part of Figure 1, the input is a document and the output is a set of keyphrases. Due to the concise and precise expression, keyphrases are beneficial to extensive downstream applications such as text summarization (Zhang et al., 2004; Wang and Cardie, 2013), sentiment analysis (Wilson et al., 2005; Berend, 2011), and document clustering (Hulth and Megyesi, 2006; Hammouda et al., 2005). Document: Futility-Based Offspring Sizing. Parameter control in evolutionary algorithms (EAs) has been shown to be beneficial; however, the control of offspring size has so far received very little attention. This paper introduces Futility-Based Offspring Sizing (FuBOS), a method for controlling offspring size on a per generation basis without even requiring the user to set an initial offspring size value. . . Keyphrases: { evolutionary algorithm ; parameterless evolutionary algorithm; parameter control ; offspring sizing ; optimization} Retrieved Document: An Exploration into Dynamic Population Sizing . Traditional evolutionary algorithms are powerful problem solvers that have several fixed parameters which require prior specification. . . While many methods of parameter control have been published that focus on removing the population size parameter, , all hampered by a variety of problems. This paper investigates the benefits of making  a dynamic parameter and introduces two novel methods for population control. . . Retrieved Keyphrases: { evolutionary algorithm ; parameterless evolutionary algorithm; parameter control ; population sizing ; optimization} Figure 1: An example of keyphrase generation and retrieval. can be divided into two categories: extractive and generative . Extractive methods (Medelyan et al., 2009; Mihalcea and Tarau, 2004; Zhang et al., 2016; Luan et al., 2017) identify present keyphrases that appear in the source text like parameter control in Figure 1. Although extractive methods are simple to implement, they cannot predict absent keyphrases which are not in the document like optimization in Figure 1. Generative methods (Meng et al., 2017; Chen et al., 2018a; Ye and Wang, 2018; Yuan et al., 2018) adopt the wellknown encoder-decoder generative model (Luong et al., 2015; Bahdanau et al., 2014) with copy mechanism (Gu et al., 2016; See et al., 2017) to produce keyphrases. In a generative model, the decoder generates keyphrases word by word through either selecting from a predefined vocabulary according to a language model or copying from the source text according to the copy probability distribution computed by a copy mechanism. Thus, these generative methods are capable of generating both present and absent keyphrases. From a high-level perspective, extractive methods directly locate essential phrases in the document while generative models try to understand the document first and then produce keyphrases. To the best of our knowledge, these two kinds of methods have been developing independently without any combinations among them. However, when human annotators are asked to assign keyphrases to a document, they usually first obtain a global sense about which parts of the document are important and then write down the keyphrases word by word based on a more detailed understanding. To achieve such a goal, we propose a multi-task learning framework to take advantage of both extractive and generative models. For keyphrase extraction, we adopt a neural sequence labeling model to output the likelihood of each word in the source text to be a keyphrase word (or the importance score of each word). These importance scores are then employed to rectify the copy probability distribution of the generative model. Since the extractive model is explicitly trained to identify keyphrases from the source text, its importance scores can help the copy mechanism to identify important source text words more accurately. Different from the copy probability distribution which is dynamic at each generation step, these importance scores are static. Therefore, they can provide a global sense about which parts of the document are important. In addition, these scores are also utilized to extract present keyphrases which will be exploited by the merging module. Moreover, human annotators can also incorporate relevant external knowledge like the keyphrases of similar documents that they read before to assign more appropriate keyphrases. Correspondingly, to incorporate external knowledge, we propose a retriever to retrieve similar documents of the given document from training data. For instance, as shown in Figure 1, we retrieve a document from the KP20k training dataset that has the highest similarity with the upper document. The retrieved document is assigned with almost the same keyphrases as the upper document. Therefore, keyphrases from similar documents (i.e., retrieved keyphrases) can give useful knowledge to guide the generation of keyphrases for the given document. More concretely, we encode the retrieved keyphrases as vector representations and use them as an external memory for the decoder of the generative model in our multitask learning framework. Besides providing external knowledge, the retrieved keyphrases themselves are regarded as a kind of keyphrase prediction and can be utilized by the merging module. Finally, to imitate the integrated keyphrase assignment process of humans more comprehensively, we further exploit the extractive model and the retrieved keyphrases by proposing a merging module. This merging module collects and re-ranks the predictions from our aforementioned components. First, keyphrase candidates are collected from three different sources: (1) keyphrases generated by the enhanced generative model; (2) keyphrases extracted by the extractive model; and (3) the retrieved keyphrases. Then, we design a neural-based merging algorithm to merge and re-rank all the keyphrase candidates, and consequently return the top-ranked candidates as our final keyphrases. We extensively evaluate the performance of our proposed approach on five popular benchmarks. Experimental results demonstrate the effectiveness of the extractive model and the retrieved keyphrases in our multi-task learning framework. Furthermore, after introducing the merging module, our integrated approach consistently outperforms all the baselines and becomes the new state-of-the-art approach for keyphrase generation. In summary, our main contributions include: (1) a new multi-task learning framework that leverages an extractive model and external knowledge to improve keyphrase generation; (2) a novel neural-based merging module that combines the predicted keyphrases from extractive, generative, and retrieval methods to further improve the performance; and (3) the new state-of-the-art performance on five real-world benchmarks. Keyphrase extraction focuses on predicting the keyphrases that are present in the source text. Existing methods can mainly be categorized into two-step extraction approaches and sequence labeling models. Two-step extraction approaches first identify a set of candidate phrases from the document using different heuristics, such as the phrases that match specific part-of-speech (POS) tags (Liu et al., 2011; Wang et al., 2016; Le et al., 2016). Then, they learn a score for each candidate and select the top-ranked candidates as predicted keyphrases. The scores can be learned by either supervised methods with hand-crafted textual features (Medelyan et al., 2009; Witten et al., 1999; Nguyen and Kan, 2007; Frank et al., 1999; Hulth, 2003) or unsupervised graph ranking methods (Mihalcea and Tarau, 2004; Grineva et al., 2009; Wan and Xiao, 2008). Sequence labeling models are built on a recurrent neural network to sequentially go through a source text and learn the likelihood of each word in the source text to be a keyphrase word (Zhang et al., 2016; Luan et al., 2017; Gollapalli et al., 2017). In contrast to these extractive methods, our approach can generate both absent and present keyphrases. Keyphrase generation aims at predicting both present and absent keyphrases for a source text. Meng et al. (2017) proposed CopyRNN, which is built on the attentional encoder-decoder model (Bahdanau et al., 2014) with copy mechanism (Gu et al., 2016) to generate keyphrases. CorrRNN (Chen et al., 2018a), an extension of CopyRNN, was proposed to model the correlations among keyphrases. This model utilizes hidden states and attention vectors of previously generated keyphrases to avoid generating repetitive keyphrases. The title information of the source text was explicitly exploited by Ye and Wang (2018) and Chen et al. (2018b) to further improve the performance. Ye and Wang (2018) first considered a semi-supervised setting for keyphrase generation. In contrast, inspired by Hsu et al. (2018) and Cao et al. (2018), we enhance existing generative methods by adopting an extractive model to assist the copy mechanism and exploiting external knowledge from retrieved keyphrases to help the generation. Furthermore, we also design a merging module to combine the predictions from different components. As shown in Figure 2, our integrated framework consists of a retriever, two encoders, an extractor, a decoder, and a merging module. Given a document x , the retriever returns the keyphrases r retrieved from the training corpus. In addition to acting as keyphrase candidates, these retrieved keyphrases are also exploited to provide external guidance for the decoder. Then keyphrase extraction and generation are jointly conducted by the extractor and the decoder through sharing an en-Encoder1 Decoder Extractor Merging Module The source text  The retrieved keyphrases  Encoder2 Extracted Candidates GeneratedCandidates Retrieved Candidates     Final Predictions Retriever Figure 2: Our integrated framework. coder. Besides extracting keyphrase candidates, the importance scores of the source text words,  , predicted by the extractor are also employed to rescale the original copy probability distribution of the decoder. Thus, they can help the copy mechanism to detect important words more accurately. Finally, the merging module merges the candidates from three different sources (i.e., the retrieved, extracted, and generated candidates) and output the final predictions. Given a document x , the retriever module retrieves top K (document, keyphrases) pairs from the training corpus. The retrieval is based on the Jaccard similarities of the non-stop-word sets between x and the corpus documents. After that, the keyphrases of the top K pairs are returned and used in the later modules in two ways. First, these retrieved keyphrases are regarded as the keyphrase candidates of x and directly fed into the final merging module. In addition, these keyphrases are concatenated together as a guidance input r for the decoder to provide useful external knowledge for the generation process. A separator token is inserted among keyphrases when concatenating them together. We propose a multi-task learning framework which simultaneously learns to extract keyphrases from the source text and generate keyphrases word The inputs of the multi-task learning framework are the source text x and the concatenated retrieved keyphrases r . Both x and r are a sequence of tokens (i.e., x = [ x 1 , ..., x L x ] , r = [ r 1 , ..., r L r ] ), where L x and L r are the length of x and r respectively. The output of the extractor is a sequence of importance scores  = [  1 , ...,  L x ] , where  i is the probability of the i -th source word of being a keyphrase word. The output of the generator is a set of keyphrases Y = { y i } i =1 ,..,N , where N is the keyphrase number of x and y i = [ y i 1 , ..., y iL y i ] is a token sequence with length L y i . To fit the encoder-decoder framework, N tuples { ( x , r ,   , ( y i )  ) } i =1 ,...,N are split during training, where   and ( y i )  are the gold binary importance scores and one of the gold keyphrases of x correspondingly. For simplicity, we adopt ( x , r ,   , y  ) to represent such a tuple. Two encoders are employed in our multi-task learning framework. One is for the source text encoding (i.e., Encoder1 in Figure 2) and the other is for retrieved keyphrases encoding (i.e., Encoder2 in Figure 2). Both of them employ a bidirectional GRU (Cho et al., 2014) layer to obtain a context-aware representation of each word: u i = BiGRU 1 ( x i ,  u i  1 ,  u i +1 ) , (1) v j = BiGRU 2 ( r j ,  v j  1 ,  v j +1 ) , (2) where i = 1 , 2 , ..., L x and j = 1 , 2 , ..., L r . x i and r j are the d e -dimensional embedding vectors of the i -th source text word x i and j -th retrieved keyphrases word r j respectively. u i = [  u i ;  u i ]  R d and v j = [  v j ;  v j ]  R d are regarded as the corresponding context-aware representations, where d is the hidden size of the biderectional GRU layer. Finally, we obtain the internal memory bank U = [ u 1 , ..., u L x ] for later extraction and generation, and the external memory bank V = [ v 1 , ..., v L r ] for later generation. Based on the internal memory bank, we use the following sequence identifier as our extractor to identify whether the word is a keyphrase word in the source text. We denote the importance score P (  j = 1 | u j , s j , d ) as  j for simplicity:  j = sigmoid ( W c u j + u Tj W s d  u Tj W n tanh ( s j ) + b ) , (3) where d = tanh ( W d [  u L x ;  u 1 ] + b ) is the global document representation and s j = (cid:80) j  1 i =1 u i  i is current summary representation. W c , W s , and W n are the content, salience and novelty weights respectively. Although this extractor is inspired by Nallapati et al. (2017), our extractor identifies important words instead of sentences within the source text. In addition to the internal memory bank u 1 , ..., u L x ] , our decoder employs the external memory bank [ v 1 , ..., v L r ] to provide external guidance for the generation process. We exploit a decoder equipped with attention and copy mechanisms (Luong et al., 2015; See et al., 2017) to generate keyphrases. This decoder mainly consists of a forward GRU layer: h t =  GRU ([ e t  1 ;  h t  1 ] , h t  1 ) , (4) c int = attn ( h t , [ u 1 , ..., u L x ] , W in ) , (5) c ext = attn ( h t , [ v 1 , ..., v L r ] , W ex ) , (6)  h t = tanh ( W 1 [ c int ; c ext ]; h t ) , (7) where e t  1 is the embedding vector of the ( t  1) -th predicted word. The attn operation in Eq. (5) is defined as c int = (cid:80) L x i =1  int,i u i , where  int,i = exp( s t,i ) / (cid:80) L x j =1 exp( s t,j ) and s t,i = ( h t ) TW in u i . Similarly, we can obtain the external aggregated vector c ext . where g t =  ( w Tg  h t + b g )  R is the soft switch between generating from the predefined vocabulary V and copying from X that are all words appearing in the source text. P v ( y t ) = softmax ( W 2  h t + b v )  R | V | is the generating probability distribution over V and P c ( y t ) = (cid:80) i : x i = y t  ct,i  R | X | is the copying probability distribution over X . Previous work either directly uses the internal attention scores as the copy probabilities (i.e.,  ct,i =  int,i ) or employs extra neural network layers to calculate new copy scores. But we employ the rescaled internal attention scores  int by the importance scores [  1 , ...,  L x ] from the extractor as the final copy probabilities:  ct,i =  int,i   i (cid:80) L x j =1  int,j   j . The purpose of this rescaling is to provide extra guidance that which words within the source text are important and thus should obtain more attention when copying. Extraction Loss. We choose the source text words appearing in the assigned keyphrases as the gold important words and use the weighted cross-entropy loss for the extraction training i.e., L e =  1 L x (cid:80) L x j =1 w  j log  j +(1    j ) log (1   j ) , where   j  { 0 , 1 } is the ground-truth label for the j -th word and w is the loss weight for the positive training samples. Generation Loss. The negative log likelihood loss is utilized for the generation training i.e., L g =  (cid:80) L y  t =1 log P ( y  t | y t  1 , x , r ) , where y t = [ y 1 , ..., y t  1 ] is the previously predicted word sequence, L y  is the length of target keyphrase y  , and y  t is the t -th target word in y  . In this module, the retrieved, extracted and generated keyphrases are collected and then merged to produce the final keyphrase predictions. Retrieved Candidate Collection. The retrieved keyphrases from the retriever are regarded as the retrieved candidates. Each retrieved candidate ( rk ) obtains a retrieval score ( rs ) that is the Jaccard similarity between the corresponding document and x . The duplicates with lower retrieval scores are removed. Finally, we get N rk retrieved keyphrase candidates rk = [ rk 1 , . . . , rk N rk ] and their retrieval scores rs = [ rs 1 , . . . , rs N rk ] . Extracted Candidate Collection. The extracted keyphrase candidates are from the extractor. We select the word x j as a keyword if its importance score  j is larger or equal than a threshold (cid:15) (i.e.,  j  (cid:15) ). The adjacent keywords compound a keyphrase candidate. If no other adjacent keywords, the keyword itself becomes a single-word keyphrase candidate. Each extracted keyphrase candidate ( ek ) is accompanied by an extraction score ( es ) that is the mean of the importance scores of the words within this candidate. Similarly, duplicates with lower extraction scores are removed. Consequently, we obtain N ek extracted keyphrase candidates ek = [ ek 1 , . . . , ek N ek ] and the corresponding extraction scores es = [ es 1 , . . . , es N ek ] . Generated Candidate Collection. The generated keyphrase candidates directly come from the beam search process of the decoder. Each generated phrase is a keyphrase candidate. The beam search score of the generated candidate ( gk ) represents its generation score ( gs ). Duplicates with lower generation scores are removed. Then, we get N gk generated candidates gk = [ gk 1 , . . . , gk N gk ] and their generation scores gs = [ gs 1 , . . . , gs N gk ] . In addition to the original importance scores (i.e., rs , es , gs ), we also employ an auxiliary scorer to assign an auxiliary importance score to each keyphrase candidate. Given a document-candidate pair ( x , candidate), the scorer should output the probability that the candidate is one of the keyphrases of x . That means the scorer should determine the relationship between the given document x and the candidate, which is similar to a natural language inference (NLI) problem. Therefore, we adapt the most popular NLI model (Parikh et al., 2016) as our scorer. Different from typical natural language inference which is a multi-class classification problem, we use a binary classification setting to train the scorer. Besides, we learn the word embeddings and use two bi-directional GRU to obtain the input representations. The positive samples are the ground-truth keyphrases. The negative samples come from either the phrases in the document or the retrieved candidates. Notably, the ground-truth keyphrases are filtered when selecting negative samples. Consequently, a cross-entropy loss is utilized to train the scorer. Finally, the trained scorer is used to help the merging process as shown in Algorithm 1. The u gs u rs and u gs u es factors are used to enforce the average of rs and es to be the same with the average of gs and thus these three scores become comparable. Similar to Meng et al. (2017), we use KP20k dataset (Meng et al., 2017) to train our models. The released dataset contains 530,809 articles for training, 20,000 for validation, and the other 20,000 for testing. However, there exist duplicates in the KP20k training dataset with itself, the KP20k validation dataset, the KP20k testing dataset, and other four popular testing datasets (i.e., Inspec (Hulth, 2003), Krapivin (Krapivin et al., 2009), NUS (Nguyen and Kan, 2007), and SemEval (Kim et al., 2010)). After removing these duplicates, we maintain 509,818 articles in the training dataset. As for testing, following Meng et al. (2017), we employ five popular testing datasets from scientific publications as our testbeds for the baselines and our methods, which include Inspec , Krapivin , NUS , SemEval , and KP20k . For a comprehensive evaluation, we compare our methods with the traditional extractive baselines and the state-of-the-art generative methods. The extractive baselines include two unsupervised methods (i.e., TF-IDF and TextRank (Mihalcea and Tarau, 2004)) and one supervised method Maui (Medelyan et al., 2009). The generative baselines consist of CopyRNN (Meng et al., 2017) and CorrRNN (Chen et al., 2018a). We also conduct several ablation studies as follows:  KG-KE. KG-KR. The encoder-decoder generative model with retrieved keyphrases as external knowledge, but without combining with the extractive model and using the merging process. KG-KE-KR. The joint extraction and generation model with the retrieved keyphrases without using the merging process. All the above ablation models directly use the generated candidates as the final predictions. We denote our final integrated method which combines all the proposed modules as KG-KE-KR-M . Similar to CopyRNN and CorrRNN, we adopt macro-averaged recall (R) and F-measure (F 1 ) as our evaluation metrics. In addition, we also apply Porter Stemmer before determining whether two keyphrases are identical. Duplications are removed after stemming. We apply similar preprocessing procedures with Meng et al. (2017) including lowercasing, tokenizing and replacing digits with (cid:104) digit (cid:105) symbol. The title and the abstract of each article are concatenated as the source text input. We use the KP20k training dataset as the retrieval corpus. The implementations of our models are based on the OpenNMT system (Klein et al., 2017). The encoders, the decoder, and the scorer have the same vocabulary V with 50,000 tokens. The multi-task learning model and the scorer are trained separately. The embedding dimension d e and the hidden size d are set to 100 and 300 respectively. The initial state of the decoder GRU cell (i.e., h 0 ) is set to [  u L x ;  u 1 ] . The other GRU cells are set to zero. The retrieval number K is set to 3 after evaluating the retrieved keyphrases on the evaluation dataset. When concatenating the retrieved keyphrases together as an external knowledge input, we use ;' as the separator among them. During training, all the trainable parameters including the embeddings are randomly initialized with uniform distribution in [-0.1, 0.1]. We engage Adam (Kingma and Ba, 2014) as the optimizer with positive extraction loss weight w =9.0, batch size=64, dropout rate=0.1, max gradient norm=1.0, initial learning rate=0.001. The training is early stopped when the validation perplexity stops dropping for several continuous evaluations. While testing, the beam search depth, and beam size are set as 6 and 200 Model Inspec Krapivin NUS SemEval KP20k F 1 @5 F 1 @10 F 1 @5 F 1 @10 F 1 @5 F 1 @10 F 1 @5 F 1 @10 F 1 @5 F 1 @10 TF-IDF 0.188 0.269 0.092 0.120 0.103 0.142 0.076 0.135 0.087 0.113 TextRank 0.194 0.244 0.142 0.128 0.147 0.153 0.107 0.130 0.151 0.132 Maui 0.037 0.032 0.196 0.181 0.205 0.234 0.032 0.036 0.223 0.204 CorrRNN * 0.229 7 0.248 9 0.255 2 0.238 4 0.273 5 0.265 4 0.197 3 0.221 5 0.291 2 0.264 2 CopyRNN * 0.251 7 0.279 3 0.268 4 0.243 1 0.275 2 0.268 2 0.190 6 0.214 5 0.306 1 0.273 0 KG-KE 0.254 4 0.281 2 0.265 3 0.240 1 0.278 4 0.273 1 0.207 4 0.227 7 0.307 0 0.274 0 KG-KR 0.244 2 0.275 1 0.266 5 0.247 1 0.278 2 0.276 2 0.189 7 0.215 7 0.311 1 0.278 0 KG-KE-KR 0.245 1 0.278 4 0.267 3 0.246 2 0.285 9 0.279 4 0.194 4 0.220 2 0.314 0 0.280 0 KG-KE-KR-M 0.257 2 0.284 3 0.272 3 0.250 2 0.289 4 0.286 4 0.202 6 0.223 3 0.317 0 0.282 0 Table 1: Total keyphrase prediction results on all testing datasets. The extraction threshold (cid:15) is set to 0.7 after evaluating the extracted keyphrases on the evaluation dataset. Notably, the stemmer is not applied to the gold keyphrases of SemEval testing dataset since they have already been stemmed. We do not remove any single-word predictions for KP20k but only keep one single-word prediction for other testing datasets. The averaged results of three different random seed are reported 1 . Unlike the previous works which only separately analyze the present and absent keyphrase prediction ability, we also compare the whole keyphrase prediction ability regardless of the presence or absence of keyphrases, which is more reasonable in real applications. We show the F 1 scores at the top 5 and 10 predictions on Table 1. This table displays our KG-KE-KR-M method consistently outperforms the state-of-the-art mod-1 Our code is available at https://github.com/Chen-Wang-CUHK/KG-KE-KR-M Inspec Krapivin NUS SemEval KP20k 0.300 0.325 0.350 0.375 0.400 ( a ) P r e s en t F 1 @ 5 M ea s u r e CorrRNNCopyRNNKG-KEKG-KRKG-KE-KRKG-KE-KR-M Inspec Krapivin NUS SemEval KP20k 0.000 0.025 0.050 0.075 0.100 0.125 ( b ) A b s en t R @ 10 M ea s u r e Figure 3: The present and absent keyphrase prediction performance of all neural-based methods. els CopyRNN and CorrRNN demonstrating the effectiveness of our method. Moreover, we also observe that the KG-KE model exceeds CopyRNN and CorrRNN on most datasets, which indicates the strength of our combination with the extractive model. Besides, we also see the KG-KR model perform comparably or better than the baselines, suggesting the effective guidance ability of the retrieved keyphrases. In addition, after combining these two ideas, the KG-KE-KR model surpasses both or one of KG-KE and KG-KR on all datasets, which shows the effectiveness of the combination with extraction model and the retrieved keyphrases again. Finally, the performance gap between KG-KE-KR and KG-KE-KR-M implies the power of our merging module. For mean average precision (MAP) metric which considers prediction orders, we obtain similar conclusions as shown in Table 2. 5.2 Present and Absent Keyphrase Prediction In this section, we analyze the performance of present and absent keyphrase prediction. Only Candidate Sources Total F 1 @10 Present F 1 @5 Absent R @10 gk , ek , rk 0.250  0.002 0.330  0.002 0.172  0.002 gk , ek 0.249  0.003 0.328  0.003 0.154  0.002 gk , rk 0.249  0.002 0.329  0.002 0.172  0.002 gk 0.248  0.003 0.327  0.003 0.154  0.002 gk , no merging 0.246  0.002 0.324  0.002 0.158  0.002 ek , no merging 0.152  0.005 0.226  0.010 N/A rk , no merging 0.093  0.000 0.121  0.000 0.107  0.000 Table 3: Ablation study of the candidate sources of Algorithm 1 on Krapivin dataset. the present (absent) predictions and gold present (absent) keyphrases are preserved for the corresponding evaluation. We use F 1 @5 metric for present predictions and R@10 for absent predictions. Since the neural-based baselines are the state-of-the-art models, we focus on the comparison with them in this section. The results are depicted on Figure 3. The main observations are similar to the conclusions of total keyphrase prediction. Besides, we also note that after incorporating retrieved keyphrases, KG-KR model achieves substantial improvement gains over baselines on absent keyphrase prediction on Krapivin , NUS , and KP20k . These results demonstrate that the retrieved keyphrases indeed help the model to understand the main topics of the given document since generating absent keyphrase is an abstractive process and requires more powerful text understanding abilities. We notice that the KG-KE-KR-M method does not outperform the KG-KE-KR model on absent keyphrase prediction on Inspec dataset. One potential reason is that the merging module only merges two sources for absent keyphrases (i.e., the generated and retrieved keyphrases) instead of three sources like the present keyphrases do. Hence, the improvement for the absent keyphrases from the merging module is less stable than that for the present keyphrases. Moreover, we find that after combining with the extraction model, the KG-KE model achieves a huge improvement gain over CopyRNN on present keyphrase prediction on SemEval dataset, which manifests such a combination can improve the keyphrase extraction ability of the generative model. We also conduct in-depth ablation studies on our merging module. The objectives of these ablation studies are to (1) evaluate the effects of different Scoring Method Total F 1 @10 Present F 1 @5 Absent R @10 Combined 0.250  0.002 0.330  0.002 0.172  0.002 Only gs , es , rs 0.248  0.003 0.325  0.003 0.166  0.003 Only scorer 0.210  0.005 0.291  0.006 0.106  0.005 Table 4: Ablation study of the scoring method of Algorithm 1 on Krapivin dataset. candidate sources (i.e., what kinds of candidates are merged), and (2) analyze the effects of different final importance score calculating methods. Concerning candidate sources, we show the ablation study results on Table 3. When comparing  gk  with  gk , no merging, we can see that the merging algorithm improves the performance of total and present keyphrase predictions, but it degrades the performance of absent keyphrase prediction. These results indicate the trained scorer performs better on scoring present keyphrases than scoring absent keyphrases. One possible reason is that scoring absent keyphrases requires a stronger text understanding ability than scoring present keyphrases. However, as shown in the row of  gk , rk  on Table3, this problem can be solved by incorporating the retrieved keyphrases which provide external information to this module. Besides absent keyphrase prediction, it is observed that the retrieved keyphrases can also ben-efit the present keyphrase prediction. For the extracted keyphrases, as shown in the  gk , ek  row, they only improve the present keyphrase prediction ability and do not affect absent keyphrases as we anticipated. Regarding the scoring method, we further explore the effects of not using or only using the scorer in Algorithm 1. We show the results on Table 4. From this table, we note that after removing the scorer (i.e., Only gs , es , rs ), both present and absent keyphrase prediction performance become worse, which demonstrates the effectiveness of the combination with the scorer . Moreover, if we totally ignore the previously obtained retrieval, extraction and generation scores, and only use the scorer to predict the final keyphrase importance score (i.e., Only scorer ), we find the performance decreases dramatically, which indicates the indispensability of the previously obtained retrieval, extraction, and generation scores. Approximating minimum power covers of intersecting families and directed edge connectivity problems . Given a ( directed ) graph with costs on the edges , the power of a node is the maximum cost of an edge leaving it , and the power of the graph is the sum of the powers of its nodes . . . We consider problems that seek to find a min power spanning subgraph G of g that satisfies a prescribed edge connectivity property . . . We give approximation algorithms with ratio o ( k ln vertical bar v vertical bar ) . Our algorithms are based on a more general o ( ln vertical bar v vertical bar ) approximation algorithm for the problem of finding a min power directed edge cover of an intersecting set family . . . (a) Present Keyphrases {approximation algorithms; edge connectivity; intersecting families} CopyRNN : 1. approximation algorithms , 2. edge connectivity , 3. algorithms, 4. set cover, 5. connectivity, . . . Retrieved: 1. power, 2. graphs, 3. approximation, 4. edge connectivity , 5. approximation algorithms , . . . KG-KE-KR : 1. approximation algorithms , 2. edge connectivity , 3. power, 4. set cover, 5. minimum power, . . . 7. intersecting families , . . . KG-KE-KR-M : 1. approximation algorithms , 2. edge connectivity , 3. minimum power, . . . 6. intersecting families , . . . (b) Absent Keyphrases {wireless networks; power minimization; directed graphs} CopyRNN : 1. graph algorithms, 2. combinatorial problems, 3. computational complexity, 4. directed graphs , 5. randomized algorithms, . . . Retrieved: 1. wireless, 2. degree, 3. k connectivity, 4. tree augmentation, . . . 7. power assignment, 8. wireless networks KG-KE-KR : 1. graph algorithms, 2. directed graphs , 3. graph theory, 4. randomized algorithms, 5. spanning tree, 6. wireless networks , . . . KG-KE-KR-M : 1. graph algorithms, 2. directed graphs , 3. power assignment, 4. graph theory, 5. wireless networks , . . . Figure 4: A keyphrase prediction example of CopyRNN, KG-KE-KR, and KG-KE-KR-M. To illustrate the advantages of our proposed methods, we show an example of the present and absent keyphrase predictions in Figure 4. For fairness, we only compare with CopyRNN since our models are based on its implementation. From the results of the present keyphrase prediction, we find the extractor of the KG-KE-KR model successfully extracts all the present keyphrases from the source text, which shows the power of the extractor. With the help of the copy probability rescaling from the extractor, the KG-KE-KR model correctly predicts the keyphrase intersecting families which is not successfully predicted by CopyRNN and retrieved by the retriever. Moreover, by merging the extracted keyphrases into the final predictions, the KG-KE-KR-M model assigns a higher rank to this keyphrase (i.e., from 7 to 6). As for absent keyphrase prediction, we note that KG-KE-KR successfully predicts the keyphrase wireless net-works while CopyRNN fails. Since the retriever successfully retrieves this absent keyphrase, it shows that the retrieved keyphrases can provide effective external guidance for the generation process. Furthermore, the KG-KE-KR-M method assigns a higher rank to this keyphrase after merging the retrieved keyphrases into the final predictions (i.e., from 6 to 5). The overall results demonstrate the effectiveness of our proposed methods. In this paper, we propose a novel integrated approach for keyphrase generation. First, an end-to-end multi-task learning framework is introduced, which not only combines the keyphrase extraction and generation but also leverages the retrieved keyphrases from similar documents to guide the generation process. Furthermore, we introduce a neural-based merging algorithm to merge the candidates from three different components. Comprehensive empirical studies demonstrate the effectiveness of our approach. One interesting future work is to incorporate the similar documents themselves into keyphrase generation. The work described in this paper was partially supported by the Research Grants Council of the Hong Kong Special Administrative Region, China (No. CUHK 14208815 of the General Research Fund) and Meitu (No. 7010445). We would like to thank Jiani Zhang for her comments.
objective objective abstain method objective objective abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain objective abstain abstain abstain other abstain abstain abstain objective abstain objective method abstain abstain abstain abstain abstain abstain objective result abstain abstain method abstain objective abstain abstain method objective objective objective other other other other other other other objective other other other other other other abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain other abstain abstain abstain method abstain other abstain method method abstain other abstain abstain abstain abstain abstain abstain method abstain abstain method abstain other abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain other abstain result result abstain abstain result other abstain abstain abstain abstain result abstain result abstain abstain result abstain other abstain result abstain abstain abstain abstain result abstain result abstain result result other other abstain result abstain abstain abstain abstain abstain abstain objective abstain abstain objective abstain other other