Different texts shall by nature correspond to different number of keyphrases. This desideratum is largely missing from existing neural keyphrase generation models. In this study, we address this problem from both modeling and evaluation perspectives. We first propose a recurrent generative model that generates multiple keyphrases as delimiter-separated sequences. Generation diversity is further enhanced with two novel techniques by manipulating decoder hidden states. In contrast to previous approaches, our model is capable of generating diverse keyphrases and controlling number of outputs. We further propose two evaluation metrics tailored towards the variable-number generation. We also introduce a new dataset ( STACKEX ) that expands beyond the only existing genre (i.e., academic writing) in keyphrase generation tasks. With both previous and new evaluation metrics, our model outperforms strong baselines on all datasets. Keyphrase generation is the task of automatically predicting keyphrases given a source text. Desired keyphrases are often multi-word units that summarize the high-level meaning and highlight certain important topics or information of the source text. Consequently, models that can successfully perform this task should be capable of not only distilling high-level information from a document, but also locating specific, important snippets therein. To make the problem even more challenging, a keyphrase may or may not be a substring of the source text (i.e., it may be present or absent ). Moreover, a given source text is usually associated with  These authors contributed equally. a set of multiple keyphrases. Thus, keyphrase generation is an instance of the set generation problem, where both the size of the set and the size (i.e., the number of tokens in a phrase) of each element can vary depending on the source. Similar to summarization, keyphrase generation is often formulated as a sequence-to-sequence (Seq2Seq) generation task in most prior studies (Meng et al., 2017; Chen et al., 2018a; Ye and Wang, 2018; Chen et al., 2018b). Conditioned on a source text, Seq2Seq models generate phrases individually or as a longer sequence jointed by delimiting tokens. Since standard Seq2Seq models generate only one sequence at a time, thus to generate multiple phrases, a common approach is to over-generate using beam search (Reddy et al., 1977) with a large beam width. Models are then evaluated by taking a fixed number of top predicted phrases (typically 5 or 10) and comparing them against the ground truth keyphrases. Though this approach has achieved good empirical results, we argue that it suffers from two major limitations. Firstly, models that use beam search to generate multiple keyphrases generally lack the ability to determine the dynamic number of keyphrases needed for different source texts. Meanwhile, the parallelism in beam search also fails to model the inter-relation among the generated phrases, which can often result in diminished diversity in the output. Although certain existing models take output diversity into consideration during training (Chen et al., 2018a; Ye and Wang, 2018), the effort is significantly undermined during decoding due to the reliance on over-generation and phrase ranking with beam search. Secondly, the current evaluation setup is rather problematic, since existing studies attempt to match a fixed number of outputs against a variable number of ground truth keyphrases. Empirically, the number of keyphrases can vary drastically for different source texts, depending on a plethora of factors including the length or genre of the text, the granularity of keyphrase annotation, etc. For the several commonly used keyphrase generation datasets, for example, the average number of keyphrases per data point can range from 5.3 to 15.7, with variances sometimes as large as 64.6 (Table 1). Therefore, using an arbitrary, fixed number k to evaluate entire datasets is not appropriate. In fact, under this evaluation setup, the F1 score for the oracle model on the KP20 K dataset is 0.858 for k = 5 and 0.626 for k = 10 , which apparently poses serious normalization issues as evaluation metrics. To overcome these problems, we propose novel decoding strategies and evaluation metrics for the keyphrase generation task. The main contributions of this work are as follows: 1. We propose a Seq2Seq based keyphrase generation model capable of generating diverse keyphrases and controlling number of outputs. 2. We propose new metrics based on commonly used F 1 score under the hypothesis of variable-size outputs from models, which results in improved empirical characteristics over previous metrics based on a fixed k . 3. An additional contribution of our study is the introduction of a new dataset for keyphrase generation: STACKEX . With its marked difference in genre, we expect the dataset to bring added heterogeneity to keyphrase generation evaluation. Traditional keyphrase extraction has been studied extensively in past decades. In most existing literature, keyphrase extraction has been formulated as a two-step process. First, lexical features such as part-of-speech tags are used to determine a list of phrase candidates by heuristic methods (Witten et al., 1999; Liu et al., 2011; Wang et al., 2016; Yang et al., 2017). Second, a ranking algorithm is adopted to rank the candidate list and the top ranked candidates are selected as keyphrases. A wide variety of methods were applied for ranking, such as bagged decision trees (Medelyan et al., 2009; Lopez and Romary, 2010), Multi-Layer Per-ceptron, Support Vector Machine (Lopez and Ro-mary, 2010) and PageRank (Mihalcea and Tarau, 2004; Le et al., 2016; Wan and Xiao, 2008). Recently, Zhang et al. (2016); Luan et al. (2017); Gol-lapalli et al. (2017) used sequence labeling models to extract keyphrases from text; Subramanian et al. (2017) used Pointer Networks to point to the start and end positions of keyphrases in a source text; Sun et al. (2019) leveraged graph neural networks to extract keyphrases. The main drawback of keyphrase extraction is that sometimes keyphrases are absent from the source text, thus an extractive model will fail predicting those keyphrases. Meng et al. (2017) first proposed the CopyRNN, a neural model that both generates words from vocabulary and points to words from the source text. Based on the CopyRNN architecture, Chen et al. (2018a); Zhao and Zhang (2019) leveraged attention to help reducing duplication and improving coverage. Ye and Wang (2018) proposed semi-supervised methods by leveraging both labeled and unlabeled data for training. Chen et al. (2018b); Ye and Wang (2018) proposed to use structure information (e.g., title of source text) to improve keyphrase generation performance. Chan et al. (2019) introduced RL to the keyphrase generation task. Chen et al. (2019a) retrieved similar documents from training data to help producing more accurate keyphrases. Sequence to Sequence (Seq2Seq) learning was first introduced by Sutskever et al. (2014); together with the soft attention mechanism of (Bahdanau et al., 2014), it has been widely used in natural language generation tasks. Gulcehre et al. (2016); Gu et al. (2016) used a mixture of generation and pointing to overcome the problem of large vocabulary size. Paulus et al. (2017); Zhou et al. (2017) applied Seq2Seq models on summary generation tasks, while Du et al. (2017); Yuan et al. (2017) generated questions conditioned on documents and answers from machine comprehension datasets. Given a piece of source text, our objective is to generate a variable number of multi-word phrases. To this end, we opt for the sequence-to-sequence (Seq2Seq) (Sutskever et al., 2014) framework as the basis of our model, combined with attention and pointer softmax mechanisms in the decoder. Since each data example contains one source text sequence and multiple target phrase sequences (dubbed ONE 2M ANY , and each sequence can be of multi-word), two paradigms can be adopted for training Seq2Seq models. The first one (Meng et al., 2017) is to divide each ONE 2M ANY data example into multiple ONE 2O NE examples, and the resulting models (e.g., CopyRNN) can generate one phrase at once and must rely on beam search technique to produce more unique phrases. To enable models to generate multiple phrases and control the number to output, we propose the second training paradigm ONE 2S EQ , in which we concatenate multiple phrases into a single sequence with a delimiter (cid:104) sep (cid:105) , and this concatenated sequence is then used as the target for sequence generation during training. An overview of the model's structure is shown in Figure 1. 1 Notations In the following subsections, we use w to denote input text tokens, x to denote token embeddings, h to denote hidden states, and y to denote output text tokens. Superscripts denote time-steps in a sequence, and subscripts e and d indicate whether a variable resides in the encoder or the decoder of the model, respectively. The absence of a superscript indicates multiplicity in the time dimension. L refers to a linear transformation and L f refers to it followed by a non-linear activation function f . Angled brackets, (cid:104)(cid:105) , denote concatenation. We develop our model based on the standard Seq2Seq (Sutskever et al., 2014) model with attention mechanism (Bahdanau et al., 2014) and pointer softmax (Gulcehre et al., 2016). Due to 1 We release the code, datasets and model outputs for reproducing our results in https://github.com/memray/ OpenNMT-kpg-release . There are usually multiple keyphrases for a given source text because each keyphrase represents certain aspects of the text. Therefore keyphrase diversity is desired for the keyphrase generation. Most previous keyphrase generation models generate multiple phrases by over-generation, which is highly prone to generate similar phrases due to the nature of beam search. Given our objective to generate variable numbers of keyphrases, we need to adopt new strategies for achieving better diversity in the output. Recall that we represent variable numbers of keyphrases as delimiter-separated sequences. One particular issue we observed during error analysis is that the model tends to produce identical tokens following the delimiter token. For example, suppose a target sequence contains n delimiter tokens at time-steps t 1 , . . . , t n . During training, the model is rewarded for generating the same delimiter token at these time-steps, which presumably introduces much homogeneity in the corresponding decoder states h t 1 d , . . . , h t n d . When these states are subsequently used as inputs at the time-steps immediately following the delimiter, the decoder naturally produces highly similar distributions over the following tokens, resulting in identical tokens being decoded. To alleviate this problem, we propose two plug-in components for the sequential generation model. We propose a mechanism called semantic coverage that focuses on the semantic representations of generated phrases. Specifically, we introduce another uni-directional recurrent model GRUSC (dubbed target encoder ) which encodes decoder-generated tokens y  , where   [0 , t ) , into hidden states h t SC . This state is then taken as an extra input to the decoder GRU, modifying equation of the decoder GRU to: h td = GRU d ( (cid:104) x td , h t SC (cid:105) , h t  1 d ) . (1) If the target encoder were to be updated with the training signal from generation (i.e., backpropagat-ing error from the decoder GRU to the target en-coder), the resulting decoder is essentially a 2-layer GRU with residual connections. Instead, inspired Source Encoder MLP <s> linear PCA <sep> <sep> linear PCA <sep> convex function convex function <sep> SVD Decoder Target Encoder : A : C : B </s> SVD Figure 1: The architecture of the proposed model for improving keyphrase diversity. A represents last states of a bi-directional source encoder; B represents the last state of target encoder; C indicates decoder states where target tokens are either delimiters or end-of-sentence tokens. During orthogonal regularization, all C states are used; during target encoder training, we maximize mutual information between states A with B . Red dash arrow indicates a detached path, i.e., no back-propagation through such path. by previous representation learning works (Lo-geswaran and Lee, 2018; van den Oord et al., 2018; Hjelm et al., 2018), we train the target encoder in an self-supervised fashion (Figure 1). Specifically, due to the autoregressive nature of the RNN-based decoder, we follow Contrastive Predictive Coding (CPC) (van den Oord et al., 2018), where a Noise-Contrastive Estimation(NCE) loss is used to maximize a lower bound on mutual information. That is, we extract target encoder's final hidden state vector h M SC , where M is the length of target sequence, and use it as a general representation of the target phrases. We train by maximizing the mutual information between these phrase representations and the final state of the source encoder h Te as follows. For each phrase representation vector h M SC , we take the encodings H Te = { h Te, 1 , . . . , h Te,N } of N different source texts, where h Te,true is the encoder representation for the current source text, and the remaining N  1 are negative samples (sampled at random) from the training data. The target encoder is trained to minimize the classification loss: LSC =  log g ( h Te,true , h M SC ) (cid:80) i  [1 ,N ] g ( h Te,i , h M SC ) , g ( h a , h b ) = exp( h (cid:62) a Bh b ) (2) where B is bi-linear transformation. The motivation here is to constrain the overall representation of generated keyphrase to be semantically close to the overall meaning of the source text. With such representations as input to the decoder, the semantic coverage mechanism can potentially help to provide useful keyphrase information and guide generation. We also propose orthogonal regularization, which explicitly encourages the delimiter-generating decoder states to be different from each other. This is inspired by Bousmalis et al. (2016), who use orthogonal regularization to encourage representations across domains to be as distinct as possible. Specifically, we stack the decoder hidden states corresponding to delimiters together to form matrix H = (cid:104) h t 1 d , . . . , h t n d (cid:105) and use the following equation as the orthogonal regularization loss: LOR = (cid:13)(cid:13)(cid:13) H (cid:62) H (cid:12) ( 1  I n ) (cid:13)(cid:13)(cid:13) 2 , (3) where H (cid:62) is the matrix transpose of H , I n is the identity matrix of rank n , (cid:12) indicates element wise multiplication, (cid:107) M (cid:107) 2 indicates L 2 norm of each element in a matrix M . This loss function prefers orthogonality among the hidden states h t 1 d , . . . , h t n d and thus improves diversity in the tokens following the delimiters. We adopt the widely used negative log-likelihood loss in our sequence generation model, denoted as LNLL . The overall loss we use for optimization is: L = LNLL +  OR  LOR +  SC  LSC , (4) where  OR and  SC are hyper-parameters. According to different task requirements, various decoding methods can be applied to generate the target sequence y . Prior studies Meng et al. (2017); Yang et al. (2017) focus more on generating excessive number of phrases by leveraging beam search to proliferate the output phrases. In contrast, models trained under ONE 2S EQ paradigm are capable of determining the proper number of phrases to output. In light of previous research in psychology (Van Zandt and Townsend, 1993; Forster and Bednall, 1976), we name these two de-coding/search strategies as Exhaustive Decoding and Self-terminating Decoding, respectively, due to their resemblance to the way humans behave in serial memory tasks. Simply speaking, the major difference lies in whether a model is capable of controlling the number of phrases to output. We describe the detailed decoding strategies used in this study as follows: 3.3.1 Exhaustive Decoding As traditional keyphrase tasks evaluate models with a fixed number of top-ranked predictions (say F-score @5 and @10), existing keyphrase generation studies have to over-generate phrases by means of beam search (commonly with a large beam size, e.g., 150 and 200 in (Chen et al., 2018b; Meng et al., 2017), respectively), a heuristic search algorithm that returns K approximate optimal sequences. For the ONE 2O NE setting, each returned sequence is a unique phrase itself. But for ONE 2S EQ , each produced sequence contains several phrases and additional processes (Ye and Wang, 2018) are needed to obtain the final unique (ordered) phrase list. It is worth noting that the time complexity of beam search is O ( Bm ) , where B is the beam width, and m is the maximum length of generated sequences. Therefore the exhaustive decoding is generally very computationally expensive, especially for ONE 2S EQ setting where m is much larger than in ONE 2O NE . It is also wasteful as we observe that less than 5% of phrases generated by ONE 2S EQ models are unique. An innate characteristic of keyphrase tasks is that the number of keyphrases varies depending on the document and dataset genre, therefore dynamically outputting a variable number of phrases is a desirable property for keyphrase generation models 2 . Since our model is trained to generate a variable number of phrases as a single sequence joined by delimiters, we can obtain multiple phrases by simply decoding a single sequence for each given 2 Note this is fundamentally different from other NLG tasks. In specific, the number of keyphrases is variable, the length of each keyphrase is also variable. source text. The resulting model thus implicitly performs the additional task of dynamically estimating the proper size of the target phrase set: once the model believes that an adequate number of phrases have been generated, it outputs a special token < /s > to terminate the decoding process. One notable attribute of the self-terminating decoding strategy is that, by generating a set of phrases in a single sequence, the model conditions its current generation on all previously generated phrases. Compared to the exhaustive strategy (i.e., phrases being generated independently by beam search in parallel), our model can model the dependency among its output in a more explicit fashion. Additionally, since multiple phrases are decoded as a single sequence, decoding can be performed more efficiently than exhaustive decoding by conducting greedy search or beam search on only the top-scored sequence. Formally, given a source text, suppose that a model predicts a list of unique keyphrases  Y = ( y 1 , . . . ,  y m ) ordered by the quality of the predictions  y i , and that the ground truth keyphrases for the given source text is the oracle set Y . When only the top k predictions  Y : k = ( y 1 , . . . ,  y min( k,m ) ) are used for evaluation, precision , recall , and F 1 score are consequently conditioned on k and defined as: P @ k = |  Y : k  Y| |  Y : k | , R @ k = |  Y : k  Y| |Y| , F 1 @ k = 2  P @ k  R @ k P @ k + R @ k . (5) As discussed in Section 1, the number of generated keyphrases used for evaluation can have a critical impact on the quality of the resulting evaluation metrics. Here we compare three choices of k and the implications on keyphrase evaluation for each choice:  F 1 @ k : where k is a pre-defined constant (usu-ally 5 or 10). Due to the high variance of the number of ground truth keyphrases, it is often that |  Y : k |  k < |Y| , and thus R @ k  and in turn F 1 @ k  of an oracle model can be smaller than 1 . This undesirable property is unfortunately prevalent in the evaluation metrics adopted by all existing keyphrase generation studies to our knowledge. A simple remedy is to set k as a variable number which is specific to each data example. Here we define two new metrics: Kp20K Inspec Krapivin NUS SemEval Model @5 @10 @ O @5 @10 @ O @5 @10 @ O @5 @10 @ O @5 @10 @ O Abstractive Neural CopyRNN (Meng et al.) 32.8 25.5  29.2 33.6  30.2 25.2  34.2 31.7  29.1 29.6  CopyRNN* 31.7 27.3 33.5 24.4 28.9 29.0 30.5 26.6 32.5 37.6 35.2 40.6 31.8 31.8 31.7 CorrRNN (Chen et al.) ---31.8 27.8 -35.8 33.0 -32.0 32.0 ParaNetT+CoAtt (ZhaoandZhang) 36.0 28.9 -29.6 35.7 -32.9 28.2 -36.0 35.0 -31.1 31.2 catSeqTG-2RF1  (Chan et al.) 32.1 -35.7 25.3 -28.0 30.0 -34.8 37.5 -25.5 28.7 -29.8 KG-KE-KR-M  (Chen et al.) 31.7 28.2 38.8 25.7 28.4 31.4 27.2 25.0 31.7 28.9 28.6 38.4 20.2 22.3 30.3 CatSeq (Ours) 31.4 27.3 31.9 29.0 30.0 30.7 30.7 27.4 32.4 35.9 34.9 38.3 30.2 30.6 31.0 CatSeqD (Ours) 34.8 29.8 35.7 27.6 33.3 33.1 32.5 28.5 37.1 37.4 36.6 40.6 32.7 35.2 35.7 Extractive IR TfIdf (Hasan and Ng) 7.2 9.4 6.3 16.0 24.4 20.8 6.7 9.3 6.8 11.2 14.0 12.2 8.8 14.7 11.3 TextRank (Mihalcea and Tarau) 18.1 15.1 18.4 28.6 33.9 33.5 18.5 16.0 21.1 23.0 21.6 23.8 21.7 22.6 22.9 KEA (Witten et al.) 4.6 4.4 5.1 2.2 2.2 2.2 1.8 1.7 1.7 7.3 7.1 8.1 6.8 6.5 6.6 Maui (Medelyan et al.) 0.5 0.5 0.4 3.5 4.6 3.9 0.5 0.7 0.6 0.4 0.6 0.6 1.1 1.4 1.1 Extractive Neural DivGraphPointer (Sun et al.) 36.8 29.2 -38.6 41.7 -46.0 40.2 -40.1 38.9 -36.3 29.7 w/ Additional Data Semi-Multi (Ye and Wang) 32.8 26.4 -32.8 31.8 -32.3 25.4 -36.5 32.6 -31.9 31.2 TG-Net (Chen et al.) 37.2 31.5 -31.5 38.1 -34.9 29.5 -40.6 37.0 -31.8 32.2 Table 2: Performance (F 1 -score) of present keyphrase prediction on scientific publications datasets. In this case, k = |Y| , which means for each data example, the number of predicted phrases taken for evaluation is the same as the number of ground truth keyphrases. F 1 @ M : M denotes the number of predicted keyphrases. In this case, k = |  Y| and we simply take all the predicted phrases for evaluation without truncation. By simply extending the constant number k to different variables accordingly, both F 1 @ O and F 1 @ M are capable of reflecting the nature of variable number of phrases for each document, and a model can achieve the maximum F 1 score of 1 . 0 if and only if it predicts the exact same phrases as the ground truth. Another merit of F 1 @ O is that it is independent from model outputs, therefore we can use it to compare existing models. In this section, we report our experiment results on multiple datasets and compare with existing models. We use CatSeq to refer to the delimiter-3 We acknowledge that F 1 @ O scores of Chan et al. (2019) and Chen et al. (2019a) might be not completely comparable with ours. This is due to additional post-processing and filtering methods might have been applied in different work. We elaborate the data pre-processing and evaluation protocols used in this work in Appendix E. concatenated sequence-to-sequences model described in Section 3; CatSeqD refers to the model augmented with orthogonal regularization and semantic coverage mechanism. To construct target sequences for training CatSeq and CatSeqD , ground truth keyphrases are sorted by their order of first occurrence in the source text. Keyphrases that do not appear in the source text are appended to the end. This order may guide the attention mechanism to attend to source positions in a smoother way. Implementation details can be found in Appendix D. As for the pre-processing and evaluation, we follow the same steps as in (Meng et al., 2017). More details are provide in Appendix E for reproducing our results. We include a set of existing models (Meng et al., 2017; Chen et al., 2018a; Chan et al., 2019; Zhao and Zhang, 2019; Chen et al., 2019a) as baselines, they all share same behavior of abstractive keyphrase generation with our proposed model. Specially for computing existing model's scores with our proposed new metrics ( F 1 @ O and F 1 @ M ), we implemented our own version of CopyRNN (Meng et al., 2017) based on their open sourced code, denoted as CopyRNN*. We also report the scores of models from Chan et al. and Chen et al. based on their publicly released outputs. We also include a set of models that use sim-Present Absent Model F 1 @5 F 1 @10 F 1 @ O R@10 R@50 TfIdf 8.0 8.9 5.2 -TextRank 12.1 10.1 11.6 -KEA 4.9 4.8 5.3 -Maui 35.8 23.3 51.8 -CopyRNN* 44.2 30.3 66.2 48.8 66.0 CatSeq 48.3 45.5 63.5 40.7 42.2 CatSeqD 48.7 43.9 65.6 54.8 65.7 Table 3: Model performance on STACKEX dataset. ilar strategies but can not directly compare with. This includes four non-neural extractive models: TfIdf (Hasan and Ng, 2010), TextRank (Mihalcea and Tarau, 2004), KEA (Witten et al., 1999), and Maui (Medelyan et al., 2009); one neural extractive model (Sun et al., 2019); and two neural models that use additional data (e.g., title) (Ye and Wang, 2018; Chen et al., 2019b). In Section 5.3, we apply the self-terminating decoding strategy. Since no existing model supports such decoding strategy, we only report results from our proposed models. They can be used for comparison in future studies. Our first dataset consists of a collection of scientific publication datasets, namely KP20 K , INSPEC , KRAPIVIN , NUS , and SEMEVAL , that have been widely used in existing literature (Meng et al., 2017; Chen et al., 2018a; Ye and Wang, 2018; Chen et al., 2018b; Chan et al., 2019; Zhao and Zhang, 2019; Chen et al., 2019a; Sun et al., 2019). KP20 K , for example, was introduced by Meng et al. (2017) and comprises more than half a million scientific publications. For each article, the abstract and title are used as the source text while the author keywords are used as target. The other four datasets contain much fewer articles, and thus used to test transferability of our model. We report our model's performance on the present -keyphrase portion of the KP20 K dataset in Table 2. 4 To compare with previous works, we provide compute F 1 @5 and F 1 @10 scores. The new proposed F 1 @ O metric indicates consistent ranking with F 1 @5 / 10 for most cases. Due to its target number sensitivity, we find that its value is closer to F 1 @5 for KP20 K and KRAPIVIN where average target keyphrases is less and closer to F 1 @10 for the other three datasets. From the result we can see that our CatSeqD outperform existing abstractive models on most of the datasets. Our implemented CopyRNN* achieves better or comparable performance against the original model, and on NUS and SemEval the advantage is more salient. As for the proposed models, both CatSeq and CatSeqD yield comparable results to CopyRNN, indicating that ONE 2S EQ paradigm can work well as an alternative option for the keyphrase generation task. CatSeqD outperforms CatSeq on all metrics, suggesting the semantic coverage and orthogonal regularization help the model to generate higher quality keyphrases and achieve better generalizability. To our surprise, on the metric F 1 @10 for KP20 K and KRAPIVIN (average number of keyphrases is only 5), where high-recall models like CopyRNN are more favored, CatSeqD is still able to outperform ONE 2O NE baselines, indicating that the proposed mechanisms for diverse generation are effective. Inspired by the StackLite tag recommendation task on Kaggle, we build a new benchmark based on the public StackExchange data 5 . We use questions with titles as source, and user-assigned tags as target keyphrases. We provide details regarding our data collection in Appendix C. Since oftentimes the questions on StackExchange contain less information than in scientific publications, there are fewer keyphrases per data point in STACKEX (statistics are shown in Table 1). Furthermore, StackExchange uses a tag recommendation system that suggests topic-relevant tags to users while submitting questions; therefore, we are more likely to see general terminology such as 5 https://archive.org/details/stackexchange, we choose 19 computer science related topics from Oct. 2017 dump. Linux and Java 6 . This characteristic challenges models with respect to their ability to distill major topics of a question rather than selecting specific snippets from the text. We report our models' performance on STACKEX in Table 3. Results show CatSeqD performs the best in general; on the absent -keyphrase generation tasks, it outperforms CatSeq by a large margin. One key advantage of our proposed model is the capability of predicting the number of keyphrases conditioned on the given source text. We thus conduct a set of experiments on KP20 K and STACKEX present keyphrase generation tasks, as shown in Table 4, to study such behavior. We adopt the self-terminating decoding strategy (Section 3.3), and use both F 1 @ O and F 1 @ M (Section 4) to evaluate. In these experiments, we use beam search as in most Natural Language Generation (NLG) tasks, i.e., only use the top ranked prediction sequence as output. We compare the results with greedy search. Since no existing model is capable of generating variable number of keyphrases, in this subsection we only report performance on such setting from CatSeq and CatSeqD . From Table 4 we observe that in the variable number generation setting, greedy search outperforms beam search consistently. This may because beam search tends to generate short and similar sequences. We can also see the resulting F 1 @ O scores are generally lower than results reported in previous subsections, this suggests an over-generation decoding strategy may still benefit from achieving higher recall. We conduct an ablation experiment to study the effects of orthogonal regularization and semantic coverage mechanism on CatSeq . As shown in Table 5, semantic coverage provides significant boost to CatSeq 's performance on all datasets. Orthogonal regularization hurts performance when is solely applied to CatSeq model. Interestingly, when both components are enabled ( CatSeqD ), the model outperforms CatSeq by a noticeable margin on all datasets, this suggests the two components help keyphrase generation in a synergistic way. One future direction is to apply orthogonal regularization directly on target encoder, since the regularizer can potentially diversify target representations at phrase level, which may further encourage diverse keyphrase generation in decoder. To verify our assumption that target encoding and orthogonal regularization help to boost the diversity of generated sequences, we use two metrics, one quantitative and one qualitative, to measure diversity of generation. First, we simply calculate the average unique predicted phrases produced by both CatSeq and CatSeqD in experiments shown in Section 5.1 (beam size is 50). The resulting numbers are 20.38 and 89.70 for CatSeq and CatSeqD respectively. Second, from the model running on the KP20 K validation set, we randomly sample 2000 decoder hidden states at k steps following a delimiter ( k = 1 , 2 , 3 ) and apply an unsupervised clustering method (t-SNE (van der Maaten and Hinton, 2008)) on them. From the Figure 2 we can see that hidden states sampled from CatSeqD are easier to cluster while hidden states sampled from CatSeq yield one mass of vectors with no obvious distinct clusters. Results on both metrics suggest target encoding and orthogonal regularization indeed help diversifying generation of our model. To illustrate the difference of predictions between our proposed models, we show an example cho-sen from the KP20 K validation set in Appendix F. In this example there are 29 ground truth phrases. Neither of the models is able to generate all of the 6 One example is shown in Appendix F. Figure 2: t-SNE results on decoder hidden states. keyphrases, but it is obvious that the predictions from CatSeq all start with test, while predictions from CatSeqD are diverse. This to some extent verifies our assumption that without the target encoder and orthogonal regularization, decoder states following delimiters are less diverse. We propose a recurrent generative model that sequentially generates multiple keyphrases, with two extra modules that enhance generation diversity. We propose new metrics to evaluate keyphrase generation. Our model shows competitive performance on a set of keyphrase generation datasets, including one introduced in this work. In future work, we plan to investigate how target phrase order affects the generation behavior, and further explore set generation in an order invariant fashion. This work is supported by the National Science Foundation under grant No. 1525186. This research was also supported in part by the University of Pittsburgh Center for Research Computing through the resources provided. The authors thank the anonymous ACL reviewers for their helpful feedback and suggestions.
abstain abstain abstain objective abstain objective objective objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain objective objective objective objective objective method other other other other other other other other other other other other other other other other abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain method abstain method method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain other abstain method abstain method abstain other abstain abstain method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain other method abstain abstain other abstain abstain abstain abstain abstain abstain result result result abstain abstain abstain abstain abstain result other abstain abstain abstain abstain abstain abstain method method abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method objective objective result objective other other other