Gated Self-Matching Networks for Reading Comprehension and Question Answering
In this paper, we present the gated selfmatching networks for reading comprehension style question answering, which aims to answer questions from a given passage. We first match the question and passage with gated attention-based recurrent networks to obtain the question-aware passage representation. Then we propose a self-matching attention mechanism to refine the representation by matching the passage against itself, which effectively encodes information from the whole passage. We finally employ the pointer networks to locate the positions of answers from the passages. We conduct extensive experiments on the SQuAD dataset. The single model achieves 71.3% on the evaluation metrics of exact match on the hidden test set, while the ensemble model further boosts the results to 75.9%. At the time1 of submission of the paper, our model holds the first place on the SQuAD leaderboard for both single and ensemble model.
1 Introduction
In this paper, we focus on reading comprehension style question answering which aims to answer questions given a passage or document. We specifically focus on the Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al., 2016), a largescale dataset for reading comprehension and question answering which is manually created through crowdsourcing. SQuAD constrains answers to the space of all possible spans within the reference passage, which is different from cloze-style reading comprehension datasets (Hermann et al.,
1On Feb. 6, 2017
2015; Hill et al., 2016) in which answers are single words or entities. Moreover, SQuAD requires different forms of logical reasoning to infer the answer (Rajpurkar et al., 2016).
Rapid progress has been made since the release of the SQuAD dataset. Wang and Jiang (2016b) build question-aware passage representation with match-LSTM (Wang and Jiang, 2016a), and predict answer boundaries in the passage with pointer networks (Vinyals et al., 2015). Seo et al. (2016) introduce bi-directional attention flow networks to model question-passage pairs at multiple levels of granularity. Xiong et al. (2016) propose dynamic co-attention networks which attend the question and passage simultaneously and iteratively refine answer predictions. Lee et al. (2016) and Yu et al. (2016) predict answers by ranking continuous text spans within passages.
Inspired by Wang and Jiang (2016b), we introduce a gated self-matching network, illustrated in Figure 1, an end-to-end neural network model for reading comprehension and question answering. Our model consists of four parts: 1) the recurrent network encoder to build representation for questions and passages separately, 2) the gated matching layer to match the question and passage, 3) the self-matching layer to aggregate information from the whole passage, and 4) the pointernetwork based answer boundary prediction layer. The key contributions of this work are three-fold.
First, we propose a gated attention-based recurrent network, which adds an additional gate to the attention-based recurrent networks (Bahdanau et al., 2014; Rocktäschel et al., 2015; Wang and Jiang, 2016a), to account for the fact that words in the passage are of different importance to answer a particular question for reading comprehension and question answering. In Wang and Jiang (2016a), words in a passage with their corresponding attention-weighted question context are en-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
coded together to produce question-aware passage representation. By introducing a gating mechanism, our gated attention-based recurrent network assigns different levels of importance to passage parts depending on their relevance to the question, masking out irrelevant passage parts and emphasizing the important ones.
Second, we introduce a self-matching mechanism, which can effectively aggregate evidence from the whole passage to infer the answer. Through a gated matching layer, the resulting question-aware passage representation effectively encodes question information for each passage word. However, recurrent networks can only memorize limited passage context in practice despite its theoretical capability. One answer candidate is often unaware of the clues in other parts of the passage. To address this problem, we propose a self-matching layer to dynamically refine passage representation with information from the whole passage. Based on question-aware passage representation, we employ gated attention-based recurrent networks on passage against passage itself, aggregating evidence relevant to the current passage word from every word in the passage. A gated attention-based recurrent network layer and self-matching layer dynamically enrich each passage representation with information aggregated from both question and passage, enabling subsequent network to better predict answers.
Lastly, the proposed method yields state-of-theart results against strong baselines. Our single model achieves 71.3% exact match accuracy on the hidden SQuAD test set, while the ensemble model further boosts the result to 75.9%. At the time of submission of this paper, our model holds the first place on the SQuAD leader board.
2 Task Description
For reading comprehension style question answering, a passage P and question Q are given, our task is to predict an answer A to question Q based on information found in P. The SQuAD dataset further constrains answer A to be a continuous subspan of passage P. Answer A often includes nonentities and can be much longer phrases. This setup challenges us to understand and reason about both the question and passage in order to infer the answer. Table 1 shows a simple example from the SQuAD dataset.
Passage: Tesla later approached Morgan to ask for more funds to build a more powerful transmitter. When asked where all the money had gone, Tesla responded by saying that he was affected by the Panic of 1901, which he (Morgan) had caused. Morgan was shocked by the reminder of his part in the stock market crash and by Tesla’s breach of contract by asking for more funds. Tesla wrote another plea to Morgan, but it was also fruitless. Morgan still owed Tesla money on the original agreement, and Tesla had been facing foreclosure even before construction of the tower began. Question: On what did Tesla blame for the loss of the initial money? Answer: Panic of 1901
Table 1: An example from the SQuAD dataset.
3 Gated Self-Matching Networks
Figure 1 gives an overview of the gated selfmatching networks. First, the question and passage are processed by a bi-directional recurrent network (Mikolov et al., 2010) separately. We then match the question and passage with gated attention-based recurrent networks, obtaining question-aware representation for the passage. On top of that, we apply self-matching attention to aggregate evidence from the whole passage and refine the passage representation, which is then fed into the output layer to predict the boundary of the answer span.
3.1 Question and Passage Encoder
Consider a question Q = {wQt }mt=1 and a passage P = {wPt }nt=1. We first convert the words to their respective word-level embeddings ({eQt }mt=1 and {ePt }nt=1) and character-level embeddings ({cQt }mt=1 and {cPt }nt=1). The character-level embeddings are generated by taking the final hidden states of a bi-directional recurrent neural network (RNN) applied to embeddings of characters in the token. Such character-level embeddings have been shown to be helpful to deal with out-ofvocab (OOV) tokens. We then use a bi-directional RNN to produce new representation uQ1 , . . . , u Q m and uP1 , . . . , u P n of all words in the question and passage respectively:
uQt = BiRNNQ(u Q t−1, [e Q t , c Q t ]) (1) uPt = BiRNNP (u P t−1, [e P t , c P t ]) (2)
We choose to use Gated Recurrent Unit (GRU) (Cho et al., 2014) in our experiment since it performs similarly to LSTM (Hochreiter and Schmidhuber, 1997) but is computationally cheaper.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
𝑢1 𝑄 𝑢2 𝑄 𝑢𝑚 𝑄
Question
Attention Question
Vector
𝑣1 𝑃 𝑣2 𝑃 𝑣3 𝑃
𝑢1 𝑃 𝑢2 𝑃 𝑢3 𝑃
Passage
𝑣1 𝑃 𝑣2 𝑃 𝑣3 𝑃 𝑣𝑛 𝑃
ℎ1 𝑃 ℎ2 𝑃 ℎ3 𝑃
Attention
ℎ1 𝑎 ℎ2 𝑎
Question and Passage GRU Layer
Question and Passage Matching Layer
Passage Self-Matching Layer
Output Layer Start End
𝑢𝑛 𝑃…
…𝑣𝑛 𝑃…
ℎ𝑛 𝑃…
When was tested The delay in
…
test……
𝑟𝑄
Figure 1: Gated Self-Matching Networks structure overview.
3.2 Gated Attention-based Recurrent Networks
We propose a gated attention-based recurrent network to incorporate question information into passage representation. It is a variant of attentionbased recurrent networks, with an additional gate to determine the importance of information in the passage regarding a question. Given question and passage representation {uQt }mt=1 and {uPt }nt=1, Rocktäschel et al. (2015) propose generating sentence-pair representation {vPt }nt=1 via soft-alignment of words in the question and passage as follows:
vPt = RNN(v P t−1, ct) (3)
where ct = att(uQ, [uPt , v P t−1]) is an attentionpooling vector of the whole question (uQ):
stj = v Ttanh(WQu u Q j + W P u u P t + W P v v P t−1)
ati = exp(s t i)/Σ m j=1exp(s t j)
ct = Σ m i=1a t iu Q i (4)
Each passage representation vPt dynamically incorporates aggregated matching information from the whole question.
Wang and Jiang (2016a) introduce matchLSTM, which takes uPt as an additional input into the recurrent network:
vPt = RNN(v P t−1, [u P t , ct]) (5)
To determine the importance of passage parts and attend to the ones relevant to the question, we add another gate to the input ([uPt , ct]) of RNN:
gt = sigmoid(Wg[u P t , ct])
[uPt , ct] ∗ = gt [uPt , ct] (6)
Different from the gates in LSTM or GRU, the additional gate is based on the current passage word and its attention-pooling vector of the question, which focuses on the relation between the question and current passage word. The gate effectively model the phenomenon that only parts of the passage are relevant to the question in reading comprehension and question answering. [uPt , ct] ∗ is utilized in subsequent calculations instead of [uPt , ct]. We call this gated attention-based recurrent networks. It can be applied to variants of RNN, such as GRU and LSTM. We also conduct experiments to show the effectiveness of the additional gate on both GRU and LSTM.
3.3 Self-Matching Attention
Through gated attention-based recurrent networks, question-aware passage representation {vPt }nt=1 is generated to pinpoint important parts in the passage. One problem with such representation is that it has very limited knowledge of context. One answer candidate is often oblivious to important
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
cues in the passage outside its surrounding window. Moreover, there exists some sort of lexical or syntactic divergence between the question and passage in the majority of SQuAD dataset (Rajpurkar et al., 2016). Passage context is necessary to infer the answer. To address this problem, we propose directly matching the question-aware passage representation against itself. It dynamically collects evidence from the whole passage for words in passage and encodes the evidence relevant to the current passage word and its matching question information into the passage representation hPt :
hPt = BiRNN(h P t−1, [v P t , ct]) (7)
where ct = att(vP , vPt ) is an attention-pooling vector of the whole passage (vP ):
stj = v Ttanh(WPv v P j + W P̃ v v P t )
ati = exp(s t i)/Σ n j=1exp(s t j)
ct = Σ n i=1a t iv P i (8)
An additional gate as in gated attention-based recurrent networks is applied to [vPt , ct] to adaptively control the input of RNN.
Self-matching extracts evidence from the whole passage according to the current passage word and question information.
3.4 Output Layer
We follow Wang and Jiang (2016b) and use pointer networks (Vinyals et al., 2015) to predict the start and end position of the answer. In addition, we use an attention-pooling over the question representation to generate the initial hidden vector for the pointer network. Given the passage representation {hPt }nt=1, the attention mechanism is utilized as a pointer to select the start position (p1) and end position (p2) from the passage, which can be formulated as follows:
stj = v Ttanh(WPh h P j + W a hh a t−1)
ati = exp(s t i)/Σ n j=1exp(s t j)
pt = arg max(at1, . . . , a t n) (9)
Here hat−1 represents the last hidden state of the answer recurrent network (pointer network). The input of the answer recurrent network is the attention-pooling vector based on current predicted probability at:
ct = Σ n i=1a t ih P i
hat = RNN(h a t−1, ct) (10)
When predicting the start position, hat−1 represents the initial hidden state of the answer recurrent network. We utilize the question vector rQ as the initial state of the answer recurrent network. rQ = att(uQ, V Qr ) is an attention-pooling vector of the question based on the parameter V Qr :
sj = v Ttanh(WQu u Q j + W Q v V Q r )
ai = exp(si)/Σ m j=1exp(sj)
rQ = Σmi=1aiu Q i (11)
To train the network, we minimize the sum of the negative log probabilities of the ground truth start and end position by the predicted distributions.
4 Experiment
4.1 Implementation Details
We specially focus on the SQuAD dataset to train and evaluate our model, which has garnered a huge attention over the past few months. SQuAD is composed of 100,000+ questions posed by crowd workers on 536 Wikipedia articles. The dataset is randomly partitioned into a training set (80%), a development set (10%), and a test set (10%). The answer to every question is a segment of the corresponding passage.
We use the tokenizer from Stanford CoreNLP (Manning et al., 2014) to preprocess each passage and question. The Gated Recurrent Unit (Cho et al., 2014) variant of LSTM is used throughout our model. For word embedding, we use pretrained case-sensitive GloVe embeddings2 (Pennington et al., 2014) for both questions and passages, and it is fixed during training; We use zero vectors to represent all out-of-vocab words. We utilize 1 layer of bi-directional GRU to compute character-level embeddings and 3 layers of bi-directional GRU to encode questions and passages, the gated attention-based recurrent network for question and passage matching is also encoded bidirectionally in our experiment. The hidden vector length is set to 75 for all layers. We also apply dropout (Srivastava et al., 2014) between layers with a dropout rate of 0.2. The model is optimized with AdaDelta (Zeiler, 2012) with an initial learning rate of 1.
2Downloaded from http://nlp.stanford.edu/ data/glove.840B.300d.zip.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Dev Set Test Set Single model EM / F1 EM / F1 LR Baseline (Rajpurkar et al., 2016) 40.0 / 51.0 40.4 / 51.0 Dynamic Chunk Reader (Yu et al., 2016) 62.5 / 71.2 62.5 / 71.0 Match-LSTM with Ans-Ptr (Wang and Jiang, 2016b) 64.1 / 73.9 64.7 / 73.7 Dynamic Coattention Networks (Xiong et al., 2016) 65.4 / 75.6 66.2 / 75.9 RaSoR (Lee et al., 2016) 66.4 / 74.9 - / - BiDAF (Seo et al., 2016) 68.0 / 77.3 68.0 / 77.3 jNet (USTC&National Research Council Canada&York University) - / - 68.7 / 77.4 Multi-Perspective Matching (Wang et al., 2016) - / - 68.9 / 77.8 Gated Self-Matching Networks 71.1 / 79.5 71.3 / 79.7 Ensemble model Fine-Grained Gating (Yang et al., 2016) 62.4 / 73.4 62.5 / 73.3 Match-LSTM with Ans-Ptr (Wang and Jiang, 2016b) 67.6 / 76.8 67.9 / 77.0 RaSoR (Lee et al., 2016) 68.2 / 76.7 - / - Dynamic Coattention Networks (Xiong et al., 2016) 70.3 / 79.4 71.6 / 80.4 BiDAF (Seo et al., 2016) 73.3 / 81.1 73.3 / 81.1 Multi-Perspective Matching (Wang et al., 2016) - / - 73.8 / 81.3 Gated Self-Matching Networks 75.6 / 82.8 75.9 / 82.9 Human Performance (Rajpurkar et al., 2016) 80.3 / 90.5 77.0 / 86.8 FastQA* (German Research Center for Artificial Intelligence) - / - 68.4 / 77.1 FastQAExt* (German Research Center for Artificial Intelligence) - / - 70.8 / 78.9
Table 2: The performance of our gated self-matching networks and competing approaches. * indicates that there is no explicit label to suggest whether the model is a single or ensemble model.
Single Model EM / F1 Gated Self-Matching (GRU) 71.1 / 79.5 -Character embedding 70.3 / 78.9 -Gating 67.9 / 77.1 -Self-Matching 66.9 / 76.4 -Gating, -Self-Matching 65.2 / 74.7
Table 3: Ablation tests of single model on the SQuAD dev set.
4.2 Main Results
Two metrics are utilized to evaluate model performance: Exact Match (EM) and F1 score. EM measures the percentage of the prediction that matches one of the ground truth answers exactly. F1 measures the overlap between the prediction and ground truth answers which takes the maximum F1 over all of the ground truth answers. The scores on dev set are evaluated by the official script3. Since the test set is hidden, we are required to submit the model to Stanford NLP group to obtain the test scores.
Table 2 shows exact match and F1 scores on the 3Downloaded from http://stanford-qa.com
dev and test set4 of our model and competing approaches. As we can see, our method clearly outperforms the baseline and several strong state-ofthe-art systems for both single model and ensembles.
4.3 Ablation Study
We do ablation tests on the dev set to analyze the contribution of components of gated self-matching networks. As illustrated in Table 3, the gated attention-based recurrent network (GARNN) and self-matching attention mechanism positively contribute to the final results of gated self-matching
4Extracted from SQuAD leaderboard http: //stanford-qa.com on Feb. 6, 2017.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Figure 2: Part of the attention matrices for self-matching. Each row is the attention weights of the whole passage for the current passage word. The darker the color is the higher the weight is. Some key evidence relevant to the question-passage tuple is more encoded into answer candidates.
networks. Removing self-matching results in 4.2 point EM drop, which reveals that information in the passage plays an important role. Characterlevel embeddings contribute towards the model’s performance since it can better handle out-ofvocab or rare words. To show the effectiveness of GARNN for variant RNNs, we conduct experiments on the base model (Wang and Jiang, 2016b) of different variant RNNs. The base model match the question and passage via a variant of attentionbased recurrent network (Wang and Jiang, 2016a), and employ pointer networks to predict the answer. Character-level embeddings are not utilized. As shown in Table 4, the gate introduced in question and passage matching layer is helpful for both GRU and LSTM on the SQuAD dataset.
5 Discussion
5.1 Encoding Evidence from Passage
To show the ability of the model for encoding evidence from the passage, we draw the alignment of the passage against itself in self-matching. The attention weights are shown in Figure 2, in which the darker the color is the higher the
weight is. We can see that key evidence aggregated from the whole passage is more encoded into the answer candidates. For example, the answer “Egg of Columbus” pays more attention to the key information “Tesla”, “device” and the lexical variation word “known” that are relevant to the question-passage tuple. The answer “world classic of epoch-making oratory” mainly focuses on the evidence “Michael Mullet”, “speech” and lexical variation word “considers”. For other words, the attention weights are more evenly distributed between evidence and some irrelevant parts. Selfmatching do adaptively aggregate evidence for words in passage.
5.2 Result Analysis
To further analyse the model’s performance, we show the exact match and F1 score for different question types (Figure 3(a)), different answer lengths (Figure 3(b)), different passage lengths (Figure 3(c)) and different question lengths (Figure 3(d)). As we can see, the type of “what” takes up a majority of questions. Our model is better at “when” and “who” questions, but poorly on “why”
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
(a) (b)
(c) (d)
Figure 3: Model performance on different question types (a), different answer lengths (b), different passage lengths (c), different question lengths (d). The point on the x-axis of figure (c) and (d) represent the datas whose passages length or questions length are between the value of current point and last point.
questions. From the Graph 3(b), the majority of answers are in short length. With the increase of answer length, the performance of our model obviously drops. Besides, the gap between exact match and F1 widens as answer length increases, which indicates our model could locate the core of the answer to some extent. From Graph 3(c) and 3(d), the majority of passages and questions are not too long. Moreover, we discover that the performance remains stable with the increase in length, the obvious drop in longer passage is mainly because the proportion is too small. Our model is largely agnostic to long passages and focuses on important part of the passage.
6 Related Work
Reading Comprehension and Question Answering Dataset Benchmark datasets play an important role in recent progress in reading comprehension and question answering research. Existing datasets can be classified into two categories according to whether they are manually labeled. Those that are labeled by humans are always in high quality (Richardson et al., 2013; Berant et al., 2014; Yang et al., 2015), but are too small for training modern data-intensive models. Those that
are automatically generated from natural occurring data can be very large (Hill et al., 2016; Hermann et al., 2015), which allow the training of more expressive models. However, they are in cloze style, in which the goal is to predict the missing word (often a named entity) in a passage. Moreover, Chen et al. (2016) have shown that the CNN / Daily News dataset (Hermann et al., 2015) requires less reasoning than previously thought, and conclude that performance is almost saturated.
Different from above datasets, the SQuAD provides a large and high-quality dataset. The answers in SQuAD often include non-entities and can be much longer phrase, which is more challenging than cloze-style datasets. Moreover, Rajpurkar et al. (2016) show that the dataset retains a diverse set of answers and requires different forms of logical reasoning, including multi-sentence reasoning. MS MARCO (Nguyen et al., 2016) is also a large-scale dataset. The questions in the dataset are real anonymized queries issued through Bing or Cortana and the passages are related web pages. For each question in the dataset, several related passages are provided. However, the answers are human generated, which is different from SQuAD where answers must be a span of the passage.
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
End-to-end Neural Networks for Reading Comprehension Along with cloze-style datasets, several powerful deep learning models (Hermann et al., 2015; Hill et al., 2016; Chen et al., 2016; Kadlec et al., 2016; Sordoni et al., 2016; Cui et al., 2016; Trischler et al., 2016; Dhingra et al., 2016; Shen et al., 2016) have been introduced to solve this problem. Hermann et al. (2015) first introduce attention mechanism into reading comprehension. Hill et al. (2016) propose a windowbased memory network for CBT dataset. Kadlec et al. (2016) introduce pointer networks with one attention step to predict the blanking out entities. Sordoni et al. (2016) propose an iterative alternating attention mechanism to better model the links between question and passage. Trischler et al. (2016) solve cloze-style question answering task by combining an attentive model with a reranking model. Dhingra et al. (2016) propose iteratively selecting important parts of the passage by a multiplying gating function with the question representation. Cui et al. (2016) propose a two-way attention mechanism to encode the passage and question mutually. Shen et al. (2016) propose iteratively inferring the answer with a dynamic number of reasoning steps and is trained with reinforcement learning.
Neural network-based models demonstrate the effectiveness on the SQuAD dataset. Wang and Jiang (2016b) combine match-LSTM and pointer networks to produce the boundary of the answer. Xiong et al. (2016) and Seo et al. (2016) employ variant coattention mechanism to match the question and passage mutually. Xiong et al. (2016) propose a dynamic pointer network to iteratively infer the answer. Yu et al. (2016) and Lee et al. (2016) solve SQuAD by ranking continuous text spans within passage. Yang et al. (2016) present a fine-grained gating mechanism to dynamically combine word-level and character-level representation and model the interaction between questions and passages. Wang et al. (2016) propose matching the context of passage with the question from multiple perspectives.
Different from the above models, we introduce self-matching attention in our model. It dynamically refines the passage representation by looking over the whole passage and aggregating evidence relevant to the current passage word and question, allowing our model make full use of passage information. Weightedly attending to word context
has been proposed in several works. Ling et al. (2015) propose considering window-based contextual words differently depending on the word and its relative position. Cheng et al. (2016) propose a novel LSTM network to encode words in a sentence which considers the relation between the current token being processed and its past tokens in the memory. Parikh et al. (2016) apply this method to encode words in a sentence according to word form and its distance. Since passage information relevant to question is more helpful to infer the answer in reading comprehension, we apply self-matching based on question-aware representation and gated attention-based recurrent networks. It helps our model mainly focus on question-relevant evidence in the passage and dynamically look over the whole passage to aggregate evidence.
Another key component of our model is the attention-based recurrent network, which has demonstrated success in a wide range of tasks. Bahdanau et al. (2014) first propose attentionbased recurrent networks to infer word-level alignment when generating the target word. Hermann et al. (2015) introduce word-level attention into reading comprehension to model the interaction between questions and passages. Rocktäschel et al. (2015) and Wang and Jiang (2016a) propose determining entailment via word-by-word matching. The gated attention-based recurrent network is a variant of attention-based recurrent network with an additional gate to model the fact that passage parts are of different importance to the particular question for reading comprehension and question answering.
7 Conclusion
In this paper, we present gated self-matching networks for reading comprehension and question answering. We introduce the gated attentionbased recurrent networks and self-matching attention mechanism to obtain representation for the question and passage, and then use the pointernetworks to locate answer boundaries. Our model achieves state-of-the-art results on the SQuAD dataset, outperforming several strong competing systems. As for future work, we are applying the gated self-matching networks to other reading comprehension and question answering datasets, such as the MS MARCO dataset (Nguyen et al., 2016).
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

This paper presents the gated self-matching network for reading comprehension style question answering. There are three key components in the solution:   (a) The paper introduces the gated attention-based recurrent network to obtain the question-aware representation for the passage. Here, the paper adds an additional gate to attention-based recurrent networks to determine the importance of passage parts and attend to the ones relevant to the question. Here they use word as well as character embeddings to handle OOV words. Overall, this component is inspired from Wang and Jiang 2016.  (b) Then the paper proposes a self-matching attention mechanism to improve the representation for the question and passage by looking at wider passage context necessary to infer the answer. This component is completely novel in the paper.  (c) At the output layer, the paper uses pointer networks to locate answer boundaries. This is also inspired from Wang and Jiang 2016  Overall, I like the paper and think that it makes a nice contribution.  - Strengths:  The paper clearly breaks the network into three component for descriptive purposes, relates each of them to prior work and mentions its novelties with respect to them. It does a sound empirical analysis by describing the impact of each component by doing an ablation study. This is appreciated.  The results are impressive!  - Weaknesses:  The paper describes the results on a single model and an ensemble model. I could not find any details of the ensemble and how was it created. I believe it might be the ensemble of the character based and word based model. Can the authors please describe this in the rebuttal and the paper.  - General Discussion:  Along with the ablation study, it would be nice if we can have a qualitative analysis describing some example cases where the components of gating, character embedding, self embedding, etc. become crucial ... where a simple model doesn't get the question right but adding one or more of these components helps. This can go in some form of appendix or supplementary.


Keyphrase provides highly-summative information that can be effectively used for understanding, organizing and retrieving text content. Though previous studies have provided many workable solutions for automated keyphrase extraction, they commonly divide the to-besummarized content into multiple text chunks, then rank and select the most meaningful ones. These approaches can neither identify keyphrases that do not appear in the text, nor capture the real semantic meaning behind the text. We propose a generative model for keyphrase prediction with an encoder-decoder framework, which can effectively overcome the above drawbacks. We name it as deep keyphrase generation since it attempts to capture the deep semantic meaning of content with a deep learning method. Empirical analysis on six datasets demonstrates that our proposed model not only achieves a significant performance boost on extracting keyphrases that appear in the source text, but also generates absent keyphrases based on the semantic meaning of the text.
1 Introduction
Keyphrase or keyword is a piece of short and summative content that expresses the main semantic meaning of a long text. A typical use of keyphrase or keyword is in scientific publications, to provide the core information of a paper. We use the term “keyphrase”, interchangeable as “keyword”, in the rest of this paper, as it implies that it may contain multiple words. High-quality keyphrases can facilitate the understanding, organizing and accessing of document content. As a result, many stud-
ies have devoted to studying the ways of automatic extracting keyphrases from textual content (Liu et al., 2009; Medelyan et al., 2009a; Witten et al., 1999). Due to the public accessibility, many scientific publication datasets are often used as the test beds for keyphrase extraction algorithms. Therefore, this study also focuses on extracting keyphrases from scientific publications.
Automatically extracting keyphrases from a document is called Keypharase Extraction, and it has been widely exploited in many applications, such as information retrieval (Jones and Staveley, 1999), text summarization (Zhang et al., 2004), text categorization (Hulth and Megyesi, 2006), and opinion mining (Berend, 2011). Most of the existing keyphrase extraction algorithms addressed this problem through two steps (Liu et al., 2009; Tomokiyo and Hurst, 2003). The first step is to acquire a list of keyphrase candidates. Researchers have tried to use n-grams or noun phrases with certain part-of-speech patterns for identifying the potential candidates (Hulth, 2003; Le et al., 2016; Liu et al., 2010; Wang et al., 2016). The second step is to rank candidates regarding their importance to the document, either through supervised or unsupervised machine learning methods with a set of manuallydefined features (Frank et al., 1999; Liu et al., 2009, 2010; Kelleher and Luz, 2005; Matsuo and Ishizuka, 2004; Mihalcea and Tarau, 2004; Song et al., 2003; Witten et al., 1999).
There are two major drawbacks for the above keyphrase extraction approaches.
Firstly, they can only extract the keyphrases that appear in the source text, whereas they fail at predicting the meaningful keyphrases with a slightly different sequential order or using synonym words. However, it is common in a scientific publication where authors assign keyphrases based on the semantic meaning instead of follow-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
ing the written content in the publication. In this paper, we denote the phrases that do not match any contiguous subsequence of source text as absent keyphrases, and the ones that fully match a part of the text as present keyphrases. Table 1 shows the proportion of present and absent keyphrases from the document abstract in four commonlyused datasets, where we observe large portions of absent keyphrases in all the datasets. The absent keyphrases cannot be extracted through previous approaches, which further urges the development of a more powerful keyphrase prediction model.
Secondly, when ranking phrase candidates, previous approaches often adopted the machine learning features such as TF-IDF and PageRank. However, these features only target to detect the importance of each word in the document based on the statistics of word occurrence and co-occurrence, whereas they can hardly reveal the semantics behind the document content.
Table 1: Proportion of the present keyphrases and absent keyphrases in four public datasets
Dataset # Keyphrase % Present % Absent Inspec 19,275 55.69 44.31
Krapivin 2,461 44.74 52.26 NUS 2,834 67.75 32.25
SemEval 12,296 42.01 57.99
To overcome the limitations of previous studies, we re-examine the process of Keyphrase Prediction, about how real human annotators would assign keyphrases. Given a document, human annotators will firstly read the text to get a basic understanding of the content, then they try to digest its essential content and summarize into keyphrases. Their generation of keyphrases relies on the understanding of the content, which not necessarily to be the words that occurred in the source text. For example, when human annotators see “Latent Dirichlet Allocation” in the text, they could write down “topic modeling” and/or “text mining” as possible keyphrases. In addition to the semantic understanding, human annotators might also go back and picks up the most important parts based on syntactic features. For example, the phrases following “we propose/apply/use” are supposed to be important in the text. As a result, a better keyphrase prediction model should understand the semantic meaning of the content, as well as capture the contextual features.
To effectively capture semantic and syntactic features, we utilize recurrent neural networks (RNN) (Cho et al., 2014; Gers and Schmidhuber, 2001) to compress the semantic information in the given text into a dense vector (i.e., semantic understanding). Furthermore, we incorporate a copy mechanism (Gu et al., 2016) to equip our model with the capability of finding important parts based on language syntax (i.e., syntactic understanding). Thus, our model can generate keyphrases based on the understanding of the text, no matter whether the keyphrases are present in the text or not; meanwhile, it does not lose important in-text information.
The contribution of this paper is in three-fold: a) we propose to apply an RNN-based generative model to keyphrase prediction, and we also incorporate a copy mechanism in RNN, which enables the model to successfully predict rarely-occurred phrases; b) this is the first work concerning about the problem of absent keyphrase prediction for scientific publications, and our model recalls up to 20% of absent keyphrases; and c) we conduct a comprehensive comparison against six important baselines on a broad range of datasets, and the results show that our proposed model significantly outperforms existing supervised and unsupervised extraction methods.
In the remainder of this paper, we firstly review the related work in Section 2. Then we elaborate the proposed model in Section 3. After that, we present the experiment setting in Section 4 and results in Section 5, followed by our discussion in Section 6. Section 7 concludes the paper.
2 Related Work
2.1 Automatic Keyphrase Extraction
Keyphrase provides a succinct and accurate way of describing a subject or a subtopic in a document. A number of extraction algorithms have been proposed, and typically the process of extracting can be broken down into two steps.
The first step is to generate a list of phrase candidates with heuristic methods. As these candidates are prepared for further filtering, a considerable amount of candidates are produced in this step to increase the possibility that most of the correct keyphrases are kept. The primary ways of extracting candidates include retaining word sequences that match certain part-of-speech tag patterns (e.g., nouns, adjectives) (Liu et al., 2011;
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Wang et al., 2016; Le et al., 2016), and extracting important n-grams or noun phrases (Hulth, 2003; Medelyan et al., 2008).
The second step is to score each candidate phrase regarding its likelihood of being a keyphrase in the given document. The top-ranked candidates are returned as keyphrases. Both supervised and unsupervised machine learning methods are widely employed here. For supervised methods, this task is solved as a binary classification problem, and various of learning methods and features have been explored (Frank et al., 1999; Witten et al., 1999; Hulth, 2003; Medelyan et al., 2009b; Lopez and Romary, 2010; Gollapalli and Caragea, 2014). As for the unsupervised approaches, primary ideas include finding the central nodes in text graph (Mihalcea and Tarau, 2004; Grineva et al., 2009), detecting representative phrases from topical clusters (Liu et al., 2009, 2010) and so on.
Aside from the commonly adopted two steps process, another two previous studies realized the keyphrase extraction in entirely different ways. Tomokiyo and Hurst (2003) applied two language models to measure the phraseness and informativeness of phrases. Liu et al. (2011) share the most similar idea to our work. They used a word alignment model, which learns the translation from the documents to the keyphrases. This approach alleviates the problem of vocabulary gap between source and target to a certain degree. However, this translation model can hardly deal with semantic meaning. Additionally, they trained the model with the target of title/summary to enlarge the number of training samples, which may diverge away from the real objective of generating keyphrases.
2.2 Encoder-Decoder Model
The RNN Encoder-Decoder model (also referred as Sequence-to-Sequence Learning) is an end-toend approach. It was firstly introduced by Cho et al. (2014) and Sutskever et al. (2014) to solve translation problems. As it provides a powerful tool for modeling variable-length sequence in an end-to-end fashion, it fits many Natural Language Processing tasks and soon achieves great successes (Rush et al., 2015; Vinyals et al., 2015; Serban et al., 2016).
Different strategies have been explored to improve the performance of Encoder-Decoder model. The attention mechanism (Bahdanau et al.,
2014) is a soft alignment approach that allows the model to automatically locate the relevant input component. In order to make use of the important information in the source text, some studies sought ways to copy certain parts of content from the source text and paste them into target text (Allamanis et al., 2016; Gu et al., 2016; Zeng et al., 2016). There exists a discrepancy between the optimizing objective during training and the metrics during evaluation. A few studies attempted to eliminate this discrepancy by incorporating new training algorithm (Marc’Aurelio Ranzato et al., 2016) or modifying optimizing objective(Shen et al., 2016).
3 Methodology
This section will introduce in detail our proposed deep keyphrase generation. Firstly, the task of keyphrase generation is defined, followed by the overview of how we apply the RNN EncoderDecoder model. Details of the framework as well as the copy mechanism will be introduced in Section 3.3 and 3.4.
3.1 Problem Definition
Given a keyphrase dataset consisting of N data samples, the i-th data sample (x(i),p(i)) contains one source text x(i), and Mi target keyphrases p(i) = (p(i,1),p(i,2), . . . ,p(i,Mi)). Both the source text x(i) and keyphrase p(i,j) are sequences of words:
x(i) = x (i) 1 , x (i) 2 , . . . , x (i) L xi
p(i,j) = y (i,j) 1 , y (i,j) 2 , . . . , y (i,j) L p(i,j)
Lx(i) and Lp(i,j)denotes the length of word sequence of x(i) and p(i,j) respectively.
Now for each data sample, there are one source text sequence and multiple target phrase sequences. To apply the RNN Encoder-Decoder model, the data need to be converted into textkeyphrase pairs which contain only one source sequence and one target sequence. One simple way is to split the data sample (x(i),p(i)) intoMi pairs: (x(i),p(i,1)), (x(i),p(i,2)), . . . , (x(i),p(i,Mi)). Then the Encoder-Decoder model is ready to be applied to learn the mapping from source sequence to target sequence. For the purpose of simplicity, (x,y) is used to denote each data pair in the rest of this section, where x is the word sequence of a source text and y is the word sequence of its keyphrase.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
3.2 Encoder-Decoder Model
The basic idea of our keyphrase generation model is to compress the content of source text into a hidden representation with encoder, and generate corresponding keyphrases by the decoder based on the representation . Both the encoder and decoder are implemented with recurrent neural networks (RNN).
The encoder RNN converts the variable-length input sequence x = (x1, x2, ..., xT ) into a set of hidden representation h = (h1, h2, . . . , hT ), by iterating the following equations along the time t:
ht = f (xt,ht−1) (1)
where f is a non-linear function. And we get the context vector c, acting as the representation of the whole input x, through a non-linear function q.
c = q(h1, h2, ..., hT ) (2)
The decoder is another RNN, decompresses the context vector and generates a variable-length sequence y = (y1, y2, ..., yT ′) word by word, through a conditional language model:
st = g(yt−1, st−1, c) p(yt|y1,...,t−1,x) = g(yt−1, st, c) (3)
where st is the hidden state of decoder RNN at time t. The non-linear function g is a softmax classifier which outputs the probabilities of all the words in the vocabulary. yt is the predicted word at time t, by taking the word with largest probability after g(·).
The encoder and decoder networks are trained jointly to maximize the conditional probability of the target sequence given a source sequence. After training, we use the beam search to generate phrases and a max heap is maintained to get the predictions with highest probabilities.
3.3 Details of Encoder and Decoder
A bidirectional Gated Recurrent Unit (GRU) is applied as our encoder to replace the simple recurrent neural network. Previous studies (Bahdanau et al., 2014; Cho et al., 2014) indicate it can provide a general better performance language modeling than simple RNN and a simpler structure than Long Short-term Memory Networks(Hochreiter and Schmidhuber, 1997). So the above non-linear function f is replaced by the GRU function (see in (Cho et al., 2014)).
Another forward GRU is utilized as the decoder. In addition, an attention mechanism is adopted to improve the performance. The attention mechanism was firstly introduced by Bahdanau et al. (2014) to make the model focus on the important parts in input dynamically. The context vector c is computed as a weighted sum of hidden representation h = (h1, . . . , hT ):
ci = T∑ j=1 αijhj αij = exp(a(si−1, hj))∑T k=1 exp(a(si−1, hk))
(4)
where a(si−1, hj) is a soft alignment function that measures the similarity between si−1 and hj , namely to which degree the inputs around position j and the output at position i match.
3.4 Copy Mechanism
To ensure the quality of learned representation and reduce the size of vocabulary, typically the RNN model only considers a certain number of frequent words (e.g. 30,000 words in (Cho et al., 2014)), but a large amount of long-tail words are simply ignored. Therefore the RNN is not able to predict any keyphrase which contains out-of-vocabulary words. Actually, the important phrases can also be identified by their syntactic and location features, even though we don’t know the meanings. The copy mechanism is one feasible solution that enables RNN to predict unknown words based on contextual features.
By incorporating the copy mechanism, the probability of predicting each new word yt would consist of two parts. The first term is the probability of generating the term (see Equation 3) and the second one is the probability of copying it from source text:
p(yt|y1,...,t−1,x) = pg(yt|y1,...,t−1,x) + pc(yt|y1,...,t−1,x) (5)
Similar to attention mechanism, the copy mechanism pinpoints the appearance of yt−1 in source text, and use its location information (ρtτ ) to compute the weighted hidden representation (ζ(yt−1)) of input x.
ζ(yt−1) = TS∑ τ=1 ρtτhτ (6)
ρtτ = { 1 K p(xτ , c|st−1) xτ = yt−1 0 Otherwise
(7)
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
where K is the normalization term. Subsequently, we obtain the probabilities of copying pc(yt|y1,...,t−1) following the Equation 3, in which the target words are replaced by the words in source text, and the vector of yt−1 is replaced by [e(yt−1), ζ(yt−1)]
T , where the e(yt−1) is the embedding of last predicted word.
4 Experiment Settings
This section starts with discussing how we design our evaluation experiments, followed by the description of training and testing datasets. Then, we introduce evaluation metrics and baselines.
4.1 Training Dataset
There are several publicly-available datasets for evaluating keyphrase generation. The largest one came from Krapivin et al. (2008), which contained 2,304 scientific publications. However, this amount of data is unable to train a robust recurrent neural network model. In fact, there are millions of scientific papers available online, each of which contains the keyphrases assigned by authors. Therefore, we collected a large amount of high-quality scientific metadata in Computer Science domain from various online digital libraries, including ACM Digital Library, ScienceDirect, Wiley, Web of Science, and so on. In total, we obtained 567,830 articles after removing duplicates and overlaps, which is 200 times larger than the one of Krapivin et al. (2008). Note that our model is only trained on 527,830 articles since 40,000 publications are held out for training baselines and for building a test dataset.
4.2 Testing Datasets
For evaluating the proposed model more comprehensively, four widely-adopted scientific publication datasets are used. In addition, since these datasets only contain few hundreds or thousands of publications, we contribute a new testing dataset KP20k with a much larger number of scientific articles. We take the title and abstract as the source text. Each dataset is described in details in the below text.
– Inspec (Hulth, 2003): This dataset provides 2,000 paper abstracts. We adopt the 500 testing papers and the corresponding uncontrolled keyphrases for evaluation, and the remaining 1,500 papers are used for training the supervised baseline models.
– Krapivin (Krapivin et al., 2008): This dataset provides 2,304 papers with full-text and author-assigned keyphrases. However, the author did not mention how to split testing data, so we simply select the first 400 papers in alphabetical order as the testing data, and the remaining papers are used to train the supervised baselines.
– NUS (Nguyen and Kan, 2007): We use the author-assigned keyphrases and treat all 211 papers as the testing data. Since the NUS dataset did not specifically mention the ways of splitting training and testing data, the results of the supervised baseline models are obtained through a five-fold cross validation.
– SemEval-2010 (Kim et al., 2010): 288 articles are collected from ACM Digital Library. 100 articles are used for testing and the rest are for training supervised baselines.
– KP20k: We build a new testing dataset that contains the title, abstract and keyphrases of 20,000 scientific articles in Computer Science. They are randomly selected from our obtained 567,830 articles. For the supervised baselines, another 20,000 articles are randomly selected for training. Therefore, our proposed model is trained on 527,830 that holds out these 40,000 articles.
4.3 Implementation Details
In total, there are 2,780,316 〈text, keyphrase〉 pairs for training, in which text refers to the concatenation of the title and abstract of a publication, and keyphrase indicates an author-assigned keyword. The text pre-processing steps including tokenization, lowercasing and replacing all digits with symbol 〈digit〉 are applied. Two encoder-decoder models are trained, one with only attention mechanism (RNN) and one with both attention and copy mechanism enabled (CopyRNN). For both models, we choose the top 50,000 frequently-occurred words as our vocabulary, the dimension of embedding is set to 150, and the dimension of hidden layers is set to 300. Models are optimized using Adam (Kingma and Ba, 2014) with initial learning rate = 10−4, gradient clipping = 0.1 and dropout rate = 0.5. The max depth of beam search is set to 6, and the beam size is set to 200. In the generation of keyphrases, we find that the model tends to assign higher probabilities for shorter keyphrases,
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
whereas most of keyphrases contain more than two words. To resolve this problem, we apply a simple heuristic by preserving only the first singleword phrase (with the highest generating probability) and removing the rest.
4.4 Baseline Models
Four unsupervised algorithms (Tf-Idf, TextRank (Mihalcea and Tarau, 2004), SingleRank (Wan and Xiao, 2008), ExpandRank (Wan and Xiao, 2008)) and two supervised algorithms (KEA (Witten et al., 1999) and Maui (Medelyan et al., 2009a)) are adopted as baselines. We set up the four unsupervised methods following the optimal settings in (Hasan and Ng, 2010) and the two supervised methods following the default setting as specified in their papers.
4.5 Evaluation Metric
Three evaluation metrics, the macro-averaged precision, recall and F-measure (F1) are employed for measuring the algorithm performance. Follow the standard definition, precision is defined as the number of correctly-predicted keyphrases over the number of all predicted keyphrases, recall is computed by the number of correctly-predicted keyphrases over the total data records. Note that, when determining the match of two keyphrases, we use Porter Stemmer for pre-processing.
5 Results and Analysis
We conduct an empirical study on three different tasks to evaluate our model.
5.1 Predicting Present Keyphrases
This is the same as the keyphrase extraction task in prior studies, in which we would like to analyze how well our proposed model perform on the commonly-defined task. To make a fair comparison, we only consider the present keyphrases for evaluation in this task. Table 2 provides the performances of the six baseline models, as well as our proposed models (i.e., RNN and CopyRNN). For each method, the table lists its F-measure at top 5 and top 10 predictions on the five datasets. The best scores are highlighted in bold and the underlines indicate the second best performances.
The results show that the four unsupervised models (Tf-idf, TextTank, SingleRank and ExpandRank) perform robust across different datasets. The ExpandRank fails to return any result on the KP20k dataset due to its high time com-
plexity. The measures on NUS and SemEval here are higher than the ones reported in (Hasan and Ng, 2010) and (Kim et al., 2010), probably because we utilized the paper abstract instead of the full-text for training, which may filter out some noisy information. The performances of the two supervised models (i.e., Maui and KEA) are unstable on some datasets, but Maui achieves the best performances on three datasets among all the baseline models.
As for our proposed keyphrase prediction approaches, the RNN model with the attention mechanism does not perform as well as we expected. It might be because the RNN model only concerns on finding the hidden semantics behind the text, which may tend to generate keyphrases or words that are too general and may not necessarily referring to the source text. In addition, it fails to recall keyphrases that contain out-of-vocabulary words (since the RNN model only takes the top 50,000 words in vocabulary). This indicates that a pure generative model may not fit the extraction task, and we need to further link back to the language usage in the source text. Indeed, the copyRNN model, by considering more contextual information, significantly outperforms not only the RNN model but also all baselines, exceeding the best baselines by more than 20% on average. This result demonstrates the importance of source text for extraction task. Besides, nearly 2% of all the correct predictions contain out-of-vocabulary words.
The example in Figure 1(a) shows the result of predicted present keyphrases by RNN and CopyRNN for an article about video search. We see that both models can generate phrases that related to the topic of information retrieval and video. However most of RNN predictions are high-level terminologies, which are too general to be selected as keyphrases. The CopyRNN, on the other hand, predicts more detailed phrases like “video metadata” and “integrated ranking”. An interesting bad case is, “rich content” is coordinate with a keyphrase “video metadata”, and the CopyRNN puts it into prediction mistakenly.
5.2 Predicting Absent Keyphrases
As stated, one important motivation for this work is that we are interested in the proposed model’s capability for predicting absent keyphrases based on the “understanding” of content. It is worth noting that such prediction is a very challenging task, and, to the best of our knowledge, no ex-
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Method Inspec Krapivin NUS SemEval KP20k
F1@5 F1@10 F1@5 F1@10 F1@5 F1@10 F1@5 F1@10 F1@5 F1@10
Tf-Idf 0.221 0.313 0.129 0.160 0.136 0.184 0.128 0.194 0.102 0.126 TextRank 0.223 0.281 0.189 0.162 0.195 0.196 0.176 0.187 0.175 0.147
SingleRank 0.214 0.306 0.189 0.162 0.140 0.173 0.135 0.176 0.096 0.119 ExpandRank 0.210 0.304 0.081 0.126 0.132 0.164 0.139 0.170 N/A N/A
Maui 0.040 0.042 0.249 0.216 0.249 0.268 0.044 0.039 0.270 0.230 KEA 0.098 0.126 0.110 0.152 0.069 0.084 0.025 0.026 0.171 0.154
RNN 0.085 0.064 0.135 0.088 0.169 0.127 0.157 0.124 0.179 0.189 CopyRNN 0.278 0.342 0.311 0.266 0.334 0.326 0.293 0.304 0.333 0.262
Table 2: The performance of predicting present keyphrase of various models on five benchmark datasets
isting methods can handle this task. Therefore, we only provide the RNN and copyRNN performances in the discussion of the results of this task. Here, we evaluate the performance with the recall of top 10 and top 50 results, to see how many absent keyphrases can be correctly predicted. We use the absent keyphrases in the testing datasets for evaluation.
Dataset RNN CopyRNN R@10 R@50 R@10 R@50
Inspec 0.031 0.061 0.047 0.100 Krapivin 0.095 0.156 0.113 0.202
NUS 0.050 0.089 0.058 0.116 SemEval 0.041 0.060 0.043 0.067 KP20k 0.083 0.144 0.125 0.211
Table 3: Absent keyphrases prediction performance of RNN and CopyRNN on five datasets
Table 3 present the recalls of the top 10/50 predicted keyphrases for our RNN and CopyRNN models, in which we observe that the CopyRNN can, on average, recall around 8% (15%) of keyphrases at top 10 (50) predictions. This indicates that, to some extent, both models can capture the hidden semantics behind the textual content and make reasonable predictions. In addition, with the advantage of features from the source text, the CopyRNN model also outperforms the RNN model in this condition, though not improve as much as the present keyphrase extraction task. An example is shown in Figure 1(b), in which we see that two absent keyphrases “video retrieval” and “video indexing” are correctly recalled by both models. The interesting thing is, the term “indexing” does not appear in the text, but the models
may detect the information “index videos” in the first sentence and paraphrase it to the target phrase. And the CopyRNN successfully predicts another two keyphrases by capturing the detailed information from the text (highlighted text segments).
5.3 Transfer to News Articles
The RNN and CopyRNN are supervised models, and they are trained on data in specific domain and writing style. However, with sufficient training on a large-scale dataset, we expect the models to be able to learn universal language features that are effective in other corpus as well. Thus in this task, we will test our model on another type of text, to see whether the model would work when being transferred to a different environment.
We utilize the popular news article dataset DUC-2001 (Wan and Xiao, 2008) for analysis. The dataset consists of 308 news articles and 2,488 manually annotated keyphrases. The result is shown in Table 4, from which we could see that the CopyRNN could extract a portion of correct keyphrases from a unfamiliar text. Compare to the results reported in (Hasan and Ng, 2010), the performance of CopyRNN is better than TextRank (Mihalcea and Tarau, 2004) and KeyCluster (Liu et al., 2009), but lags behind the other
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
three baselines. As transfered to corpus in a complete strange type and domain, the model encounters more unknown words and has to rely more on the syntactic features in the text. In this experiment, the CopyRNN recalls 766 keyphrases. 14.3% of them contain out-of-vocabulary words and many names of persons and places are correctly predicted.
6 Discussion
Our experimental results demonstrate that the CopyRNN model not only performs well on predicting present keyphrases but also has the ability of generating topical relevant keyphrases that are absent in the text. In a broader sense, this model attempts to map a long text (i.e., paper abstract) with representative short text chunks (i.e., keyphrases), which can potentially be applied to improve information retrieval performance by generating high-quality index terms, as well as assisting user browsing by summarizing long documents into short readable phrases.
So far we have examined our model on scientific publications and news articles, demonstrating that our model has the ability to capture universal language patterns and extract key information from unfamiliar texts. We believe that the models have a greater potential to be generalized to other domains and types, like books, online reviews etc., if it is trained on larger data corpus. Also, we directly apply our model, which is trained on publication dataset, into generating keyphrases for news articles without any adaptive training. We believe that with proper training on news data, the
model would make further improvement. Additionally, this work mainly studies the problem of discovering core content from textual materials. Here, the encoder-decoder framework is applied to model language; however, such framework can also be extended to locate the core information on other data resources such as to summarize content from images and videos.
7 Conclusion and Future Work
In this paper, we propose an RNN-based generative model for predicting keyphrase in scientific text. To the best of our knowledge, this is the first application of the encoder-decoder model to keyphrase prediction task. Our model summarizes phrases based the deep semantic meaning of the text and it is able to handle rarely-occurred phrases by incorporating a copy mechanism. Comprehensive empirical studies demonstrate the effectiveness of our proposed model for generating both present and absent keyphrases for different types of text. Our future work may include the following two directions.
– In this work, we only evaluated the performance of proposed model by conducting offline experiments. In the future, we are interested in comparing the model with human annotators and evaluating the quality of predicted phrases by human judges.
– Our current model does not fully consider the correlation among target keyphrases. It would also be interesting to explore the multiple-output optimization on our model.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

This paper divides the keyphrases into two types: (1) Absent key phrases (such phrases do not match any contiguous subsequences of the source document) and (2) Present key phrases (such key phrases fully match a part of the text). The authors used RNN based generative models (discussed as RNN and Copy RNN) for keyphrase prediction and copy mechanism in RNN to predict the already occurred phrases.   Strengths:  1. The formation and extraction of key phrases, which are absent in the current document is an interesting idea of significant research interests.   2. The paper is easily understandable.  3. The use of RNN and Copy RNN in the current context is a new idea. As, deep recurrent neural networks are already used in keyphrase extraction (shows very good performance also), so, it will be interesting to have a proper motivation to justify the use of  RNN and Copy RNN over deep recurrent neural networks.   Weaknesses:  1. Some discussions are required on the convergence of the proposed joint learning process (for RNN and CopyRNN), so that readers can understand, how the stable points in probabilistic metric space are obtained? Otherwise, it may be tough to repeat the results.  2. The evaluation process shows that the current system (which extracts 1. Present and 2. Absent both kinds of keyphrases) is evaluated against baselines (which contains only  present  type of keyphrases). Here there is no direct comparison of the performance of the current system w.r.t. other state-of-the-arts benchmark systems on only  present  type of key phrases. It is important to note that local phrases (keyphrases) are also important for the document. The experiment does not discuss it explicitly. It will be interesting to see the impact of the RNN and Copy RNN based model on automatic extraction of local or  present  type of key phrases.  3. The impact of document size in keyphrase extraction is also an important point. It is found that the published results of [1], (see reference below) performs better than (with a sufficiently high difference) the current system on Inspec (Hulth, 2003) abstracts dataset.   4. It is reported that current system uses 527,830 documents for training, while 40,000 publications are held out for training baselines. Why are all publications not used in training the baselines? Additionally,        The topical details of the dataset (527,830 scientific documents) used in training RNN and Copy RNN are also missing. This may affect the chances of repeating results.  5. As the current system captures the semantics through RNN based models. So, it would be better to compare this system, which also captures semantics. Even, Ref-[2] can be a strong baseline to compare the performance of the current system.  Suggestions to improve:  1. As, per the example, given in the Figure-1, it seems that all the  absent  type of key phrases are actually  Topical phrases . For example:  video search ,  video retrieval ,  video indexing  and  relevance ranking , etc. These all define the domain sub-domain topics of the document. So, In this case, it will be interesting to see the results (or will be helpful in evaluating  absent type  keyphrases): if we identify all the topical phrases of the entire corpus by using tf-idf and relate the document to the high-ranked extracted topical phrases (by using Normalized Google Distance, PMI, etc.). As similar efforts are already applied in several query expansion techniques (with the aim to relate the document with the query, if matching terms are absent in document).  Reference: 1. Liu, Zhiyuan, Peng Li, Yabin Zheng, and Maosong Sun. 2009b. Clustering to find exemplar terms for keyphrase extraction. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 257 266.  2. Zhang, Q., Wang, Y., Gong, Y., & Huang, X. (2016). Keyphrase extraction using deep recurrent neural networks on Twitter. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (pp. 836-845).