A Syntactic Neural Model for General-Purpose Code Generation
We consider the problem of parsing natural language descriptions into source code written in a general-purpose programming language like Python. Existing datadriven methods treat this problem as a language generation task without considering the underlying syntax of the target programming language. Informed by previous work in semantic parsing, in this paper we propose a novel neural architecture powered by a grammar model to explicitly capture the target syntax as prior knowledge. Experiments find this an effective way to scale up to generation of complex programs from natural language descriptions, achieving state-of-the-art results that well outperform previous code generation and semantic parsing approaches.
1 Introduction
Every programmer has experienced the situation where they know what they want to do, but do not have the ability to turn it into a concrete implementation. For example, a Python programmer may want to “sort my list in descending order,” but not be able to come up with the proper syntax sorted(my list, reverse=True) to realize his intention. To resolve this impasse, it is common for programmers to search the web in natural language (NL), find an answer, and modify it into the desired form (Brandt et al., 2009, 2010). However, this is time-consuming, and thus the software engineering literature is ripe with methods to directly generate code from NL descriptions, mostly with hand-engineered methods highly tailored to specific programming languages (Balzer, 1985; Little and Miller, 2009; Gvero and Kuncak, 2015).
In parallel, the NLP community has developed methods for data-driven semantic parsing, which attempt to map NL to structured logical forms executable by computers. These logical forms can be general-purpose meaning representations (Clark and Curran, 2007; Banarescu et al., 2013), formalisms for querying knowledge bases (Tang and Mooney, 2001; Zettlemoyer and Collins, 2005; Berant et al., 2013) and instructions for robots or personal assistants (Artzi and Zettlemoyer, 2013; Quirk et al., 2015), among others. While these methods have the advantage of being learnable from data, compared to the programming languages (PLs) in use by programmers, the domainspecific languages targeted by these works have a schema and syntax that is relatively simple.
Recently, Ling et al. (2016) have proposed a data-driven code generation method for high-level, general-purpose PLs like Python and Java. This work treats code generation as a sequence-tosequence modeling problem, and introduce methods to generate words from character-level models, and copy variable names from input descriptions. However, unlike most work in semantic parsing, it does not consider the fact that code has to be well-defined programs in the target syntax.
In this work, we propose a data-driven syntaxbased neural network model tailored for generation of general-purpose PLs like Python. In order to capture the strong underlying syntax of the PL, we define a model that transduces an NL statement into an Abstract Syntax Tree (AST; Fig. 1(a), § 2) for the target PL. ASTs can be deterministically generated for all well-formed programs using standard parsers provided by the PL, and thus give us a way to obtain syntax information with minimal engineering. Once we generate an AST, we can use deterministic generation tools to convert the AST into surface code. We hypothesize that such a structured approach has two benefits.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
Production Rule Role Explanation Call 7→ expr[func] expr*[args] keyword*[keywords] Function Call . func: the function to be invoked . args: arguments list . keywords: keyword arguments list If 7→ expr[test] stmt*[body] stmt*[orelse] If Statement . test: condition expression . body: statements inside the If clause . orelse: elif or else statements For 7→ expr[target] expr*[iter] stmt*[body] For Loop . target: iteration variable . iter: enumerable to iterate over . body: loop body . orelse: else statementsstmt*[orelse] FunctionDef 7→ identifier[name] arguments*[args] Function Def. . name: function name . args: function arguments
. body: function bodystmt*[body]
Table 1: Example production rules for common Python statements (Python Software Foundation, 2016)
First, we hypothesize that structure can be used to constrain our hypothesis space, ensuring generation of well-formed code. To this end, we propose a syntax-driven neural code generation model. The backbone of our approach is a grammar model (§ 3) which formalizes the generation story of a derivation AST into sequential application of actions that either apply production rules (§ 3.1), or emit terminal tokens (§ 3.2). The underlying syntax is therefore encoded in the grammar model a priori as the set of possible actions. Our approach frees the model from recovering the underlying grammar from limited training data, and instead enables the system to focus on learning the compositionality among existing grammar rules. Xiao et al. (2016) have noted that this imposition of structure on neural models is useful for semantic parsing, and we expect this to be even more important for general-purpose PLs where the syntax trees are larger and more complex.
Second, we hypothesize that structural information helps to model information flow within the network, which naturally reflects the recursive structure of PLs. To test this, we extend a standard recurrent neural network (RNN) decoder to allow for additional neural connections which reflect the recursive structure of an AST (§ 4.2). As an example, when expanding the node ? in Fig. 1(a), we make use of the information from both its parent and left sibling (the dashed rectangle). This enables us to locally pass information of relevant code segments via neural network connections, resulting in more confident predictions.
Experiments (§ 5) on two Python code generation tasks show 11.7% and 9.3% absolute improvements in accuracy against the state-of-the-art system (Ling et al., 2016). Our model also gives competitive performance on a standard semantic parsing benchmark.
2 The Code Generation Problem
Given an NL description x, our task is to generate the code snippet c in a modern PL based on the in-
tent of x. We attack this problem by first generating the underlying AST. We define a probabilistic grammar model of generating an AST y given x: p(y|x). The best-possible AST ŷ is then given by
ŷ = argmax y
p(y|x). (1)
ŷ is then deterministically converted to the corresponding surface code c.1 While this paper uses examples from Python code, our method is PLagnostic.
Before detailing our approach, we first present a brief introduction of the Python AST and its underlying grammar. The Python abstract grammar contains a set of production rules, and an AST is generated by applying several production rules composed of a head node and multiple child nodes. For instance, the first rule in Tab. 1 is used to generate the function call sorted(·) in Fig. 1(a). It consists of a head node of type Call, and three child nodes of type expr, expr* and keyword*, respectively. Labels of each node are noted within brackets. In an AST, non-terminal nodes sketch the general structure of the target code, while terminal nodes can be categorized into two types: operation terminals and variable terminals. Operation terminals correspond to basic arithmetic operations like AddOp.Variable terminal nodes store values for variables and constants of built-in data types2. For instance, all terminal nodes in Fig. 1(a) are variable terminal nodes.
3 Grammar Model
Before detailing our neural code generation method, we first introduce the grammar model at its core. Our probabilistic grammar model defines the generative story of a derivation AST. We factorize the generation process of an AST into sequential application of actions of two types:
• APPLYRULE[r] applies a production rule r to the current derivation tree;
1We use astor library to convert ASTs into Python code. 2bool, float, int, str.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Expr
root
expr[value]
Call
expr*[args] keyword*[keywords]
Name
str(sorted)
expr[func]
expr
Name
str(my_list)
keyword
str(reverse) expr[value]
Name
str(True)
( _ , = )
⋆
t1
t2
t3
t4
t5
t6
t7
t8
t9
t10
t11
t12
t14
root ↦ Expr
Expr ↦ expr[value]
expr ↦ Call
expr ↦ Name
Name ↦ str
[sorted]
[</n>]
expr* ↦ expr
expr ↦ Name
Name ↦ str
[my_list]
keyword* ↦ keyword
Call ↦ expr[func]  expr*[args]  keyword*[keywords]
Action Flow Parent Feeding
ti Apply Rule
ti Generate Token
ti GenToken with Copyt13 [</n>]
t1
t2
t3
t4
t5
t6t7t8
t4
t9
t10
t11 t12 t13
t4
t14
t15 t16 t17 t16
t18
t19 t20 t21
t0
(a) (b)
_ Input: Code:
. . .
Figure 1: (a) the Abstract Syntax Tree (AST) for the given example code. Dashed nodes denote terminals. Nodes are labeled with time steps during which they are generated. (b) the action sequence (up to t14) used to generate the AST in (a)
• GENTOKEN[v] populates a variable terminal node by appending a terminal token v.
Fig. 1(b) shows the generation process of the target AST in Fig. 1(a). Each node in Fig. 1(b) indicates an action. Action nodes are connected by solid arrows which depict the chronological order of the action flow. The generation proceeds in depth-first, left-to-right order (dotted arrows represent parent feeding, explained in § 4.2.1).
Formally, under our grammar model, the probability of generating an AST y is factorized as:
p(y|x) = T∏ t=1 p(at|x, a<t), (2)
where at is the action taken at time step t, and a<t is the sequence of actions before t. We will explain how to compute Eq. (2) in § 4. Put simply, the generation process begins from a root node at t0, and proceeds by the model choosing APPLYRULE actions to generate the overall program structure from a closed set of grammar rules, then at leaves of the tree corresponding to variable terminals, the model switches to GENTOKEN actions to generate variables or constants from the open set. We describe this process in detail below.
3.1 APPLYRULE Actions
APPLYRULE actions generate program structure, expanding the current node (the frontier node at time step t: nft) in a depth-first, left-to-right traversal of the tree. Given a fixed set of production rules, APPLYRULE chooses a rule r from the subset that has a head matching the type of nft , and uses r to expand nft by appending all child nodes specified by the selected production. As an
example, in Fig. 1(b), the rule Call 7→ expr. . . expands the frontier node Call at time step t4, and its three child nodes expr, expr* and keyword* are added to the derivation.
APPLYRULE actions grow the derivation AST by appending nodes. When a variable terminal node (e.g., str) is added to the derivation and becomes the frontier node, the grammar model then switches to GENTOKEN actions to populate the variable terminal with tokens.
Unary Closure Sometimes, generating an AST requires applying a chain of unary productions. For instance, it takes three time steps (t9 − t11) to generate the sub-structure expr* 7→ expr 7→ Name 7→ str in Fig. 1(a). This can be effectively reduced to one step of APPLYRULE action by taking the closure of the chain of unary productions and merging them into a single rule: expr* 7→∗ str. Unary closures reduce the number of actions needed, but would potentially increase the size of the grammar. In our experiments we tested our model both with and without unary closures (§ 5).
3.2 GENTOKEN Actions
Once we reach a frontier node nft that corresponds to a variable type (e.g., str), GENTOKEN actions are used to fill this node with values. For generalpurpose PLs like Python, variables and constants have values with one or multiple tokens. For instance, a node that stores the name of a function (e.g., sorted) has a single token, while a node that denotes a string constant (e.g., a=‘hello world’) could have multiple tokens. Our model copes with both scenarios by firing GENTOKEN actions at one or more time steps. At each time
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
step, GENTOKEN appends one terminal token to the current frontier variable node. A special </n> token is used to “close” the node. The grammar model then proceeds to the new frontier node.
Terminal tokens can be generated from a predefined vocabulary, or be directly copied from the input NL. This is motivated by the observation that the input description often contains out-ofvocabulary (OOV) variable names or literal values that are directly used in the target code. For instance, in our running example the variable name my list can be directly copied from the the input at t12. We give implementation details in § 4.2.2.
4 Estimating Action Probabilities
We estimate action probabilities in Eq. (2) using attentional neural encoder-decoder models with an information flow structured by the syntax trees.
4.1 Encoder
For an NL description x consisting of n words {wi}ni=1, the encoder computes a context sensitive embedding hi for each wi using a bidirectional Long Short-Term Memory (LSTM) network (Hochreiter and Schmidhuber, 1997), similar to the setting in (Bahdanau et al., 2014). See supplementary materials for detailed equations.
4.2 Decoder
The decoder uses a RNN to model the sequential generation process of an AST defined as Eq. (2). Each action step in the grammar model naturally grounds to a time step in the decoder RNN. Therefore, the action sequence in Fig. 1(b) can be interpreted as unrolling RNN time steps, with solid arrows indicating RNN connections. The RNN maintains an internal state to track the generation process (§ 4.2.1), which will then be used to compute action probabilities p(at|x, a<t) (§ 4.2.2).
4.2.1 Tracking Generation States
Our implementation of the decoder resembles a vanilla LSTM, with additional neural connections (parent feeding, Fig. 1(b)) to reflect the topological structure of an AST. The decoder’s internal hidden state at time step t, st, is given by:
st = fLSTM([at−1 : ct : pt : nft ], st−1), (3) where fLSTM(·) is the LSTM update function. [:] denotes vector concatenation. st will then be used to compute action probabilities p(at|x, a<t) in Eq. (2). Here, at−1 is the embedding of the previous action. ct is a context vector retrieved from
input encodings {hi} via soft attention. pt is a vector that encodes the information of the parent action. nft denotes the node type embedding of the current frontier node nft
3. Intuitively, feeding the decoder the information of nft helps the model to keep track of the frontier node to expand. Action Embedding at We maintain two action embedding matrices, WR and WG. Each row in WR (WG) corresponds to an embedding vector for an action APPLYRULE[r] (GENTOKEN[v]). Context Vector ct The decoder RNN uses soft attention to retrieve a context vector ct from the input encodings {hi} pertain to the prediction of the current action. We follow Bahdanau et al. (2014) and use a Deep Neural Network (DNN) with a single hidden layer to compute attention weights. Parent Feeding pt Our decoder RNN uses additional neural connections to directly pass information from parent actions. For instance, when computing s9, the information from its parent action step t4 will be used. Formally, we define the parent action step pt as the time step at which the frontier node nft is generated. As an example, for t9, its parent action step p9 is t4, since nf9 is the node ?, which is generated at t4 by the APPLYRULE[Call7→. . .] action.
We model parent information pt from two sources: (1) the hidden state of parent action spt , and (2) the embedding of parent action apt . pt is the concatenation. The parent feeding schema enables the model to utilize the information of parent code segments to make more confident predictions. Similar approaches of injecting parent information were also explored in the SEQ2TREE model in Dong and Lapata (2016)4.
3We maintain an embedding for each node type. 4SEQ2TREE generates tree-structured outputs by condi-
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
4.2.2 Calculating Action Probabilities
In this section we explain how action probabilities p(at|x, a<t) are computed based on st. APPLYRULE The probability of applying rule r as the current action at is given by a softmax5:
p(at = APPLYRULE[r]|x, a<t) = softmax(WR · g(st))ᵀ · e(r) (4)
where g(·) is a non-linearity tanh(W ·st+b), and e(r) the one-hot vector for rule r. GENTOKEN As in § 3.2, a token v can be generated from a predefined vocabulary or copied from the input, defined as the marginal probability:
p(at = GENTOKEN[v]|x, a<t) = p(gen|x, a<t)p(v|gen, x, a<t) + p(copy|x, a<t)p(v|copy, x, a<t).
The selection probabilities p(gen|·) and p(copy|·) are given by softmax(WS · st). The probability of generating v from the vocabulary, p(v|gen, x, a<t), is defined similarly as Eq. (4), except that we use the embedding matrix of GENTOKEN, WG. To model the copy probability, we follow recent advances in modeling copying mechanism in neural networks (Gu et al., 2016; Jia and Liang, 2016; Ling et al., 2016), and use a pointer network (Vinyals et al., 2015) to compute the probability of copying the i-th word from the input by attending to input representations {hi}:
p(wi|copy, x, a<t) = exp(ω(hi, st, ct))∑n
i′=1 exp(ω(hi′ , st, ct)) ,
where ω(·) is a DNN with a single hidden layer.
4.3 Training and Inference
Given a dataset of pairs of NL descriptions xi and code snippets ci, we parse ci into its AST yi and decompose yi into a sequence of oracle actions under the grammar model. The model is then optimized by maximizing the log-likelihood of the oracle action sequence. At inference time, we use beam search to approximate the best AST ŷ in Eq. (1). See supplementary materials for the pseudo-code of the inference algorithm.
5 Experimental Evaluation
5.1 Datasets and Metrics
HEARTHSTONE (HS) dataset (Ling et al., 2016) is a collection of Python classes that implement
tioning on the hidden states of parent non-terminals, while our parent feeding uses the states of parent actions.
5We do not show bias terms for all softmax equations.
cards for the card game HearthStone. Each card comes with a set of fields (e.g., name, cost, and description), which we concatenate to create the input sequence. This dataset is relatively difficult: input descriptions are short, while the target code is in complex class structures, with each AST having 137 nodes on average. DJANGO dataset (Oda et al., 2015) is a collection of lines of code from the Django web framework, each with a manually annotated NL description. Compared with the HS dataset where card implementations are somewhat homogenous, examples in DJANGO are more diverse, spanning a wide variety of real-world use cases like string manipulation, IO operations, and exception handling. IFTTT dataset (Quirk et al., 2015) is a domainspecific benchmark that provides an interesting side comparison. Different from HS and DJANGO which are in a general-purpose PL, programs in IFTTT are written in a domain-specific language used by the IFTTT task automation App. Users of the App write simple instructions (e.g., If Instagram.AnyNewPhotoByYou Then Dropbox.AddFileFromURL) with NL descriptions (e.g., “Autosave your Instagram photos to Dropbox”). Each statement inside the If or Then clause consists of a channel (e.g., Dropbox) and a function (e.g., AddFileFromURL)6. This simple structure results in much more concise ASTs (7 nodes on average). Because all examples are created by ordinary Apps users, the dataset is highly noisy, with input NL very loosely con-
6Like Beltagy and Quirk (2016), we strip function parameters since they are mostly specific to users.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
nected to target ASTs. The authors thus provide a high-quality filtered test set, where each example is verified by at least three annotators. We use this set for evaluation. Also note IFTTT’s grammar has more productions (Tab. 2), but this does not imply that its grammar is more complex. This is because for HS and DJANGO terminal tokens are generated by GENTOKEN actions, but for IFTTT, all the code is generated directly by APPLYRULE actions. Metrics As is standard in semantic parsing, we measure accuracy, the fraction of correctly generated examples. However, because generating an exact match for complex code structures is nontrivial, we follow Ling et al. (2016), and use tokenlevel BLEU-4 with as a secondary metric, defined as the averaged smoothed BLEU (Lin and Och, 2004) scores over all examples.
5.2 Setup
Preprocessing All input descriptions are tokenized using NLTK. We perform simple canonicalization for DJANGO, such as replacing quoted strings in the inputs with place holders. See supplementary materials for details. We extract unary closures whose frequency is larger than a threshold k (k = 30 for HS and 50 for DJANGO). Configuration The size of all embeddings is 128, except for node type embeddings, which is 64. The dimensions of RNN states and hidden layers are 256 and 50, respectively. Since our datasets are relatively small for a data-hungry neural model, we impose strong regularization using recurrent dropouts (Gal and Ghahramani, 2016), together with standard dropout layers added to the inputs and outputs of the decoder RNN. We validate the dropout probability from {0, 0.2, 0.3, 0.4}. For decoding, we use a beam size of 15.
5.3 Results
Evaluation results for Python code generation tasks are listed in Tab. 3. Numbers for our systems are averaged over three runs. We compare primarily with two approaches: (1) Latent Predictor Network (LPN), a state-of-the-art sequenceto-sequence code generation model (Ling et al., 2016), and (2) SEQ2TREE, a neural semantic parsing model (Dong and Lapata, 2016). SEQ2TREE generates trees one node at a time, and the target grammar is not explicitly modeled a priori, but implicitly learned from data. We test both the original SEQ2TREE model released by the authors and our revised one (SEQ2TREE–UNK) that uses unknown word replacement to handle rare
words (Luong et al., 2015). For completeness, we also compare with a strong neural machine translation (NMT) system (Neubig, 2015) using a standard encoder-decoder architecture with attention and unknown word replacement, and include numbers from other baselines used in Ling et al. (2016). On the HS dataset, which has relatively large ASTs, we use unary closure for our model and SEQ2TREE, and for DJANGO we do not.
System Comparison As in Tab. 3, our model registers 11.7% and 9.3% absolute improvements over LPN in accuracy on HS and DJANGO. This boost in performance strongly indicates the importance of modeling grammar in code generation. For the baselines, we find LPN outperforms others in most cases. We also note that SEQ2TREE achieves a decent accuracy of 13.6% on HS, which is due to the effect of unknown word replacement, since we only achieved 1.5% without it. A closer comparison with SEQ2TREE is insightful for understanding the advantage of our syntax-driven approach, since both SEQ2TREE and our system output ASTs: (1) SEQ2TREE predicts one node each time step, and requires additional “dummy” nodes to mark the boundary of a subtree. The sheer number of nodes in target ASTs makes the prediction process error-prone. In contrast, the APPLYRULE actions of our grammar model allows for generating multiple nodes at a single time step. Empirically, we found that in HS, SEQ2TREE takes more than 300 time steps on average to generate a target AST, while our model takes only 170 steps. (2) SEQ2TREE does not directly use productions in the grammar, which possibly leads to grammat-
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
0 10 20 30 40 50 Reference AST Size (# nodes)
0.0
0.2
0.4
0.6
0.8
1.0
P e rf
o rm
a n ce
BLEU acc
Figure 3: Performance w.r.t reference AST size on DJANGO
50 100 150 200 250 Reference AST Size (# nodes)
0.0
0.2
0.4
0.6
0.8
1.0
P e rf
o rm
a n ce
BLEU acc
Figure 4: Performance w.r.t reference AST size on HS
ically incorrect ASTs and thus empty code outputs. We observe that the ratio of grammatically incorrect ASTs predicted by SEQ2TREE on HS and DJANGO are 21.2% and 10.9%, respectively, while our system guarantees grammaticality. Ablation Study We also ablated our bestperforming models to analyze the contribution of each component. “–frontier embed.” removes the frontier node embedding nft from the decoder RNN inputs (Eq. (3)). This yields worse results on DJANGO while gives slight improvements in accuracy on HS. This is probably because that the grammar of HS has fewer node types, and thus the RNN is able to keep track of nft without depending on its embedding. Next, “–parent feed.” removes the parent feeding mechanism. The accuracy drops significantly on HS, with a marginal deterioration on DJANGO. This result is interesting because it suggests that parent feeding is more important when the ASTs are larger, which will be the case when handling more complicated code generation tasks like HS. Finally, removing the pointer network (“–copy terminals”) in GENTOKEN actions gives poor results, indicating that it is important to directly copy variable names and values from the input.
The results with and without unary closure demonstrate that, interestingly, it is effective on HS but not on DJANGO. We conjecture that this is because on HS it significantly reduces the number of actions from 173 to 142 (c.f., Tab. 2), with the number of productions in the grammar remaining unchanged. In contrast, DJANGO has a broader domain, and thus unary closure results in more productions in the grammar (237 for DJANGO vs. 100 for HS), increasing sparsity. Performance by the size of AST We further investigate our model’s performance w.r.t. the size
of the gold-standard ASTs in Figs. 3 and 4. Not surprisingly, the performance drops when the size of the reference ASTs increases. Additionally, on the HS dataset, the BLEU score still remains at around 50 even when the size of ASTs grows to 200, indicating that our proposed syntax-driven approach is robust for long code segments. Domain Specific Code Generation Although this is not the focus of our work, evaluation on IFTTT brings us closer to a standard semantic parsing setting, which helps to investigate similarities and differences between generation of more complicated general-purpose code and and more limiteddomain simpler code. Tab. 4 shows the results, following the evaluation protocol in (Beltagy and Quirk, 2016) for accuracies at both channel and full parse tree (channel + function) levels. Our full model performs on par with existing neural network-based methods, while outperforming other neural models in full tree accuracy (82.0%). This score is close to the best classical method (LR), which is based on a logistic regression model with rich hand-engineered features (e.g., brown clusters and paraphrase). Also note that the performance between NMT and other neural models is much closer compared with the results in Tab. 3. This suggests that general-purpose code generation is more challenging than the simpler IFTTT setting, and therefore modeling structural information is more helpful. Case Studies Finally, we present output examples in Tab. 5. On HS, we observe that most of the time our model gives correct predictions by filling learned code templates from training data with arguments (e.g., cost) copied from input. However, we do find interesting examples indicating that the model learns to generalize beyond trivial copy-
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
input <name> Brawl </name> <cost> 5 </cost> <desc> Destroy all minions except one (chosen randomly) </desc> <rarity> Epic </rarity> ... pred. class Brawl(SpellCard): def init (self): super(). init (’Brawl’, 5, CHARACTER CLASS.
WARRIOR, CARD RARITY.EPIC) def use(self, player, game): super().use(player, game) targets = copy.copy(game.other player.minions) targets.extend(player.minions) for minion in targets: minion.die(self) A
ref. minions = copy.copy(player.minions) minions.extend(game.other player.minions) if len(minions) > 1: survivor = game.random choice(minions) for minion in minions: if minion is not survivor: minion.die(self) B
input join app config.path and string ’locale’ into a file path, substitute it for localedir. pred. localedir = os.path.join( app config.path, ’locale’) 3
input self.plural is an lambda function with an argument n, which returns result of boolean expression n not equal to integer 1 pred. self.plural = lambda n: len(n) 7 ref. self.plural = lambda n: int(n!=1)
Table 5: Predicted examples from HS (1st) and DJANGO. Copied contents (copy probability > 0.9) are highlighted.
ing. For instance, the first example is one that our model predicted wrong — it generated code block A instead of the gold B (it also missed a function definition not shown here). However, we find that the block A actually conveys part of the input intent by destroying all, not some, of the minions. Since we are unable to find code block A in the training data, it is clear that the model has learned to generalize to some extent from multiple training card examples with similar semantics or structure.
The next two examples are from DJANGO. The first one shows that the model learns the usage of common API calls (e.g., os.path.join), and how to populate the arguments by copying from inputs. The second example illustrates the difficulty of generating code with complex nested structures like lambda functions, a scenario worth further investigation in future studies. More examples are attached in supplementary materials.
6 Related Work
Code Generation and Analysis Most existing works on code generation focus on generating code for domain specific languages (DSLs) (Kushman and Barzilay, 2013; Raza et al., 2015; Manshadi et al., 2013), with neural network-based approaches recently explored (Parisotto et al., 2016; Balog et al., 2016). For general-purpose code gen-
eration, besides the general framework of Ling et al. (2016), existing methods often use language and task-specific rules and strategies (Lei et al., 2013; Raghothaman et al., 2016). A similar line is to use NL queries for code retrieval (Wei et al., 2015; Allamanis et al., 2015). The reverse task of generating NL summaries from source code has also been explored (Oda et al., 2015; Iyer et al., 2016). Finally, there are probabilistic models of source code (Maddison and Tarlow, 2014; Nguyen et al., 2013). The most relevant work is Allamanis et al. (2015), which uses a factorized model to measure semantic relatedness between NL and ASTs for code retrieval, while our model tackles the more challenging generation task. Semantic Parsing Our work is related to the general topic of semantic parsing, where the target logical forms can be viewed as DSLs. The parsing process is often guided by grammatical formalisms like combinatory categorical grammars (Kwiatkowski et al., 2013; Artzi et al., 2015), dependency-based syntax (Liang et al., 2011; Pasupat and Liang, 2015) or task-specific formalisms (Clarke et al., 2010; Yih et al., 2015; Krishnamurthy et al., 2016; Misra et al., 2015; Mei et al., 2016). Recently, there are efforts in designing neural network-based semantic parsers (Misra and Artzi, 2016; Dong and Lapata, 2016; Neelakantan et al., 2016). Several approaches have be proposed to utilize grammar knowledge in a neural parser, such as augmenting the training data by generating examples guided by the grammar (Kociský et al., 2016; Jia and Liang, 2016). Liang et al. (2016) used a neural decoder which constrains the space of next valid tokens in the query language for question answering. Finally, the structured prediction approach proposed by Xiao et al. (2016) is closely related to our model in using the underlying grammar as prior knowledge to constrain the generation process of derivation trees, while our method is based on a unified grammar model which jointly captures production rule application and terminal symbol generation, and scales to general purpose code generation tasks.
7 Conclusion
This paper proposes a syntax-driven neural code generation approach that generates an abstract syntax tree by sequentially applying actions from a grammar model. Experiments on both code generation and semantic parsing tasks demonstrate the effectiveness of our proposed approach.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

Summary: The paper proposes a neural model for predicting Python syntax trees from text descriptions. Guided by the actual Python grammar, the model generates tree nodes sequentially in a depth-first fashion. Key ideas include injecting the information from the parent node as part of the LSTM input, a pointer network for copying the terminals, and unary closure which collapses chains of unary productions to reduce the tree size. The model is evaluated on three datasets from different domains and outperforms almost all previous work.  Strengths:  The paper is overall very well-written. The explanation of system is clear, and the analysis is thorough.  The system itself is a natural extension of various ideas. The most similar work include tree-based generation with parent feeding (Dong and Lapata, 2016) and various RNN-based semantic parsing with copy mechanism (Jia and Liang, 2016; Ling et al., 2016). [The guidance of parsing based on grammar is also explored in Chen Liang et al., 2016 (https:  arxiv.org abs 1611.00020) where a code-assist system is used to ensure that the code is valid.] Nevertheless, the model is this paper stands out as it is able to generate much longer and more complex programs than most previous work mentioned.   Weaknesses:  The evaluation is done on code accuracy (exact match) and BLEU score. These metrics (especially BLEU) might not be the best metrics for evaluating the correctness of programs. For instance, the first example in Table 5 shows that while the first two lines in boxes A and B are different, they have the same semantics. Another example is that variable names can be different. Evaluation based on what the code does (e.g., using test cases or static code analysis) would be more convincing.  Another point about evaluation: other systems (e.g., NMT baseline) may generate code with syntactic error. Would it be possible to include the result on the highest-scoring well-formed code (e.g., using beam search) that these baseline systems generate? This would give a fairer comparison since these system can choose to prune malformed code.  General Discussion:  * Lines 120-121: some approaches that use domain-specific languages were also guided by a grammar. One example is Berant and Liang, 2014, which uses a pretty limited grammar for logical forms (Table 1). In addition to comparing to that line of work, emphasizing that the grammar in this paper is much larger than most previous work would make this work stronger.  * Lines 389-397: For the parent feeding mechanism, is the child index being used? In other words, is p_t different when generating a first child versus a second child? In Seq2Tree (Dong and Lapata, 2016) the two non-terminals would have different hidden states.  * Line 373: Are the possible tokens embedded? Is it assumed that the set of possible tokens is known beforehand?  * The examples in the appendix are nice.  ---  I have read the author response.


PositionRank: An Unsupervised Approach to Keyphrase Extraction from Scholarly Documents
The large and growing amounts of online scholarly data present both challenges and opportunities to enhance knowledge discovery. One such challenge is to automatically extract a small set of keyphrases from a document that can accurately describe the document’s content and can facilitate fast information processing. In this paper, we propose PositionRank, an unsupervised model for keyphrase extraction from scholarly documents that incorporates information from all positions of a word’s occurrences into a biased PageRank. Our model obtains remarkable improvements in performance over PageRank models that do not take into account word positions as well as over strong baselines for this task. Specifically, on several datasets of research papers, PositionRank achieves improvements as high as 29.09%.
1 Introduction
The current Scholarly Web contains many millions of scientific documents. For example, Google Scholar is estimated to have more than 100 million documents. On one hand, these rapidly-growing scholarly document collections offer benefits for knowledge discovery, and on the other hand, finding useful information has become very challenging. Keyphrases associated with a document typically provide a high-level topic description of the document and can allow for efficient information processing. In addition, keyphrases are shown to be rich sources of information in many natural language processing and information retrieval tasks such as scientific paper summarization, classification, recommendation, clustering, and search (Abu-Jbara and Radev, 2011; Qazvinian et al.,
2010; Jones and Staveley, 1999; Zha, 2002; Zhang et al., 2004; Hammouda et al., 2005). Due to their importance, many approaches to keyphrase extraction have been proposed in the literature along two lines of research: supervised and unsupervised (Hasan and Ng, 2014, 2010).
In the supervised line of research, keyphrase extraction is formulated as a binary classification problem, where candidate phrases are classified as either positive (i.e., keyphrases) or negative (i.e., non-keyphrases) (Frank et al., 1999; Hulth, 2003). Various feature sets and classification algorithms yield different extraction systems. For example, Frank et al. (1999) developed a system that extracts two features for each candidate phrase, i.e., the tf-idf of the phrase and its distance from the beginning of the target document, and uses them as input to Naı̈ve Bayes classifiers. Although supervised approaches typically perform better than unsupervised approaches (Kim et al., 2013), the requirement for large human-annotated corpora for each field of study has led to significant attention towards the design of unsupervised approaches.
In the unsupervised line of research, keyphrase extraction is formulated as a ranking problem with graph-based ranking techniques being considered state-of-the-art (Hasan and Ng, 2014). These graph-based techniques construct a word graph from each target document, such that nodes correspond to words and edges correspond to word association patterns. Nodes are then ranked using graph centrality measures such as PageRank (Mihalcea and Tarau, 2004; Liu et al., 2010) or HITS (Litvak and Last, 2008), and the top ranked phrases are returned as keyphrases. Since their introduction, many graph-based extensions have been proposed, which aim at modeling various types of information. For example, Wan and Xiao (2008) proposed a model that incorporates a local neighborhood of the target document correspond-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
Factorizing Personalized Markov Chains for Next-Basket Recommendation by Steffen Rendle, Christoph Freudenthaler and Lars Schmidt-Thieme Recommender systems are an important component of many websites. Two of the most popular approaches are based on matrix factorization (MF) and Markov chains (MC). MF methods learn the general taste of a user by factorizing the matrix over observed user-item preferences. [...] we present a method bringing both approaches together. Our method is based on personalized transition graphs over underlying Markov chains. [...] our factorized personalized MC (FPMC) model subsumes both a common Markov chain and the normal matrix factorization model. [...] we introduce an adaption of the Bayesian Personalized Ranking (BPR) framework for sequential basket data. [...]
Author-input keyphrases: Basket Recommendation, Markov Chain, Matrix Factorization
Figure 1: The title and abstract of a WWW paper by Rendle et al. (2010) and the author-input keyphrases for the paper. Red bold phrases represent the gold-standard keyphrases for the document.
ing to its textually-similar documents, computed using the cosine similarity between the tf-idf vectors of documents. Liu et al. (2010) assumed a mixture of topics over documents and proposed to use topic models to decompose these topics in order to select keyphrases from all major topics. Keyphrases are then ranked by aggregating the topic-specific scores obtained from several topicbiased PageRanks. We posit that other information can be leveraged that has the potential to improve unsupervised keyphrase extraction.
For example, in a scholarly domain, keyphrases generally occur on positions very close to the beginning of a document and occur frequently. Figure 1 shows an anecdotal example illustrating this behavior using the 2010 best paper award winner in the World Wide Web conference. The author input keyphrases are marked with red bold in the figure. Notice in this example the high frequency of the keyphrase “Markov chain” that occurs very early in the document (even from its title). Hence, can we design an effective unsupervised approach to keyphrase extraction by jointly exploiting words’ position information and their frequency in documents? We specifically address this question using research papers as a case study. The result of this extraction task will aid indexing of documents in digital libraries, and hence, will lead to improved organization, search, retrieval, and recommendation of scientific documents. The importance of keyphrase extraction from research papers is also emphasized by the SemEval Shared Tasks on this topic from 20171 and 2010 (Kim et al., 2010). Our contributions are as follows: • We propose an unsupervised graph-based
model, called PositionRank, that incorporates 1http://alt.qcri.org/semeval2017/task10/
information from all positions of a word’s occurrences into a biased PageRank to score keywords that are later used to score and rank keyphrases in research papers. • We show that PositionRank that aggregates
information from all positions of a word’s occurrences performs better than a model that uses only the first position of a word. • We experimentally evaluate PositionRank on
three datasets of research papers and show statistically significant improvements over PageRank-based models that do not take into account word positions, as well as over strong baselines for keyphrase extraction.
The rest of the paper is organized as follows. We summarize related work in the next section. PositionRank is described in Section 3. We then present the datasets of research papers, and our experiments and results in Section 4. Finally, we conclude the paper in Section 5.
2 Related Work
Many supervised and unsupervised approaches to keyphrase extraction have been proposed in the literature (Hasan and Ng, 2014).
Supervised approaches use annotated documents with “correct” keyphrases to train classifiers for discriminating keyphrases from nonkeyphrases for a document. KEA (Frank et al., 1999) and GenEx (Turney, 2000) are two representative supervised approaches with the most important features being the frequency and the position of a phrase in a target document. Hulth (2003) used a combination of lexical and syntactic features such as the collection frequency and the part-of-speech tag of a phrase in conjunction with a bagging technique. Nguyen and Kan
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
(2007) extended KEA to include features such as the distribution of candidate phrases in different sections of a research paper, and the acronym status of a phrase. In a different work, Medelyan et al. (2009) extended KEA to integrate information from Wikipedia. Lopez and Romary (2010) used bagged decision trees learned from a combination of features including structural features (e.g., the presence of a phrase in particular sections of a document) and lexical features (e.g., the presence of a candidate phrase in WordNet or Wikipedia). Chuang et al. (2012) proposed a model that incorporates a set of statistical and linguistic features (e.g., tf-idf, BM25, part-of-speech filters) for identifying descriptive terms in a text. Caragea et al. (2014) designed features based on information available in a document network (such as a citation network) and used them with traditional features in a supervised framework.
In unsupervised approaches, various measures such as tf-idf and topic proportions are used to score words, which are later aggregated to obtain scores for phrases (Barker and Cornacchia, 2000; Zhang et al., 2007; Liu et al., 2009). The ranking based on tf-idf has been shown to work well in practice (Hasan and Ng, 2014, 2010), despite its simplicity. Graph-based ranking methods and centrality measures are considered stateof-the-art for unsupervised keyphrase extraction. Mihalcea and Tarau (2004) proposed TextRank for scoring keyphrases by applying PageRank on a word graph built from adjacent words within a document. Wan and Xiao (2008) extended TextRank to SingleRank by adding weighted edges between words that co-occur in a window of variable size w ≥ 2. Textually-similar neighboring documents are included in ExpandRank (Wan and Xiao, 2008) to compute more accurate word cooccurrence information. Gollapalli and Caragea (2014) extended ExpandRank to integrate information from citation networks where papers cite one another.
Several unsupervised approaches leverage word clustering techniques such as grouping candidate words into topics and then, extract one representative keyphrase from each topic (Liu et al., 2009; Bougouin et al., 2013). Liu et al. (2010) extended topic-biased PageRank (Haveliwala, 2003) to kephrase extraction. In particular, they decomposed a document into multiple topics, using topic models, and applied a separate topic-biased PageRank for each topic. The PageRank scores
from each topic were then combined into a single score, using as weights the topic proportions returned by topic models for the document.
The best performing keyphrase extraction system in SemEval 2010 (El-Beltagy and Rafea, 2010) used statistical observations such as term frequencies to filter out phrases that are unlikely to be keyphrases. More precisely, thresholding on the frequency of phrases is applied, where the thresholds are estimated from the data. The candidate phrases are then ranked using the tf-idf model in conjunction with a boosting factor which aims at reducing the bias towards single word terms. Danesh et al. (2015) computed an initial weight for each phrase based on a combination of statistical heuristics such as the tf-idf score and the first position of a phrase in a document. Phrases and their initial weights are then incorporated into a graph-based algorithm which produces the final ranking of keyphrase candidates. Adar and Datta (2015) extracted keyphrases by mining abbreviations from scientific literature and built a semantically hierarchical keyphrase database. Word embedding vectors were also employed to measure the relatedness between words in graph based models (Wang et al., 2014). Many of the above approaches, both supervised and unsupervised, are compared and analyzed in the ACL survey on keyphrase extraction by Hasan and Ng (2014).
In contrast to the above approaches, we propose PositionRank, aimed at capturing both highly frequent words or phrases and their position in a document. Despite that the relative position of a word in a document is shown to be a very effective feature in supervised keyphrase extraction (Hulth, 2003; Zhang et al., 2007), to our knowledge, the position information has not been used before in unsupervised methods. The strong contribution of this paper is the design of a position-biased PageRank model that successfully incorporates all positions of a word’s occurrences, which is different from supervised models that use only the first position of a word. Our model assigns higher probabilities to words found early on in a document instead of using a uniform distribution over words.
3 Proposed Model
In this section, we describe PositionRank, our fully unsupervised, graph-based model, that simultaneously incorporates the position of words and their frequency in a document to compute a biased PageRank score for each candidate word.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Graph-based ranking algorithms such as PageRank (Page et al., 1998) measure the importance of a vertex within a graph by taking into account global information computed recursively from the entire graph. For each word, we compute a weight by aggregating information from all positions of the word’s occurrences. This weight is then incorporated into a biased PageRank algorithm in order to assign a different “preference” to each word.
3.1 PositionRank
The PositionRank algorithm involves three essential steps: (1) the graph construction at word level; (2) the design of Position-Biased PageRank; and (3) the formation of candidate phrases. These steps are detailed below.
3.1.1 Graph Construction
Let d be a target document for extracting keyphrases. We first apply the part-of-speech filter using the NLP Stanford toolkit and then select as candidate words only nouns and adjectives, similar to previous works (Mihalcea and Tarau, 2004; Wan and Xiao, 2008). We build a word graph G = (V,E) for d such that each unique word that passes the part-of-speech filter corresponds to a node in G. Two nodes vi and vj are connected by an edge (vi, vj) ∈ E if the words corresponding to these nodes co-occur within a window of w contiguous tokens in the content of d. The weight of an edge (vi, vj) ∈ E is computed based on the co-occurrence count of the two words within a window ofw successive tokens in d. Note that the graph can be constructed both directed and undirected. However, Mihalcea and Tarau (2004) showed that the type of graph used to represent the text does not significantly influence the performance of keyphrase extraction. Hence, in this work, we build undirected graphs.
3.1.2 Position-Biased PageRank
Formally, letG be an undirected graph constructed as above and let M be its adjacency matrix. An element mij ∈ M is set to the weight of edge (vi, vj) if there exist an edge between nodes vi and vj , and is set to 0 otherwise. The PageRank score of a node vi is recursively computed by summing the normalized scores of nodes vj , which are linked to vi (as explained below).
Let S denote the vector of PageRank scores, for all vi ∈ V . The initial values of S are set to 1|V | . The PageRank score of each node at step t+1, can
then be computed recursively using:
S(t+ 1) = M̃ · S(t) (1)
where M̃ is the normalized form of matrixM with m̃ij ∈ M̃ defined as:
m̃ij =
{ mij/ ∑|V | j=1mij if ∑|V | j=1mij 6= 0
0 otherwise
The PageRank computation can be seen as a Markov Chain process in which nodes represent states and the links between them are the transitions. By recursively applying Eq. (1), we obtain the principal eigenvector, which represents the stationary probability distribution of each state, in our case of each node (Manning et al., 2008).
To ensure that the PageRank (or the random walk) does not get stuck into cycles of the graph, a damping factor α is added to allow the “teleport” operation to another node in the graph. Hence, the computation of S becomes:
S = α · M̃ · S + (1− α) · p̃ (2)
where S is the principal eigenvector and p̃ is a vector of length |V | with all elements 1|V | . The vector p̃ indicates that, being in a node vi, the random walk can jump to any other node in the graph with equal probability. By biasing p̃, the random walk would prefer nodes that have higher probability in the graph (Haveliwala, 2003).
The idea of PositionRank is to assign larger weights (or probabilities) to words that are found early in a document and are frequent. Specifically, we want to assign a higher probability to a word found on the 2nd position as compared to a word found on the 50th position in the same document. We weigh each candidate word with its inverse position in the document before any filters are applied. If the same word appears multiple times in the target document, then we sum all its position weights. For example, if a word is found on the following positions: 2nd, 5th and 10th, its weight is: 12 + 1 5 + 1 10 = 4 5 = 0.8. Summing up the position weights for a given word aims to grant more confidence to frequently occurring words by taking into account the position weight of each occurrence. Then, the vector p̃ is set to the normalized weights for each candidate word as follows:
p̃ = [
p1 p1+p2+...+p|V | , p2p1+p2+...+p|V | , ..., p|V | p1+p2+...+p|V |
]
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
The PageRank score of a vertex vi, i.e., S(vi), can be obtained in an algebraic way by recursively computing the following equation:
S(vi) = (1− α) · p̃i + α · ∑
vj∈Adj(vi)
wji O(vj) S(vj)
where O(vj) = ∑
vk∈Adj(vj)wjk and p̃i is the weight found in the vector p̃ for vertex vi.
In our experiments, the words’ PageRank scores are recursively computed until the difference between two consecutive iterations is less than 0.001 or a number of 100 iterations is reached.
3.1.3 Forming Candidate Phrases
Candidate words that have contiguous positions in a document are concatenated into phrases. We consider noun phrases that match the regular expression (adjective)*(noun)+, of length up to three, (i.e., unigrams, bigrams, and trigrams).
Finally, phrases are scored by using the sum of scores of individual words that comprise the phrase (Wan and Xiao, 2008). The top-scoring phrases are output as predictions (i.e., the predicted keyphrases for the document).
4 Experiments and Results
4.1 Datasets and Evaluation Metrics
In order to evaluate the performance of PositionRank, we carried out experiments on three datasets. The first and second datasets were made available by Gollapalli and Caragea (2014). These datasets were compiled from the CiteSeerX digital library (Giles et al., 1998) and consist of research papers from the ACM Conference on Knowledge Discovery and Data Mining (KDD) and the World Wide Web Conference (WWW). The third dataset was made available by Nguyen and Kan (2007) and consist of research papers from various disciplines. In experiments, we use the title and abstract of each paper to extract keyphrases. The author-input keyphrases are used as gold-standard for evaluation. All three datasets are summarized in Table 1, which shows the number of papers in each dataset, the total number of keyphrases (Kp), the average number of keyphrases per document (AvgKp), and a brief insight into the length and number of available keyphrases.
Evaluation Metrics. We use mean reciprocal rank (MRR) curves to illustrate our experimental findings. MRR gives the averaged ranking of the first correct prediction and is defined as:
MRR = 1|D| ∑ d∈D 1 rd
where D is the collection of documents and rd is the rank at which the first correct keyphrase of document d was found. We also summarize the results in terms of Precision, Recall, and F1score in a table to contrast PositionRank with previous models since these metrics are widely used in previous works (Hulth, 2003; Wan and Xiao, 2008; Mihalcea and Tarau, 2004; Hasan and Ng, 2014). To compute “performance@k” (such as MRR@k), we examine the top-k predictions (with k ranging from 1 to 10). We use average k to refer to the average number of keyphrases for a particular dataset as listed in Table 1. For example, average k = 5 for the WWW dataset. For comparison purposes, we used Porter Stemmer to reduce both predicted and gold keyphrases to a base form.
4.2 Results and Discussion
Our experiments are organized around several questions, which are discussed below.
How sensitive is PositionRank to its parameters? One parameter of our model that can influence its performance is the window size w, which determines how edges are added between candidate words in the graph. We experimented with values of w ranging from 2 to 10 in steps of 1 and chose several configurations for illustration. Figure 2 shows the MRR curves of PositionRank for different values of w, on all three datasets. As can be seen from the figure, the performance of our model does not change significantly as w changes.
In addition to the window size, our model has one more parameter, i.e., the damping factor α. In order to understand its influence on the performance of PositionRank, we experimented with several values of α, e.g., 0.75, 0.8, 0.85, 0.9, and did not find significant differences in the performance of PositionRank (results not shown due to highly overlapping curves). Hence, in Equation 2, we set α = 0.85 as in (Haveliwala, 2003).
What is the impact of aggregating information from all positions of a word over using a word’s first position only? In this experiment, we analyze the influence that position-weighted frequent words in a document would have on the performance of PositionRank. Specifically, we compare the performance of the model that aggregates information from all positions of a word’s occurrences, referred as PositionRank - full model with that of the model that uses only the first position
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Dataset #Docs Kp AvgKp unigrams bigrams trigrams n-grams (n ≥ 4) KDD 834 3093 3.70 810 1770 471 42 WWW 1350 6405 4.74 2254 3139 931 81 Nguyen 211 882 4.18 260 457 132 33
Table 1: A summary of our datasets.
Figure 2: MRR curves for PositionRank that uses different values for the window size.
of a word, referred as PositionRank - fp. In the example from the previous section, a word occurring on positions 2nd, 5th, and 10th will have a weight of 12 + 1 5 + 1 10 = 4 5 = 0.8 in the full model, and a weight of 12 = 0.5 in the first position (fp) model. Note that the weights of words are normalized before they are used in the biased PageRank.
Figure 3 shows the results of this experiment in terms of MRR for the top k predicted keyphrases, with k from 1 to 10, for all datasets, KDD, WWW, and Nguyen. As we can see from the figure, the performance of PositionRank - full model consistently outperforms its counterpart that uses the first position only, on all datasets. We can conclude from this experiment that aggregating information from all occurrences of a word acts as an important component in PositionRank. Hence, we use PositionRank - full model for further comparisons.
How well does position information aid in unsupervised keyphrase extraction from research papers? In this experiment, we compare our position-biased PageRank model (PositionRank) with two PageRank-based models, TextRank and SingleRank, that do not make use of the position information. In TextRank, an undirected graph is built for each target paper, so that nodes correspond to words and edges are drawn between two words that occur next to each other in text, i.e., the window sizew is 2. SingleRank extends TextRank by adding edges between two words that co-occur in a window of w ≥ 2 contiguous words in text.
Figure 4 shows the MRR curves comparing PositionRank with TextRank and SingleRank. As
can be seen from the figure, PositionRank substantially outperforms both TextRank and SingleRank on all three datasets, illustrating that the words’ positions contain significant hints that aid the keyphrase extraction task. PositionRank can successfully harness this information in an unsupervised setting to obtain good improvements in the extraction performance. For example, PositionRank that uses information from all positions of a word’s occurrences yields improvements in MRR@average k of 17.46% for KDD, 20.18% for WWW, and 17.03% for Nguyen over SingleRank.
How does PositionRank compare with other existing state-of-the-art methods? In Figure 5, we compare PositionRank with several strong baselines: TF-IDF, ExpandRank, and TopicalPageRank (TPR) (Hasan and Ng, 2014; Wan and Xiao, 2008; Liu et al., 2010). We selected these baselines based on the ACL survey on keyphrase extraction by Hasan and Ng (2014). In TF-IDF, we calculate the tf score of each candidate word in the target document, whereas the idf component is estimated from all three datasets. In ExpandRank, we build an undirected graph from each paper and its local textual neighborhood and calculate the candidate words’ importance scores using PageRank. We performed experiments with various numbers of textually-similar neighbors and present the best results for each dataset. In TPR, we build an undirected graph using information from the target paper. We then perform topic decomposition of the target document using topic models to infer the topic distribution
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Figure 3: The comparison of PositionRank that aggregates information from all positions of a word’s occurrences (full model) with the PositionRank that uses only the first position of a word (fp).
Figure 4: MRR curves for PositionRank and two unbiased PageRank-based models that do not consider position information.
of a document and to compute the probability of words in these topics. Last, we calculate the candidate words’ importance scores by aggregating the scores from several topic-biased PageRanks (one PageRank per topic). We used the implementation of topic models from Mallet.2 To train the topic model, we used a subset of about 45, 000 paper abstracts extracted from CiteSeerX. For all models, the score of a phrase is obtained by summing the score of the constituent words in the phrase.
From Figure 5, we can see that PositionRank achieves a significant increase in MRR over the baselines, on all datasets. For example, the highest relative improvement in MRR@average k for this experiment is as high as 29.09% achieved on the Nguyen collection. Among all models compared in Figure 5, ExpandRank is clearly the best performing baseline, while TPR achieves the lowest MRR values, on all datasets.
4.3 Overall Performance
As already mentioned, prior works on keyphrase extraction report results also in terms of precision (P), recall (R), and F1-score (F1) (Hulth, 2003; Hasan and Ng, 2010; Liu et al., 2010; Wan and
2http://mallet.cs.umass.edu/
Xiao, 2008). Consistent with these works, in Table 2, we show the results of the comparison of PositionRank with all baselines, in terms of P, R and F1 for top k = 2, 4, 6, 8 predicted keyphrases, on all three datasets. As can be seen from the table, PositionRank outperforms all baselines, on all datasets. For example, on WWW at top 6 predicted keyphrases, PositionRank achieves an F1score of 12.1% as compared to 11.2% achieved by ExpandRank and 10.7% achieved by both TFIDF and TPR. From the table, we can also see that ExpandRank is generally the best performing baseline on all datasets. However, it is interesting to note that, unlike PositionRank that uses information only from the target paper, ExpandRank adds external information from a textually-similar neighborhood of the target paper, and hence, is computationally more expensive.
PositionRank-first position only (fp) typically performs worse than PositionRank-full model, but it still outperforms the baseline methods for most top k predicted keyphrases, on all datasets. For example, on Nguyen at top 4, PositionRank-fp achieves an F1-score of 10.5% compared to the best baseline (TF-IDF in this case), which reaches only a score of 9.6%.
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Figure 5: MRR curves for PositionRank and baselines on the three datasets.
Dataset Unsupervised Top2 Top4 Top6 Top8 method P% R% F1% P% R% F1% P% R% F1% P% R% F1% KDD PositionRank 11.1 5.6 7.3 10.8 11.1 10.6 9.8 15.3 11.6 9.2 18.9 12.1 PositionRank-fp 10.3 5.3 6.8 10.2 10.4 10.0 9.1 13.8 10.9 8.6 17.2 11.3 TF-IDF 10.5 5.2 6.8 9.6 9.7 9.4 9.2 13.8 10.7 8.7 17.4 11.3 TextRank 8.1 4.0 5.3 8.3 8.5 8.1 8.1 12.3 9.4 7.6 15.3 9.8 SingleRank 9.1 4.6 6.0 9.3 9.4 9.0 8.7 13.1 10.1 8.1 16.4 10.6 ExpandRank 10.3 5.5 6.9 10.4 10.7 10.1 9.2 14.5 10.9 8.4 17.5 11.0 TPR 9.3 4.8 6.2 9.1 9.3 8.9 8.8 13.4 10.3 8.0 16.2 10.4 WWW PositionRank 11.3 5.3 7.0 11.3 10.5 10.5 10.8 14.9 12.1 9.9 18.1 12.3 PositionRank-fp 9.6 4.5 6.0 10.3 9.6 9.6 10.1 13.8 11.2 9.4 17.2 11.7 TF-IDF 9.5 4.5 5.9 10.0 9.3 9.3 9.6 13.3 10.7 9.1 16.8 11.4 TextRank 7.7 3.7 4.8 8.6 7.9 8.0 8.1 12.3 9.8 8.2 15.2 10.2 SingleRank 9.1 4.2 5.6 9.6 8.9 8.9 9.3 13.0 10.5 8.8 16.3 11.0 ExpandRank 10.4 5.3 6.7 10.4 10.6 10.1 9.5 14.7 11.2 8.6 17.7 11.2 TPR 8.8 4.2 5.5 9.6 8.9 8.9 9.5 13.2 10.7 9.0 16.5 11.2 Nguyen PositionRank 10.5 5.8 7.3 10.6 11.4 10.7 11.0 17.2 13.0 10.2 21.1 13.5 PositionRank-fp 10.0 5.4 6.8 10.4 11.1 10.5 11.2 17.4 13.2 10.1 21.2 13.3 TF-IDF 7.3 4.0 5.0 9.5 10.3 9.6 9.1 14.4 10.9 8.9 18.9 11.8 TextRank 6.3 3.6 4.5 7.4 7.4 7.2 7.8 11.9 9.1 7.2 14.8 9.4 SingleRank 9.0 5.2 6.4 9.5 9.9 9.4 9.2 14.5 11.0 8.9 18.3 11.6 ExpandRank 9.5 5.3 6.6 9.5 10.2 9.5 9.1 14.4 10.8 8.7 18.3 11.4 TPR 8.7 4.9 6.1 9.1 9.5 9.0 8.8 13.8 10.5 8.8 18.0 11.5
Table 2: PositionRank against baselines in terms of Precision, Recall and F1-score. Best results are shown in bold blue.
A striking observation is that PositionRank outperforms TPR on all datasets. Compared with our model, TPR is a very complex model, which uses topic models to learn topics of words and infer the topic proportion of documents. Additionally, TPR has more parameters (e.g., the number of topics) that need to be tuned separately for each dataset. PositionRank is much less complex, it does not require an additional dataset (e.g., to train a topic model) and its performance is better than that of TPR. TF-IDF and ExpandRank are the best performing baselines, on all datasets. For example, on KDD at k = 4, TF-IDF and ExpandRank yield an F1-score of 9.4% and 10.1%, respectively, compared with 8.4%, 9.0% and 8.9% achieved by TextRank, SingleRank and TPR, respectively.
With a paired t-test on our results, we found that the improvements in MRR, precision, recall, and F1-score for PositionRank are statistically significant (p-values < 0.05).
5 Conclusion and Future Work
We proposed a novel unsupervised graph-based algorithm, called PositionRank, which incorporates both the position of words and their frequency in a document into a biased PageRank. To our knowledge, we are the first to integrate the position information in novel ways in unsupervised keyphrase extraction. Specifically, unlike supervised approaches that use only the first position information, we showed that modeling the entire distribution of positions for a word outperforms models that use only the first position. Our experiments on three datasets of research papers show that our proposed model achieves better results than strong baselines, with relative improvements in performance as high as 29.09%.
In the future, it would be interesting to explore the performance of PositionRank on other types of documents, e.g., web pages and emails.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths: Nicely written and understandable. Clearly organized. Targeted answering of research questions, based on  different experiments.  - Weaknesses: Minimal novelty. The  first sentence  heuristic has been in the summarization literature for many years. This work essentially applies this heuristic (evolved) in the keyword extraction setting. This is NOT to say that the work is trivial: it is just not really novel.  Lack of state-of-the-art very recent methods. The experiment on the system evaluation vs state-of-the-art systems simply uses strong baselines. Even though the experiment answers the question  does it perform better than baselines? , I am not confident it illustrates that the system performs better than the current state-of-the-art. This somewhat reduces the value of the paper.  - General Discussion: Overall the paper is good and I propose that it be published and presented.   On the other hand, I would propose that the authors position themselves (and the system performance) with respect to: Martinez 0Romo, Juan, Lourdes Araujo, and Andres Duque Fernandez.  SemGraph: Extracting keyphrases following a novel semantic graph 0based approach.  Journal of the Association for Information Science and Technology 67.1 (2016): 71-82. (with which the work holds remarkable resemblance in some points)  Le, Tho Thi Ngoc, Minh Le Nguyen, and Akira Shimazu.  Unsupervised Keyphrase Extraction: Introducing New Kinds of Words to Keyphrases.  Australasian Joint Conference on Artificial Intelligence. Springer International Publishing, 2016.