Transductive Non-linear Learning for Chinese Hypernym Prediction
Finding the correct hypernyms for entities is essential for taxonomy learning, finegrained entity categorization, knowledge base construction, etc. Due to the flexibility of the Chinese language, it is challenging to identify hypernyms in Chinese accurately. Rather than extracting hypernyms from texts, in this paper, we present a transductive learning approach to establish mappings from entities to hypernyms in the embedding space directly. It combines linear and non-linear embedding projection models, with the capacity of encoding arbitrary language-specific rules. Experiments on real-world datasets illustrate that our approach outperforms previous methods for Chinese hypernym prediction.
1 Introduction
A hypernym of an entity characterizes the type or the class of the entity. For example, the word country is the hypernym of the entity Canada. The accurate prediction of hypernyms benefits a variety of NLP tasks, such as taxonomy learning (Wu et al., 2012; Fu et al., 2014), fine-grained entity categorization (Ren et al., 2016), knowledge base construction (Suchanek et al., 2007), etc.
In previous work, the detection of hypernyms requires lexical, syntactic and/or semantic analysis of relations between entities and their respective hypernyms from a language-specific knowledge source. For example, Hearst (1992) is the pioneer work to extract is-a relations from a text corpus based on handcraft patterns. The followingup work mostly focuses on is-a relation extraction using automatically generated patterns (Snow et al., 2004; Ritter et al., 2009; Sang and Hof-
mann, 2009; Kozareva and Hovy, 2010) and relation inference based on distributional similarity measures (Kotlerman et al., 2010; Lenci and Benotto, 2012; Shwartz et al., 2016).
While these approaches have relatively high precision over English corpora, extracting hypernyms for entities is still challenging for Chinese. From the linguistic perspective, Chinese is a lower-resourced language with very flexible expressions and grammatical rules (Wang et al., 2015). For instance, there are no word spaces, explicit tenses and voices, and distinctions between singular and plural forms in Chinese. The order of words can be changed flexibly in sentences. Hence, as previous research indicates, hypernym extraction methods for English are not necessarily suitable for the Chinese language (Fu et al., 2014; Wang et al., 2015; Wang and He, 2016).
Based on such conditions, several classification methods are proposed to distinguish is-a and notis-a relations based on Chinese encyclopedias (Lu et al., 2015; Li et al., 2015). Similar to Princeton WordNet, a few Chinese wordnets have also been developed (Huang et al., 2004; Xu et al., 2008; Wang and Bond, 2013). The most recent approaches for Chinese is-a relation extraction (Fu et al., 2014; Wang and He, 2016) use word embedding based linear projection models to map embeddings of hyponyms to those of their hypernyms, which outperform previous algorithms.
However, we argue that these projection-based methods may have three potential limitations: (i) Only positive is-a relations are used for projection learning. The distinctions between is-a and not-is-a relations in the embedding space are not modeled. (ii) These methods lack the capacity to encode linguistic rules, which are designed by linguists and usually have high precision. (iii) It assumes that the linguistic regularities of is-a relations can be solely captured by single or multiple
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
linear projection models. In this paper, we address these limitations by a two-stage transductive learning approach. It distinguishes is-a and not-is-a relations given a Chinese word/phrase pair as input. In the initial stage, we train linear projection models on positive and negative training data separately and predict isa relations jointly. In the transductive learning stage, the initial prediction results, linguistic rules and the non-linear mappings from entities to hypernyms are optimized simultaneously in a unified framework. This optimization problem can be efficiently solved by blockwise gradient descent. We evaluate our method over two public datasets and show that it outperforms state-of-the-art approaches for Chinese hypernym prediction.1
The rest of this paper is organized as follows. We summarize the related work in Section 2. Our approach is introduced in Section 3. Experimental results are presented in Section 4. We conclude our paper in Section 5.2
2 Related Work
In this section, we overview the related work on hypernym prediction and discuss the challenges of Chinese hypernym detection.
Pattern based methods identify is-a relations from texts by handcraft or automatically generated patterns. Hearst patterns (Hearst, 1992) are lexical patterns in English that are employed to extract isa relations for taxonomy construction (Wu et al., 2012). Automatic approaches mostly use iterative learning paradigms such that the system learns new is-a relations and patterns simultaneously. A few relevant studies can be found in (Caraballo, 1999; Etzioni et al., 2004; Sang, 2007; Pantel and Pennacchiotti, 2006; Kozareva and Hovy, 2010). To avoid “semantic drift” in iterations, Snow et al. (2004) train a hypernym classifier based on syntactic features based on parse trees. Carlson et al. (2010) exploit multiple learners to extract relations via coupled learning. These approaches are
1While there is abundant research on hypernym prediction for English with high precision, we mostly focus on the Chinese language in this paper. However, the proposed method is not entirely language-specific and has the potential to be adapted to other languages. In Section 4, we provide additional experiments to show that our approach also outperforms several existing methods for hypernym prediction in the English environment. We also discuss some potential applications of our method.
2Additionally, we present more details on algorithms and experimental settings and a prototype system for taxonomy visualization in the supplementary notes.
not effective for Chinese for two reasons: i) Chinese is-a relations are expressed in a highly flexible manner (Fu et al., 2014) and ii) the accuracy of basic NLP tasks such as dependency parsing still need improvement for Chinese (Li et al., 2013).
Inference based methods take advantage of distributional similarity measures (DSM) to infer relations between words. They assume that a hypernym may appear in all contexts of the hyponyms and a hyponym can only appear in part of the contexts of its hypernyms. In previous work, Kotlerman et al. (2010) design directional DSMs to model the asymmetric property of is-a relations. Other DSMs are introduced in (Bhagat et al., 2007; Szpektor et al., 2007; Lenci and Benotto, 2012; Santus et al., 2014). Shwartz et al. (2016) combine dependency parsing and DSM to improve the performance of hypernymy detection. The reason why DSM is not effective for Chinese is that the contexts of entities in Chinese are flexible and sparse.
Encyclopedia based methods take encyclopedias as knowledge sources to construct taxonomies. Ponzetto and Strube (2007) design features from multiple aspects to predict is-a relations between entities and categories in English Wikipedia. The taxonomy in YAGO (Suchanek et al., 2007) is constructed by linking conceptual categories in Wikipedia to WordNet synsets (Miller, 1995). For Chinese, Li et al. (2015) propose an SVM-based approach to build a large Chinese taxonomy from Wikipedia. Similar classification based algorithms are presented in (Fu et al., 2013; Lu et al., 2015). Due to the lack of Chinese version of WordNet, several Chinese semantic dictionaries have been conducted, such as Sinica BOW (Huang et al., 2004), SEW (Xu et al., 2008), COW (Wang and Bond, 2013), etc. These approaches have higher accuracy than mining hypernym relations from texts directly. However, they heavily rely on existing knowledge sources and are difficult to extend to different domains.
To tackle these challenges, word embedding based methods directly model the task of hypernym prediction as learning a mapping from entity vectors to their respective hypernym vectors in the embedding space. The vectors can be pretrained by neural language models (Mikolov et al., 2013). For the Chinese language, Fu et al. (2014) train piecewise linear projection models based on a Chinese thesaurus. The state-of-the-art method
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
(Wang and He, 2016) combines an iterative learning procedure and Chinese Hearst-style patterns to improve the performance of projection models. They can reduce data noise by avoiding direct parsing of Chinese texts, but still capture the linguistic regularities of is-a relations based on word embeddings. Additionally, several work aims to study how to combine word embeddings for relation classification, such as (Mirza and Tonelli, 2016). In our paper, we extend these approaches by modeling non-linear mappings from entities to hypernyms and adding linguistic rules via a unified transductive learning framework.
3 Proposed Approach
This section begins with a brief overview of our approach. After that, the detailed steps and the learning algorithm are introduced in detail.
3.1 Overview
Given a word/phase pair (xi, yi), the goal of our task is to learn a classification model to predict whether yi is the hypernym of xi.
As illustrated in Figure 1, our approach has two stages: initial stage and transductive learning stage. The input is a positive is-a set D+, a negative is-a set D− and an unlabeled set DU , all of which are the collections of word/phrase pairs.
Denote xi as the embedding vector of word xi, pre-trained and stored in a lookup table. In the initial stage, we train a linear projection model over D+ such that for each (xi, yi) ∈ D+, a projection matrix maps the entity vector xi to its hypernym vector yi. A similar model is also trained over D−. Based on the two models, we estimate the prediction score and the confidence score for each (xi, yi) ∈ DU . In the transductive learning stage, a joint optimization problem is formed to learn the final prediction score for each (xi, yi) ∈ DU . It aims to minimize the prediction errors based on the human labeled data, the initial model prediction and linguistic rules. It also employs nonlinear mappings to capture linguistic regularities of is-a relations other than linear projections.
3.2 Initial Model Training
The initial stage models how entities are translated to their hypernyms or non-hypernyms by projection learning. We first train a Skip-gram model (Mikolov et al., 2013) to learn word embeddings over a large text corpus. Inspired by
(Fu et al., 2014; Wang and He, 2016), for each (xi, yi) ∈ D+, we assume there is a positive projection model such that M+xi ≈ yi where M+ is an |xi| × |xi| projection matrix3. However, this model does not capture the semantics of not-isa relations. Thus, we learn a negative projection model M−xi ≈ yi where (xi, yi) ∈ D−. This approach is equivalent to learning two separate translation models within the same semantic space. For parameter estimation, we minimize the two following objectives:
J(M+) = 1
2 ∑ (xi,yi)∈D+ ‖M+xi−yi‖22+ λ 2 ‖M+‖2F
J(M−) = 1
2 ∑ (xi,yi)∈D− ‖M−xi−yi‖22+ λ 2 ‖M−‖2F
where λ > 0 is a Tikhonov regularization parameter (Golub et al., 1999).
In the testing phase, for each (xi, yi) ∈ DU , denote d+(xi, yi) = ‖M+xi − yi‖2 and d−(xi, yi) = ‖M−xi−yi‖2. The prediction score is defined as:
score(xi, yi) = tanh(d −(xi, yi)− d+(xi, yi))
where score(xi, yi) ∈ (−1, 1). Higher prediction score indicates there is a larger probability of an is-a relation between xi and yi. We choose the hyperbolic tangent function rather than the sigmoid function to avoid the widespread saturation of sigmoid function (Menon et al., 1996).
The difference between d+(xi, yi) and d−(xi, yi) can be also used to indicate whether the models are confident enough to make a prediction. In this paper, we calculate the confidence score as:
conf(xi, yi) = |d+(xi, yi)− d−(xi, yi)| max{d+(xi, yi), d−(xi, yi)} 3We have also examined piecewise linear projection models proposed in (Fu et al., 2014; Wang and He, 2016) as the initial models for transductive learning. However, we found that this practice is less efficient and the performance does not improve significantly.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
where conf(xi, yi) ∈ (0, 1). Higher confidence score means that there is a larger probability that the models can predict whether there is an is-a relation between xi and yi correctly. This score gives different data instances different weights in the transductive learning stage.
3.3 Transductive Non-linear Learning
Although linear projection methods are effective for Chinese hypernym prediction, it does not encode non-linear transformation and only leverages the positive data. We present an optimization framework for non-linear mapping utilizing both labeled and unlabeled data and linguistic rules by transductive learning (Gammerman et al., 1998; Chapelle et al., 2006).
Let Fi be the final prediction score of the word/phrase pair (xi, yi). In the initialization stage of our algorithm, we set Fi = 1 if (xi, yi) ∈ D+, Fi = −1 if (xi, yi) ∈ D− and set Fi randomly in (−1, 1) if (xi, yi) ∈ DU . In matrix representation, denote F as the m× 1 final prediction vector where m = |D+| + |D−| + |DU |. Fi is the ith element in F. The three components in our transductive learning model are as follows:
3.3.1 Initial Prediction
Denote S as an m×1 initial prediction vector. We set Si = 1 if (xi, yi) ∈ D+, Si = −1 if (xi, yi) ∈ D− and Si = score(xi, yi) if (xi, yi) ∈ DU . In order to encode the confidence of model prediction, we define W as an m ×m diagonal weight matrix. The element in the ith row and the jth column of W is set as follows:
Wi,j =  conf(xi, yi) i = j, (xi, yi) ∈ DU 1 i = j, (xi, yi) ∈ D+ ∪D−
0 Otherwise
The objective function is defined as: Os = ‖W(F−S)‖22, which encodes the hypothesis that the final prediction should be similar to the initial prediction for unlabeled data or human labeling for training data. The weight matrix W gives the largest weight (i.e., 1) to all the pairs in D+ ∪D− and a larger weight to the pair (xi, yi) ∈ DU if the initial prediction is more confident.
3.3.2 Linguistic Rules
Although linguistic rules can only cover a few circumstances, they are effective to guide the learning process. For Chinese hypernym prediction,
Li et al. (2015) study the word formation of conceptual categories in Chinese Wikipedia. In our model, let C be the collection of linguistic rules. γi is the true positive (or negative) rate with respect to the respective positive (or negative) rule ci ∈ C, estimated over the training set. Considering the word formation of Chinese entities and hypernyms, we design one positive rule (i.e., P1) and two negative rules (i.e., N1 and N2), shown in Table 1.
Let R be an m × 1 linguistic rule vector and Ri is the ith element in R. For training data, we set Ri = 1 if (xi, yi) ∈ D+ and Ri = −1 if (xi, yi) ∈ D−, which follows the same settings as those in S. For unlabeled pairs that do not match any linguistic rules in C, we update Ri = Fi in each iteration of the learning process, meaning no loss for errors imposed in this part.
For other conditions, denote C(xi,yi) ⊆ C as the collection of rules that (xi, yi) matches. If C(xi,yi) are positive rules, we set Ri as follows:
Ri = max{Fi, max cj∈C(xi,yi) γj}
Similarly, if C(xi,yi) are negative rules, we have:
Ri = −max{−Fi, max cj∈C(xi,yi) γj}
which means Fi receives a penalty only if Fi < maxcj∈C(xi,yi) γj for pairs that match positive rules or Fi > −maxcj∈C(xi,yi) γj for negative rules4. The objective function is: Or = ‖F−R‖22. In this way, our model can integrate arbitrary “soft” constraints, making it robust to false positives or negatives introduced by these rules.
3.3.3 Non-linear Learning
TransLP is a transductive label propagation framework (Liu and Yang, 2015) for link prediction, previously used for applications such as text classification (Xu et al., 2016). In our work, we extend their work for our task, modeling non-linear mappings from entities to hypernyms.
For is-a relations, we find that if y is the hypernym of x, it is likely that y is the hypernym of entities that are semantically close to x. For example,
4We do not consider the cases where a pair matches both positive and negative rules because such cases are very rare, and even non-existent in our datasets. However, our method can deal with these cases by using some simple heuristics. For example, we can update Ri using either of the following two ways: i) Ri = Fi and ii) Ri = Fi + ∑ cj∈C(xi,yi) γj .
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
P1 The head word of the entity x matches that of the candidate hypernym y. For example,动物 (Animal) is the correct hypernym of哺乳动物 (Mammal). N1 The head word of the entity x matches the non-head word of the candidate hypernym y. For example,动物学 (Zoology) is not a hypernym of哺乳动物 (Mammal). N2 The head word of the candidate hypernym y matches an entry in a Chinese lexicon extended based on the lexicon used in Li et al., (2015). It consists of 184 non-taxonomic, thematic words such as政治(Politics),军事(Military), etc.
Table 1: Three linguistic rules used in our work for Chinese hypernym prediction.
if we know United States is a country, we can infer country is the hypernym of similar entities such as Canada, Australia, etc. This intuition can be encoded in the similarity of the two pairs pi = (xi, yi) and pj = (xj , yj):
sim(pi, pj) =
{ cos(xi,xj) yi = yj
0 otherwise (1)
where xi is the embedding vector of xi5. This similarity indicates there exists a nonlinear mapping from entities to hypernyms, which can not be encoded in linear projection based methods (Fu et al., 2014; Wang and He, 2016). Based on TransLP (Liu and Yang, 2015), this intuition can be model as propagating class labels (is-a or not-is-a) of labeled word/phrase pairs to similar unlabeled ones based on Eq. (1). For example, the score of is-a relations between United State and country will propagate to pairs such as (Canada, country) and (Australia, country) by random walks.
Denote F∗ as the optimal solution of the problem min Os + Or. Inspired by (Liu and Yang, 2015; Xu et al., 2016), we can add a Gaussian prior N(F∗,Σ) to F where Σ is the covariance matrix and Σi,j = sim(pi, pj). Hence the optimization objective of this part is defined as: On = FTΣ−1F which is linearly proportional to the negative likelihood of the Gaussian random field prior. This means we minimize the training error and encourage F to have a smooth propagation with respect to the similarities among pairs defined by Eq. (1) at the same time.
3.3.4 Joint Optimization
Based on the three components, we minimize the following function:
J(F) = Os + Or + µ1 2 On + µ2 2 ‖F‖22 (2)
5We only consider the similarity between entities and not candidate hypernyms because the similar rule for candidate hypernyms is not true. For example, nouns close to country in our Skip-gram model are region, department, etc. They are not all correct hypernyms of United States, Canada, Australia, etc.
where ‖F‖22 imposes an additional smooth l2regularization on F. µ1 and µ2 are tuning parameters.
A basic approach to learn the optimal values of F is via gradient descent. Based on Eq. (2), the derivative of F with respect to J(F) can be computed as follows:
dJ(F) dF = W2(F−S)+(F−R)+µ1Σ−1F+µ2F
Optimizing Eq. (2) is computationally expensive when m is large. After W2, S, R and Σ−1 are pre-computed, the runtime complexity of the loop of gradient descent is O(tm2) where t is the number of iterations.
To speed up the learning process, we introduce a blockwise gradient descent technique. From the definition of Eq. (2), we can see that the optimal values of Fi and Fj with respect to (xi, yi) and (xj , yj) are irrelevant if yi 6= yj . Therefore, the original optimization problem can be decomposed and solved separately according to different candidate hypernyms.
LetH be the collection of candidate hypernyms in DU . For each h ∈ H , denote Dh as the collection of word/phase pairs in D+ ∪ D− ∪ DU that share the same candidate hypernym h. The original problem can be decomposed into |H| optimization subproblems over Dh for each h ∈ H . The runtime complexity is O( ∑ h∈Dh th|Dh|
2) where th is the number of iterations to solve the subproblem over Dh. Although we do not know the upper bounds on the numbers of iterations of these two learning techniques, the runtime complexity can be reduced by blockwise gradient descent for two reasons: i) ∑ h∈Dh |Dh| ≤ m and ii) th has a large probability to be smaller than t due to the smaller number of data instances. This technique can be also viewed as optimizing Eq. (2) based on blockwise matrix computation.
Finally, for each (xi, yi) ∈ DU , we predict that yi is a hypernym of xi if Fi > θ where θ ∈ (−1, 1) is a threshold tuned on the development set.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
4 Experiments
In this section, we conduct experiments to evaluate our method. Section 4.1 to Section 4.5 report the experimental steps on Chinese datasets. We present the performance on English datasets in Section 4.6 and a discussion in Section 4.7.
4.1 Experimental Data
The Chinese text corpus is extracted from the contents of 1.2M entity pages from Baidu Baike6, a Chinese online encyclopedia. It contains approximately 1.1B words. We use the open source toolkit Ansj7 for Chinese word segmentation.
We have two collections of Chinese word/phase pairs as ground truth datasets. Each pair is labeled with an is-a or not-is-a tag. The first one (denoted as FD) is from Fu et al., (2014), containing 1,391 is-a pairs and 4,294 not-is-a pairs, which is the first publicly available dataset to evaluate this task. The second one (denoted as BK) is larger in size and crawled from Baidu Baike by ourselves, consisting of <entity, category> pairs. For each pair in BK, we ask multiple human annotators to label the tag and discard the pair with inconsistent labels by different annotators. In total, it contains 3,870 is-a pairs and 3,582 not-is-a pairs8.
In the following experiments, we use 60% of the data for training, 20% for development and 20% for testing, partitioned randomly. By rotating the 5-fold subsets of the datasets, we report the performance of each method on average.
4.2 Parameter Analysis
The word embeddings are pre-trained by ourselves on the Chinese corpus. In total, we obtain the 100- dimensional embedding vectors of 5.8M distinct words. The regularization parameters are set to λ = 10−3 and µ1 = µ2 = 10−4, fine tuned on the development set.
The choice of θ reflects the precision-recall trade-off in our model. A larger value of θ means we pay more attention to precision rather than recall. Figure 2 illustrates the precision-recall curves on both datasets. It can be seen that the performance of our method is generally better in BK than FD. The most probable cause is that BK is a large dataset with more “balanced” numbers of positive
6https://baike.baidu.com/ 7https://github.com/NLPchina/ansj seg/ 8Our dataset is publicly available. See submitted dataset.
0.0 0.2 0.4 0.6 0.8 1.0 0.0
0.2
0.4
0.6
0.8
1.0
Recall
Pr ec is io n
(a) Dataset: FD
0.0 0.2 0.4 0.6 0.8 1.0 0.0
0.2
0.4
0.6
0.8
1.0
Recall
Pr ec is io n
(b) Dataset: BK
Figure 2: Precision-recall curve with respect to the tuning of θ on development sets (%).
and negative data. Finally, θ is set to 0.05 on FD and 0.1 on BK.
4.3 Performance
In a series of previous work (Fu et al., 2013, 2014; Wang and He, 2016), several pattern-based, inference-based and encyclopedia-based is-a relation extraction methods for English have been implemented for the Chinese language. As their experiments show, these methods achieve the Fmeasure of lower than 60% in most cases, which are not suggested to be strong baselines for Chinese hypernym prediction. Interested readers may refer to their papers for the experimental results.
To make the convincing conclusion, we employ two recent state-of-the-art approaches for Chinese is-a relation identification (Fu et al., 2014; Wang and He, 2016) as baselines. We also take the word embedding based classification approach (Mirza and Tonelli, 2016)9 and Chinese Wikipedia based SVM model (Li et al., 2015) as baselines to predict is-a relations between words10. The experimental results are illustrated in Table 3.
For Fu et al., (2014), we test the performance using a linear projection model (denoted as S in Table 3) and piecewise projection models (P). It shows that the semantics of is-a relations are better modeled by multiple projection models, with a slightly improvement in F-measure. By combining iterative projection models and pattern-based validation, the most recent approach (Wang and He, 2016) increases the F-measure by 4% and 2% in two datasets. In this method, the patternbased statistics are calculated using the same cor-
9Although the experiments in their paper are mostly related to temporal relations, the method can be applied to is-a relations without modification.
10Previously, these methods used different knowledge sources to train models and thus the results in their papers are not directly comparable with ours. To make fair comparison, we take the training data as the same knowledge source to train models for all methods.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Candidate Hypernym P T Candidate Hypernym P T Entity: 乙烯(Ethylene) Entity: 孙燕姿(Stefanie Sun) 化学品(Chemical) √ √ 歌手(Singer) √ √ 有机化学(Organic Chemistry) × × 明星(Star) √ √ 有机物(Organics) √ √ 人物(Person) √ √ 气体(Gas) √ √ 金曲奖 (Golden Melody Award) √
× 自然科学(Natural Science) × × 音乐人(Musician) √ √ Entity: 显卡(Graphics Card) Entity: 核反应堆(Nuclear Reactor) 硬件(Hardware) √ √ 建筑学(Architecture) × × 电子产品(Electronic Product) √ √
核科学(Nuclear Science) × × 电脑硬件(Computer Hardware) √ √ 核能 (Nuclear Energy) √ × 数码(Digit) × × 自然科学(Natural Science) × ×
Table 2: Examples of model prediction. (P: prediction result, T: ground truth, √ : positive, ×: negative)
Dataset FD Method P R F Fu et al., (2014) (S) 64.1 56.0 59.8 Fu et al., (2014) (P) 66.4 59.3 62.6 Li et al., (2015) 54.3 38.4 45.0 Mirza and Tonelli, (2016) (C) 67.7 75.2 69.7 Mirza and Tonelli, (2016) (A) 65.3 60.7 62.9 Mirza and Tonelli, (2016) (S) 71.9 60.6 65.7 Wang and He, (2016) 69.3 64.5 66.9 Ours (Initial) 70.7 69.2 69.9 Ours 72.8 70.5 71.6 Dataset BK Method P R F Fu et al., (2014) (S) 71.4 64.8 67.9 Fu et al., (2014) (P) 72.7 67.5 70.0 Li et al., (2015) 61.2 47.5 53.5 Mirza and Tonelli, (2016) (C) 80.3 75.9 78.0 Mirza and Tonelli, (2016) (A) 72.7 65.6 68.9 Mirza and Tonelli, (2016) (S) 78.4 60.7 68.4 Wang and He, (2016) 73.9 69.8 71.8 Ours (Initial) 81.7 78.5 80.0 Ours 83.6 80.6 82.1
Table 3: Performance comparison on test sets for Chinese hypernym prediction (%).
pus over which we train word embedding models. The main reason of the improvement may be that the projection models have a better generalization power by applying an iterative learning paradigm.
Mirza and Tonelli, (2016) is implemented using three different strategies in combining the word vectors of a pair: i) concatenation xi ⊕ yi (denoted as C), ii) addition xi + yi (A) and iii) subtraction xi − yi (S). As seen, the classification models using addition and subtraction have similar performance in two datasets, while the concatenation strategy outperforms previous two approaches. Although Li et al., (2015) achieve a high performance in their dataset, this method does not perform well in ours. The most case is that the features in that work are designed specifically for the Chinese Wikipedia category system. Our initial model has a higher accuracy than all the base-
TP/TN Rate Rule P1 Rule N1 Rule N2 Dataset FD 98.6 92.3 94.1 Dataset BK 97.6 96.8 97.3
Table 4: TP/TN rates of three linguistic rules (%).
lines. By utilizing the transductive learning framework, we boost the F-measure by 1.7% and 2.1%, respectively. Therefore, our method is effective to predict hypernyms of Chinese entities.
4.4 Effectiveness of Linguistic Rules
To illustrate the effectiveness of linguistic rules, we present the true positive (or negative) rate by using one positive (or negative) rule solely, shown in Table 4. These values serve as γis in the transductive learning stage. The results indicate that these rules have high precision (over 90%) over both datasets for our task.
We state that currently we only use a few handcraft linguistic rules in our work. The proposed approach is a general framework that can encode arbitrary numbers of rules of any kind and in any language.
4.5 Error Analysis and Case Studies
We analyze correct and error cases in the experiments. Some examples of prediction results are shown in Table 2. We can see that our method is generally effective. However, some mistakes occur mostly because it is difficult to distinguish strict is-a and topic-of relations. For example, the entity Nuclear Reactor is semantically close to Nuclear Energy. The error statistics show that such kind of errors account for approximately 80.2% and 78.6% in two test sets, respectively.
Based on the literature study, we find that such problem has been also reported in (Fu et al., 2013; Wang and He, 2016). To reduce such errors, we employ the Chinese thematic lexicon based on Li
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
et al., (2015) in the transductive learning stage but the coverage is still limited. Two possible solutions are: i) adding more negative training data of this kind; and ii) constructing a large-scale thematic lexicon automatically from the Web.
4.6 Experiments on English Datasets
To examine how our method can benefit hypernym prediction for the English language, we use two standard datasets in this paper. The first one is a benchmark dataset for distributional semantic evaluation, i.e., BLESS (Baroni and Lenci, 2011). Because the number of pairs in BLESS is relatively small, we also use the Shwartz (Shwartz et al., 2016) dataset. In the experiments, we treat the HYPER relations as positive data (1,337 pairs) and randomly sample 30% of the RANDOM relations as negative data (3,754 pairs) in BLESS. To create a relatively balanced dataset, we take the random split of Shwartz as input and use only 30% of the negative pairs. The dataset contains 14,135 positive pairs and 16,956 negative pairs. The text corpus is English Wikipedia and the embedding vectors are set to 100 dimensions.
For comparison, we test all the baselines over English datasets except Li et al., (2015). This is because most features in Li et al., (2015) can only be used in the Chinese environment. To implement Wang and He., (2016) for English, we use the original Hearst patterns (Hearst, 1992) to perform relation selection and do not consider not-isa patterns. We also take two recent DSM based approaches (Lenci and Benotto, 2012; Santus et al., 2014) as baselines. As for our own method, we do not use linguistic rules in Table 1 for English. The results are illustrated in Table 5. As seen, our method is superior to all the baselines over BLESS, with an F-measure of 81.9%. In Shwartz, while the approach (Mirza and Tonelli, 2016) has the highest F-measure of 80.1%, our method is generally comparable to theirs and outperforms others. The results suggest that although our method is not necessarily the state-of-the-art for English hypernym prediction, it has several potential applications. Refer to Section 4.7 for discussion.
4.7 Discussion
From the experiments, we can see that the proposed approach outperforms the state-of-the-art methods for Chinese hypernym prediction. Although the English language is not our focus, our approach still has relatively high performance.
Dataset BLESS Method P R F Lenci and Benotto, (2012) 42.8 38.6 40.6 Santus et al., (2014) 59.2 52.3 55.4 Fu et al., (2014) (S) 65.3 62.4 63.8 Fu et al., (2014) (P) 68.1 64.2 66.1 Mirza and Tonelli, (2016) (C) 79.4 84.1 81.7 Mirza and Tonelli, (2016) (A) 80.7 72.3 76.3 Mirza and Tonelli, (2016) (S) 78.0 81.2 79.6 Wang and He, (2016) 76.2 75.4 75.8 Ours (Initial) 79.3 76.3 77.7 Ours 84.4 79.5 81.9 Dataset Shwartz Method P R F Lenci and Benotto, (2012) 38.5 50.1 43.5 Santus et al., (2014) 51.2 71.5 59.6 Fu et al., (2014) (S) 65.6 66.1 65.8 Fu et al., (2014) (P) 62.3 71.9 67.3 Mirza and Tonelli, (2016) (C) 79.3 80.9 80.1 Mirza and Tonelli, (2016) (A) 79.1 79.6 79.4 Mirza and Tonelli, (2016) (S) 80.5 77.5 79.0 Wang and He, (2016) 75.1 76.3 75.6 Ours (Initial) 77.2 76.8 77.0 Ours 79.1 77.5 78.3
Table 5: Performance comparison on test sets for English hypernym prediction (%).
Additionally, our work has potential values for the following applications:11
Domain-specific or Context-sparse Relation Extraction. If the task is to predict relations between words when it is related to a specific domain or the contexts are sparse, even for English, traditional pattern-based methods are likely to fail. Our method can predict the existence of relations without explicit textual patterns and requires a relatively small amount of pairs as training data.
Under-resourced Language Learning. Our method can be adapted for relation extraction in languages with flexible expressions, few knowledge resources and/or low-performance NLP tools. This is because it does not require deep NLP parsing of sentences in a text corpus.
5 Conclusion
In summary, this paper introduces a transuctive learning approach for Chinese hypernym prediction. By modeling linear projection models, linguistic rules and non-linear mappings, our method is able to identify Chinese hypernyms with high accuracy. Experiments show that the performance of our method outperforms previous approaches. We also discuss the potential applications of our method besides Chinese hypernym prediction.
11The implementations of these applications are beyond the scope of this paper.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

The paper is clearly written, and the claims are well-supported.  The Related Work in particular is very thorough, and clearly establishes where the proposed work fits in the field.  I had two main questions about the method: (1) phrases are mentioned in section 3.1, but only word representations are discussed.  How are phrase representations derived? (2) There is no explicit connection between M^+ and M^- in the model, but they are indirectly connected through the tanh scoring function.  How do the learned matrices compare to one another (e.g., is M^- like -1*M^+?)?  Furthermore, what would be the benefits drawbacks of linking the two together directly, by enforcing some measure of dissimilarity?  Additionally, statistical significance of the observed improvements would be valuable.  Typographical comments: - Line 220:  word phase pair  should be  word phrase pair  - Line 245: I propose an alternate wording: instead of  entities are translated to,  say  entities are mapped to .  At first, I read that as a translation operation in the vector space, which I think isn't exactly what's being described. - Line 587:  slightly improvement in F-measure  should be  slight improvement in F-measure  - Line 636: extraneous commas in citation - Line 646:  The most case  should be  The most likely case  (I'm guessing) - Line 727: extraneous period and comma in citation


Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision
Harnessing the statistical power of neural networks to perform language understanding and symbolic reasoning is difficult, when it requires executing efficient discrete operations against a large knowledge-base. In this work, we introduce a Neural Symbolic Machine, which contains (a) a neural “programmer”, i.e., a sequence-to-sequence model that maps language utterances to programs and utilizes a key-variable memory to handle compositionality (b) a symbolic “computer”, i.e., a Lisp interpreter that performs program execution, and helps find good programs by pruning the search space. We apply REINFORCE to directly optimize the task reward of this structured prediction problem. To train with weak supervision and improve the stability of REINFORCE we augment it with an iterative maximum-likelihood process. NSM outperforms state-of-the-art on the WEBQUESTIONSSP dataset when trained from question-answer pairs only, without requiring any feature engineering or domain-specific knowledge.
1 Introduction
Deep neural networks have achieved impressive performance in supervised classification and structured prediction tasks such as speech recognition (Hinton et al., 2012), machine translation (Bahdanau et al., 2014; Wu et al., 2016) and more. However, training neural networks for semantic parsing (Zelle and Mooney, 1996; Zettlemoyer and Collins, 2005; Liang et al., 2011) or program induction, where language is mapped to a symbolic representation that is executed by a com-
puter, through weak supervision remains challenging. This is because the model must interact with a computer through non-differentiable operations at training time.
In semantic parsing, recent work handled this (Dong and Lapata, 2016; Jia and Liang, 2016) by training from manually annotated programs and avoiding program execution at training time. However, annotating programs is known to be expensive and scales poorly. In program induction, attempts to address this problem (Graves et al., 2014; Reed and de Freitas, 2016; Kaiser and Sutskever, 2015; Graves et al., 2016b; Andreas et al., 2016) either utilized low-level memory (Zaremba and Sutskever, 2015), or required memory to be differentiable (Neelakantan et al., 2015) so that the model can be trained with backpropagation. This makes it difficult to use the efficient discrete operations and memory of a traditional computer, and limited the application to synthetic or small knowledge bases.
In this paper, we propose to utilize the mem-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
ory and discrete operations of a traditional computer in a novel Manager-Programmer-Computer (MPC) framework for neural program induction, which integrates three components:
1. A “manager” that provides weak supervision (e.g., ‘NYC’ in Figure 1) through a reward indicating how well a task is accomplished. Unlike full supervision, weak supervision is easy to obtain at scale (Section 3.1). 2. A “programmer” that takes natural language as input and generates a program that is a sequence of tokens (Figure 2). The programmer learns from the reward and must overcome the hard search problem of finding correct programs (Section 2.2). 3. A “computer” that executes programs in a high level programming language. Its nondifferentiable memory enables abstract, scalable and precise operations, but makes training more challenging. It provides a friendly neural computer interface by detecting and eliminating invalid choices (Section 2.1).
Within this framework, we introduce the Neural Symbolic Machine (NSM) and apply it to semantic parsing. NSM contains a sequence-to-sequence (seq2seq) “programmer” (Sutskever et al., 2014) and a symbolic non-differentiable Lisp interpreter (“computer”) that executes programs against a large knowledge-base (KB).
Our technical contribution in this work is threefold. First, to support language compositionality, we augment the standard seq2seq model with a key-variable memory to save and reuse intermediate execution results (Figure 1). This is a novel application of pointer networks (Vinyals et al., 2015) to compositional semantics.
Second, to alleviate the search problem of finding correct programs when training from questionanswer pairs (Figure 1), we use the computer to execute partial programs and prune the programmer’s search space by checking the syntax and semantics of generated programs. This generalized past work on training from weak supervision (Liang et al., 2011; Berant et al., 2013) by using the semantic denotations for structural search.
Third, to train from weak supervision and directly maximize the expected reward we turn to the REINFORCE (Williams, 1992) algorithm. Since learning from scratch is difficult for REINFORCE, we combine it with an iterative maximum likelihood (ML) process, where beam search is
used to find pseudo-gold programs, which are then used to augment the objective of REINFORCE.
On the WEBQUESTIONSSP dataset (Yih et al., 2016), NSM achieves new state-of-the-art results with weak supervision, significantly closing the gap between weak and full supervision for this task. Unlike prior work, it is trained end-toend, and does not require feature engineering or domain-specific knowledge.
2 Neural Symbolic Machines
We now introduce NSM by first describe the “computer”, a non-differentiable Lisp interpreter that executes programs against a large KB and provides code assistance (Section 2.1). We then propose a seq2seq model (“programmer”) that supports compositionality using a key-variable memory to save and reuse intermediate results (Section 2.2). Finally, we describe a training procedure that is based on REINFORCE, but is augmented with pseudo-gold programs found by an iterative ML training procedure (Section 2.3).
Before diving into details, we define the semantic parsing task: given a knowledge base K, and a question x = (w1, w2, ..., wm), produce a program or logical form z that when executed against K generates the right answer y. Let E denote a set of entities (e.g., ABELINCOLN),1 and let P denote a set of properties (e.g., PLACEOFBIRTH). A knowledge base K is a set of assertions or triples (e1, p, e2) ∈ E × P × E , such as (ABELINCOLN, PLACEOFBIRTH, HODGENVILLE).
2.1 Computer: Lisp Interpreter with Code Assistance
Semantic parsing typically requires using a set of operations to query the knowledge base and process the results. Operations learned with neural networks such as addition and sorting do not perfectly generalize to inputs that are larger than the ones observed in the training data (Graves et al., 2014; Reed and de Freitas, 2016). In contrast, operations implemented in high level programming languages are abstract, scalable, and precise, thus generalizes perfectly to inputs of arbitrary size. Based on this observation, we implement operations necessary for semantic parsing with an ordinary programming language instead of trying to learn them with a neural network.
1We also consider numbers (e.g., “1.33”) and date-times (e.g., “1999-1-1”) as entities.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
( Hop v p )⇒ {e2|e1 ∈ v, (e1, p, e2) ∈ K} ( ArgMax v p )⇒ {e1|e1 ∈ v,∃e2 ∈ E : (e1, p, e2) ∈ K,∀e : (e1, p, e) ∈ K, e2 ≥ e} ( ArgMin v p )⇒ {e1|e1 ∈ v,∃e2 ∈ E : (e1, p, e2) ∈ K,∀e : (e1, p, e) ∈ K, e2 ≤ e}
( Filter v1 v2 p )⇒ {e1|e1 ∈ v1,∃e2 ∈ v2 : (e1, p, e2) ∈ K}
Table 1: Interpreter functions. v represents a variable, p a property in Freebase. ≥ and ≤ are defined on numbers and dates.
We adopt a Lisp interpreter as the “computer”. A program C is a list of expressions (c1...cN ), where each expression is either a special token ”RETURN” indicating the end of the program, or a list of tokens enclosed by parentheses ”( F A0 ... AK )”. F is one of the functions in Table 1, which takes as input K arguments of specific types. Table 1 defines the semantics of each function and the types of arguments (either a property p or a variable v). When functions are executed, they return an entity list that is the expression’s denotation in K, and save it to a new variable.
By introducing variables that save the intermediate results of execution, the program naturally models language compositionality and describes from left to right a bottom-up derivation of the full meaning of the natural language input, which is convenient in a seq2seq model (Figure 1). This is reminiscent of the floating parser (Wang et al., 2015; Pasupat and Liang, 2015), where a derivation tree that is not grounded in the input is incrementally constructed.
The set of programs defined by our functions is equivalent to the subset of λ-calculus presented in (Yih et al., 2015). We did not use full Lisp programming language here, because constructs like control flow and loops are unnecessary for most current semantic parsing tasks, and it is simple to add more functions to the model when necessary.
To create a friendly neural computer interface, the interpreter provides code assistance to the programmer by producing a list of valid tokens at each step. First, a valid token should not cause a syntax error: e.g., if the previous token is “(”, the next token must be a function name, and if the previous token is “Hop”, the next token must be a variable. More importantly, a valid token should not cause a semantic (run-time) error: this is detected using the denotation saved in the variables. For example, if the previously generated tokens were “( Hop v”, the next available token is restricted to properties {p | ∃e, e′ : e ∈ v, (e, p, e′) ∈ K} that are reachable from entities in v in the KB graph. These checks are enabled by the variables and can
be derived from the definition of the functions in Table 1. With this code assistance, the interpreter prunes the “programmer”’s search space by orders of magnitude, and enables learning from weak supervision on a large KB.
2.2 Programmer: Seq2seq Model with Key-Variable Memory
Given the “computer”, the “programmer” needs to map natural language into a program, which is a sequence of tokens that references operations and values in the “computer”. We base our programmer on a standard seq2seq model with attention, but extend it with a key-variable memory that allows the model to learn to represent and refer to program variables (Figure 2).
Sequence-to-sequence models consist of two RNNs, an encoder and a decoder. We used a 1-layer GRU (Cho et al., 2014) for both the encoder and decoder. Given a sequence of words w1, w2...wm, each word wt is mapped to an embedding qt (embedding details are Section 3). Then, the encoder reads these embeddings and updates its hidden state step by step using ht+1 = GRU(ht, qt, θEncoder), where θEncoder are the GRU parameters. The decoder updates its hidden states ut by ut+1 = GRU(ut, ct−1, θDecoder), where ct−1 is the embedding of last step’s output token at−1, and θDecoder are the GRU parameters. The last hidden state of the encoder hT is used as the decoder’s initial state. We also adopt a dot-product attention similar to Dong and Lapata (2016). The tokens of the program a1, a2...an are generated one by one using a softmax over the vocabulary of valid tokens at each step, as provided by the “computer” (Section 2.1).
To achieve compositionality, the decoder must learn to represent and refer to intermediate variables whose value was saved in the “computer” after execution. Therefore, we augment the model with a key-variable memory, where each entry has two components: a continuous embedding key vi, and a corresponding variable token Ri referencing the value in the “computer” (see Figure 2).
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Key Variable
v1 R1(m.USA) Execute ( Argmax R2 Population )
Execute Return
m.NYCKey Variable
... ...
v3 R3(m.NYC)
Key Variable
v1 R1(m.USA)
v2 R2(list of US cities)
Execute ( Hop R1 !CityIn )
Hop R1 !CityIn( )
Largest city ( Hop R1in US GO !CityIn Argmax R2( )Population)
R2 Population ReturnArgmax )(
Entity Resolver
Figure 2: Semantic Parsing with NSM. The key embeddings of the key-variable memory are the output of the sequence model at certain encoding or decoding steps. For illustration purposes, we also show the values of the variables in parentheses, but the sequence model never sees these values, and only references them with the name of the variable (“R1”). A special token “GO” indicates the start of decoding, and “RETURN” indicates the end of decoding.
During encoding, we use an entity linker to link text spans (e.g., “US”) to KB entities. For each linked entity we add a memory entry where the key is the average of GRU hidden states over the entity span, and the variable token (R1) is the name of a variable in the computer holding the linked entity (m.USA) as its value. During decoding, when a full expression is generated (i.e., the decoder reads ”)”), it gets executed, and the result is stored as the value of a new variable in the “computer”. This variable is keyed by the GRU hidden state at that step. Every time a new variable is added to memory, the variable token is added to the output vocabulary of the decoder. The final answer returned by the “programmer” is the value of the last computed variable.
Note that, similar to pointer networks (Vinyals et al., 2015), the key embeddings for variables are dynamically generated for each example. During training, the model learns to represent variables by backpropagating gradients from a time step where a variable is selected by the decoder, through the key-variable memory, to an earlier time step when the key embedding was computed. Thus, the encoder/decoder learns to generate representations for variables such that they can be used at the right time to construct the correct program.
While the key embeddings are differentiable, the values referenced by the variables (lists of entities), stored in the “computer”, are symbolic and non-differentiable. This distinguishes the keyvariable memory from other memory-augmented neural networks that use continuous differentiable embeddings as the value of each memory entry (Weston et al., 2014; Graves et al., 2016a).
2.3 Training NSM with Weak Supervision
NSM executes non-differentiable operations against a KB, and thus end-to-end backpropa-
gation is not possible. Therefore, we base our training procedure on REINFORCE (Williams, 1992; Norouzi et al., 2016).
When the reward signal is sparse and the search space is large, it is common to utilize some full supervision to pre-train REINFORCE (Silver et al., 2016). Since we assume only weak supervision, we suggest an iterative ML procedure for finding pseudo-gold programs that will bootstrap REINFORCE.
REINFORCE We can formulate training as a reinforcement learning problem: given a question x, the state, action and reward at each time step t ∈ {0, 1, ..., T} are (st, at, rt). Since the environment is deterministic, the state is defined by the question x and the action sequence: st = (x, a0:t−1), where a0:t−1 = (a0, ..., at−1) is the history of actions at time t. A valid action at time t is at ∈ A(st), where A(st) is the set of valid tokens given by the “computer”. Since each action corresponds to a token, the full history a0:T corresponds to a program. The reward rt = I[t = T ] · F1(x, a0:T ) is non-zero at the last step of decoding, and is the F1 score computed comparing the gold answer and the answer generated by executing the program a0:T . Thus, the reward of a program or action sequence is
R(x, a0:T ) = ∑ t rt = F1(x, a0:T ).
The agent’s decision making is characterized by a policy, πθ(s, a) = Pθ(at = a|x, a0:t−1), where θ are the model parameters. Since the environment is deterministic, the probability of an action sequence a0:T is
Pθ(a0:T |x) = ∏ t Pθ(at | x, a0:t−1).
We can define our objective to be the expected reward and use policy gradient methods such as
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
REINFORCE for training. The objective and gradient are:
JRL(θ) = ∑ x EPθ(a0:T |x)[R(x, a0:T )],
∇θJRL(θ) = ∑ x ∑ a0:T Pθ(a0:T | x) · [R(x, a0:T )−
B(x)] · ∇θ logPθ(a0:T | x), where B(x) = ∑
a0:T Pθ(a0:T | x)R(x, a0:T ) is a
baseline that reduces the variance of the gradient estimation without introducing bias.
While REINFORCE assumes a stochastic policy, we rely on beam search for gradient estimation. Thus, in contrast with common practice of approximating the gradient by sampling from the model, we use the top-k action sequences (programs) in the beam with normalized probabilities. This allows training to focus on sequences with high probability, which are on the decision boundaries, and reduces the variance of the gradient.
Empirically (and in line with prior work), REINFORCE converged slowly and often got stuck in local optima (see Section 3). The difficulty of training resulted from the sparse reward signal in the large search space, which caused model probabilities for programs with non-zero reward to be very small at the beginning. If the beam size k is small, good programs fall off the beam, leading to zero gradients for all programs in the beam. If the beam size k is large, training is very slow, and the normalized probabilities of good programs when the model is untrained are still very small, leading to (1) near zero baselines, thus near zero gradients on ”bad” programs (2) near zero gradients on good programs due to the low probability Pθ(a0:T | x). To combat this, we present an alternative training strategy based on maximum-likelihood.
Iterative ML If we had gold programs, we could directly optimize their likelihood. Since we do not have gold programs, we can perform an iterative procedure (similar to hard EM), where we search for good programs given fixed parameters, and then optimize the probability of the best program found so far. We do decoding on an example with a large beam size and declare abest0:T (x) to be the pseudo-gold program that achieved highest reward with shortest length among the programs decoded on x in all previous iterations. Then, we can optimize the ML objective:
JML(θ) = ∑ x logPθ(a best 0:T (x) | x) (1)
A question x is not included if we did not find any program with positive reward.
Training with iterative ML is fast because there is at most one program per example and the gradient is not weighted by model probability. while decoding with a large beam size is slow, we could train for multiple epochs after each decoding. This iterative process has a bootstrapping effect that a better model leads to a better program abest0:T (x) through decoding, and a better program abest0:T (x) leads to a better model through training.
Even with a large beam size, some programs are hard to find because of the large search space. A common solution to this problem is to use curriculum learning (Zaremba and Sutskever, 2015; Reed and de Freitas, 2016). The size of the search space is controlled by both the set of functions used in the program and the program length. We apply curriculum learning by gradually increasing both these quantities (see details in Section 3) when performing iterative ML.
Nevertheless, iterative ML uses only pseudogold programs and does not optimize the objective we truly care about. This has two adverse effects: (1) The best program abest0:T (x) could be a spurious program that accidentally produces the correct answer (e.g., using the property PLACEOFBIRTH instead of PLACEOFDEATH when the two places are the same), and thus does not generalize to other questions. (2) Because training does not observe full negative programs, the model often fails to distinguish between tokens that are related to one another. For example, differentiating PARENTSOF vs. SIBLINGSOF vs. CHILDRENOF can be challenging. We now present learning where we combine iterative ML with REINFORCE.
Augmented REINFORCE To bootstrap REINFORCE, we can use iterative ML to find pseudogold programs, and then add these programs to the beam with a reasonably large probability. This is similar to methods from imitation learning (Ross et al., 2011; Jiang et al., 2012) that define a proposal distribution by linearly interpolating the model distribution and an oracle.
Algorithm 1 describes our overall training procedure. We first run iterative ML for NML iterations and record the best program found for every example xi. Then, we run REINFORCE, where we normalize the probabilities of the programs in beam to sum to (1 − α) and add α to the probability of the best found program C∗(xi).
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Algorithm 1 IML-REINFORCE Input: question-answer pairs D = {(xi, yi)}, mix ratio α, reward function R(·), training iterations NML, NRL, and beam sizes BML, BRL. Procedure: Initialize C∗x = φ the best program so far for x Initialize model θ randomly . Iterative ML for n = 1 to NML do
for (x, y) in D do C← Decode BML programs given x for j in 1...|C| do if Rx,y(Cj) > Rx,y(C∗x) then C∗x ← Cj θ ←ML training with DML = {(x,C∗x)}
Initialize model θ randomly . REINFORCE for n = 1 to NRL do
DRL ← ∅ is the RL training set for (x, y) in D do
C← Decode BRL programs from x for j in 1...|C| do
if Rx,y(Cj) > Rx,y(C∗x) then C∗x ← Cj C← C ∪ {C∗x} for j in 1...|C| do
p̂j ← (1−α)· pj∑ j′ pj′
where pj = Pθ(Cj | x) if Cj = C∗x then p̂j ← p̂j + α DRL ← DRL ∪ {(x,Cj , p̂j)}
θ ← REINFORCE training with DRL
Consequently, the model always puts a reasonable amount of probability on a program with high reward during training.
On top of imitation learning, our approach is related to the common practice in reinforcement learning (Schaul et al., 2016) to replay rare successful experiences to reduce the training variance and improve training efficiency. This is also similar to recent developments (Wu et al., 2016) in machine translation, where ML and RL objectives are linearly combined, because anchoring the model to some high-reward outputs stabilizes training.
3 Experiments and Analysis
We now empirically show that NSM can learn a semantic parser from weak supervision over a large KB. We evaluate on WEBQUESTIONSSP, a challenging semantic parsing dataset with strong baselines. Experiments show that NSM achieves new state-of-the-art performance on WEBQUESTIONSSP with weak supervision, and significantly closing the gap between weak and full supervisions for this task. Hyper-parameter settings are in the supplemental material due to page limits.
3.1 The WEBQUESTIONSSP dataset
Collecting semantic parse labels is a difficult, time consuming task, but can increase the accuracy of state-of-the-art question-answering systems by
full supervision. The WEBQUESTIONSSP dataset (Yih et al., 2016) contains semantic parses for the questions from WEBQUESTIONS (Berant et al., 2013) that are answerable using Freebase. It consists of 3,098 question-answer pairs for training and 1,639 for testing. These questions were collected using Google Suggest API and the answers were originally obtained using Amazon Mechanical Turk and updated by annotators who were familiar with the design of Freebase. We further separated out 620 questions from the training set as a validation set. For query pre-processing we used an in-house named entity linking system to find the entities in a question. The quality of the entity linker is similar to that of (Yih et al., 2015) at 94% of the gold root entities being included. Similar to Dong and Lapata (2016), we replaced named entity tokens with a special token ”ENT”. For example, the question ”who plays meg in family guy” is changed to ”who plays ENT in ENT ENT”.
Following (Yih et al., 2015) we used the last publicly available snapshot of Freebase (Bollacker et al., 2008). Since NSM training requires random access to Freebase during decoding, we preprocessed Freebase by removing predicates that are not related to world knowledge (starting with ”/common/”, ”/type/”, ”/freebase/”),2 and removing all text valued predicates, which are rarely the answer. Out of all 23K relations, 434 relations are removed during preprocessing. This results in a graph that fits in memory with 23K relations, 82M nodes, and 417M edges.
3.2 Model And Training Details
For pre-trained word embeddings, we used the 300 dimension GloVe word embeddings trained on 840B tokens (Pennington et al., 2014). On the encoder side, we added a projection matrix to transform the embeddings into 50 dimensions. On the decoder side, we used the same GloVe embeddings to construct an embedding for each property using its Freebase id, and also added a projection matrix to transform this embedding to 50 dimension. A Freebase id contains three parts: domain, type, and property. For example, the Freebase id for PARENTSOF is “/people/person/parents”. “people” is the domain, “person” is the type and ”parents” is the property. The embedding is constructed by concatenating the average of word embeddings in the domain and type name
2We kept “/common/topic/notable types”.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
to the average of word embeddings in the property name. For example, if the embedding dimension is 300, the embedding dimension for “/people/person/parents” will be 600. The first 300 dimensions will be the average of the embeddings for “people” and “person”, and the second 300 dimensions will be the embedding for “parents”.
Inspired by the staged generation process in Yih et al. (2015), curriculum learning includes two steps. We first run iterative ML for 10 iterations with programs constrained to only use the “Hop” function and the maximum number of expressions is 2. Then, we run iterative ML again, but use both “Hop” and “Filter”, and the maximum number of expressions is 3. However, the relations used by “Hop” are restricted to those that appeared in abest0:T (q) in the first step.
3.3 Results and discussion
We evaluate performance using the offical evaluation script for WEBQUESTIONSSP. Because the answer to a question may contain multiple entities or values, precision, recall and F1 are computed based on the output of each individual question, and average F1 is reported as the main evaluation metric. Accuracy measures the proportion of questions that are answered exactly.
A comparison to STAGG, the prior state-of-theart model (Yih et al., 2016, 2015) is shown in Table 2. Our model beats STAGG with weak supervision by a significant margin on all metrics, while relying on no feature engineering or handcrafted rules. When STAGG is trained with strong supervision it obtains an F1 of 71.7, and thus NSM closes half the gap between training with weak and strong supervision.
Model Prec. Rec. F1 Acc. STAGG 67.3 73.1 66.8 58.8 NSM 70.8 76.0 69.0 59.5
Table 2: Results on the test set. Average F1 is the main evaluation metric and NSM outperforms STAGG with no domainspecific knowledge or feature engineering.
Four key ingredients lead to the final performance of NSM. The first one is the neural computer interface that provides code assistance by checking for syntax and semantic errors. We find that semantic checks are very effective for opendomain KBs with a large number of properties. For our task, the average number of choices is reduced from 23K per step (all properties) to less
than 100 (the average number of relations connected to an entity).
The second ingredient is augmented REINFORCE training. Table 3 compares augmented REINFORCE, REINFORCE, and iterative ML on the validation set. REINFORCE gets stuck in local maxima and performs poorly, while iterative ML training is not directly optimizing the F1 measure, and achieves sub-optimal results. In contrast, augmented REINFORCE is able to bootstrap using pseudo-gold programs found by iterative ML and achieves the best performance on both the training and validation set.
Settings Train F1 Valid F1 iterative ML 68.6 60.1 REINFORCE 55.1 47.8 Augmented REINFORCE 83.0 67.2
Table 3: Average F1 on the validation set for augmented REINFORCE, REINFORCE, and iterative ML.
The third ingredient is curriculum learning during iterative ML. We compare the performance of the best programs found with and without curriculum learning in Table 4. We find that the best programs found with curriculum learning are substantially better than those found without curriculum learning by a large margin on every metric.
Settings Prec. Rec. F1 Acc. No curriculum 79.1 91.1 78.5 67.2 Curriculum 88.6 96.1 89.5 79.8
Table 4: Evaluation of the program with the highest F1 score in the beam (abest0:t ) with and without curriculum learning.
The last important ingredient is reducing overfitting. Given the small size of the dataset, overfitting is a major problem for training neural network models. We show the contributions of different techniques for controlling overfitting in Table 5. Note that after all the techniques have been applied, the model is still overfitting with training F1@1=83.0% and validation F1@1=67.2%.
Error analysis Two main sources of errors arise in manual error analysis
1. Search failure: the correct program is not found during search for pseudo-gold programs, either because the beam size is not large enough, or because the set of functions implemented by the interpreter is insufficient.
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Settings ∆ F1@1 -Pretrained word embeddings -5.5 -Pretrained property embeddings -2.7 -Dropout on GRU input and output -2.4 -Dropout on softmax -1.1 -Anonymize entity tokens -2.0
Table 5: Contributions of different overfitting techniques on the validation set.
The 89.5% F1 score in Table 4 indicates that at least 10% of the queries are of this kind. 2. Ranking failure: Pseudo-gold programs with high reward are found, but are not ranked at the top. Because training error is low, this is largely due to over-fitting. The 67.2% F1 score in Table 3 indicates that about 20% of examples are of this kind.
4 Related work
Among deep learning models for program induction, Reinforcement Learning Neural Turing Machines (RL-NTMs) (Zaremba and Sutskever, 2015) are the most similar to NSM, as a nondifferentiable machine is controlled by a sequence model. Therefore, both models rely on REINFORCE for training. The main difference between the two is the abstraction level of the programming language. RL-NTM uses lower level operations such as memory address manipulation and byte reading/writing, while NSM uses a high level programming language over a large knowledge base that includes operations such as following properties from entities, or sorting based on a property, which is more suitable for representing semantics. Earlier work such as OOPS (Schmidhuber, 2004) has desirable characteristics, such as the ability to define new functions and modify code storage. These remain to be future improvements for NSM.
We formulate NSM training as an instance of reinforcement learning (Sutton and Barto, 1998) in order to directly optimize the task reward of the structured prediction problem (Norouzi et al., 2016). Compared to imitation learning methods (Daume et al., 2009; Ross et al., 2011) that interpolate a model distribution with an oracle, NSM needs to solve a challenging search problem of training from weak supervisions in a large program space. Our solution employs two techniques (a) a symbolic “computer” helps find good programs by pruning the search space (b) an it-
erative maximum likelihood (ML) process, where beam search is used to find pseudo-gold programs. Wiseman and Rush (Wiseman and Rush, 2016) proposed a max-margin approach to train a sequence-to-sequence scorer. However, their training procedure is more involved, and we did not implement it in this work.
NSM is similar to Neural Programmer (Neelakantan et al., 2015) and Dynamic Neural Module Network (Andreas et al., 2016) in that they all solve the problem of semantic parsing from structured data, and generate programs using similar semantics. The main difference between these approaches is how an intermediate result (the memory) is represented. Neural Programmer and Dynamic-NMN chose to represent results as vectors of weights (row selectors and attention vectors), which enables backpropagation and search through all possible programs in parallel. However, their strategy is not applicable to a large KB such as Freebase, which contains about 100M entities, and more than 20k properties. Instead, NSM chooses a more scalable approach, where the “computer” saves intermediate results, and the neural net only refers to them with variable names (e.g., “R1” for all cities in the US).
5 Conclusion
We propose the Manager-Programmer-Computer framework for neural program induction. It integrates neural networks with a symbolic nondifferentiable computer to support abstract, scalable and precise operations through a friendly neural computer interface. Within this framework, we introduce the Neural Symbolic Machine, which integrates a sequence-to-sequence neural “programmer” with key-variable memory, and a Lisp interpreter with code assistance. Because the interpreter is non-differentiable and to directly optimize the task reward, we apply reinforcement learning and use pseudo-gold programs found by an iterative ML training process to bootstrap training. NSM achieves new state-of-the-art results on a challenging semantic parsing dataset with weak supervision, and significantly closing the gap between weak and full supervision. It is trained endto-end, and does not require any feature engineering or domain-specific knowledge.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
A Supplemental Material
A.1 More Model Details The dimension of encoder hidden state, decoder hidden state and key embeddings are all 50. The embeddings for the functions and special tokens (e.g., ”UNK”, ”GO”) are randomly initialized by a truncated normal distribution with mean=0.0 and stddev=0.1. All the weight matrices are initialized with a uniform distribution in [− √ 3 d , √ 3 d ] where d is the input dimension. Dropout rate is set to 0.5, and we see a clear tendency that larger dropout produces better performance, indicating overfitting is a major problem for learning.
A.2 More Training Details In iterative maximum likelihood training, the decoding uses beam size k = 100 to update the approximate gold programs and the model is trained for 20 epochs after each decoding. We use Adam optimizer (Kingma and Ba, 2014) with initial learning rate 0.001 for optimization. In our experiment, this process usually converges after a few (5-8) iterations.
For REINFORCE training, the best hyperparameters are chosen using the validation set. We use beam size k = 5 for decoding, and α is set to 0.1. Because the dataset is small and some relations are only used once in the whole training set, we train the model on the entire training set for 200 iterations with the best hyperparameters. Then we train the model with learning rate decay until convergence. Learning rate is decayed as gt = g0×β max(0,t−ts) m , where g0 = 0.001, β = 0.5 m = 1000, and ts is the number of training steps at the end of iteration 200.
Since decoding needs to query the knowledge graph constantly, the speed bottleneck for training is decoding. We address this problem in our implementation by partitioning the dataset, and using multiple decoders in parallel to handle each partition. We use 100 decoders, which queries 50
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
KG servers, and one trainer. The neural network model is implemented in TensorFlow. Since the model is small, we didn’t see a significant speedup by using GPU, so all the decoders and the trainer are using CPU only.
A.3 Extra Figures
Programmer ComputerManager
program
code assist
question & reward
answer built-in functions
Weak supervision
Neural knowledge base (built-in data)
non-differentiable
Symbolic
Figure 3: Manager-Programmer-Computer framework
Encoder a0 a1 an
SoftMax
q0 q1 qm
dot product
Linear Projection 300*50=15k
Linear Projection 600*50=30k
GRU 2*50*3*50=15k
Attention
Decoder
GRU 2*50*3*50=15k
Linear Projection 100*50=5k
Figure 4: Seq2Seq model architecture with dot-product attention and Dropout at GRU input, output, and softmax layers.
KG server 1
KG server m
…...
Decoder 1
Decoder 2
Decoder n
…...
Trainer
QA pairs 1
QA pairs 2
QA pairs n
Solutions 1
Solutions 2
Solutions n
…...…...
Model checkpoint
Figure 5: System Architecture. 100 decoders, 50 KG servers and 1 trainer.

This paper introduces Neural Symbolic Machines (NSMs) --- a deep neural model equipped with discrete memory to facilitate symbolic execution. An NSM includes three components: (1) a manager that provides weak supervision for learning, (2) a differentiable programmer based on neural sequence to sequence model, which encodes input instructions and predicts simplified Lisp programs using partial execution results stored in external discrete memories. (3) a symbolic computer that executes programs and provide code assistance to the programmer to prune search space. The authors conduct experiments on a semantic parsing task (WebQuestionsSP), and show that (1) NSM is able to model language compositionality by saving and reusing intermediate execution results, (2) Augmented REINFORCE is superior than vanilla REINFROCE for sequence prediction problems, and (3) NSM trained end-to-end with weak supervision is able to outperform existing sate-of-the-art method (STAGG).  - Strengths  * The idea of using discrete, symbolic memories for neural execution models is novel.                    Although in implementation it may simply reduce to copying previously executed variable tokens from an extra buffer, this approach is still impressive since it works well for a large-scale semantic parsing task.  * The proposed revised REINFORCE training schema using imperfect hypotheses derived from maximum likelihood training is interesting and effective, and could inspire future exploration in mixing ML RL training for neural sequence-to-sequence models.  * The scale of experiments is larger than any previous works in modeling neural execution and program induction. The results are impressive.  * The paper is generally clear and well-written, although there are some points which might require further clarification (e.g., how do the keys ($v_i$'s in Fig. 2) of variable tokens involved in computing action probabilities? Conflicting notations: $v$ is used to refer to variables in Tab. 1 and memory keys in Fig 1.).  Overall, I like this paper and would like to see it in the conference.  * Weaknesses  * [Choice of Dataset] The authors use WebQuestionsSP as the testbed. Why not using the most popular WebQuestions (Berant et al., 2013) benchmark set? Since NSM only requires weak supervision, using WebQuestions would be more intuitive and straightforward, plus it could facilitate direct comparison with main-stream QA research.  * [Analysis of Compositionality] One of the contribution of this work is the usage of symbolic intermediate execution results to facilitate modeling language compositionality. One interesting question is how well questions with various compositional depth are handled. Simple one-hop questions are the easiest to solve, while complex multi-hop ones that require filtering and superlative operations (argmax min) would be highly non-trivial. The authors should present detailed analysis regarding the performance on question sets with different compositional depth.  * [Missing References] I find some relevant papers in this field missing. For example, the authors should cite previous RL-based methods for knowledge-based semantic parsing (e.g., Berant and Liang., 2015), the sequence level REINFORCE training method of (Ranzato et al., 2016) which is closely related to augmented REINFORCE, and the neural enquirer work (Yin et al., 2016) which uses continuous differentiable memories for modeling neural execution.  * Misc.  * Why is the REINFORCE algorithm randomly initialized (Algo. 1) instead of using parameters pre-trained with iterative ML?  * What is KG server in Figure 5?