Using Ontology-Grounded Token Embeddings To Predict Prepositional Phrase Attachments
Type-level word embeddings use the same set of parameters to represent all instances of a word regardless of its context, ignoring the inherent lexical ambiguity in language. Instead, we embed semantic concepts (or synsets) as defined in WordNet and represent a word token in a particular context by estimating a distribution over relevant semantic concepts. We use the new, context-sensitive embeddings in a model for predicting prepositional phrase (PP) attachments and jointly learn the concept embeddings and model parameters. We show that using context-sensitive embeddings improves the accuracy of the PP attachment model by 5.4% absolute points, which amounts to a 34.4% relative reduction in errors.
1 Introduction
Type-level word embeddings map a word type (i.e., a surface form) to a dense vector of real numbers such that similar word types have similar embeddings. When pre-trained on a large corpus of unlabeled text, they provide an effective mechanism for generalizing statistical models to words which do not appear in the labeled training data for a downstream task.
In this paper, we make the following distinction between types and tokens: By word types, we mean the surface form of the word, whereas by tokens we mean the instantiation of the surface form in a context. For example, the same word type ‘pool’ occurs as two different tokens in the sentences “He sat by the pool,” and “He played a game of pool.”
Most word embedding models define a single vector for each word type. However, a fundamen-
tal flaw in this design is their inability to distinguish between different meanings and abstractions of the same word. In the two sentences shown above, the word ‘pool’ has different meanings, but the same representation is typically used for both of them. Similarly, the fact that ‘pool’ and ‘lake’ are both kinds of water bodies is not explicitly incorporated in most type-level embeddings. Furthermore, it has become a standard practice to tune pre-trained word embeddings as model parameters during training for an NLP task (e.g., Chen and Manning, 2014; Lample et al., 2016), potentially allowing the parameters of a frequent word in the labeled training data to drift away from related but rare words in the embedding space.
Previous work partially addresses these problems by estimating concept embeddings in WordNet (e.g., Rothe and Schütze, 2015), or improving word representations using information from knowledge graphs (e.g., Faruqui et al., 2015). However, it is still not clear how to use a lexical ontology to derive context-sensitive token embeddings.
In this paper, we represent a word token in a given context by estimating a context-sensitive probability distribution over relevant concepts in WordNet (Miller, 1995) and use the expected value (i.e., weighted sum) of the concept embeddings as the token representation (see §2). In addition to providing context-sensitive token embeddings, the proposed method implicitly regularizes the embeddings of related words by forcing related words to share similar concept embeddings. As a result, the representation of a rare word which does not appear in the training data for a downstream task benefits from all the updates to related words which share one or more concept embeddings.
Our approach to context-sensitive embeddings assumes the availability of a lexical ontology such
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
Figure 1: An example grounding for the word ‘pool’. Solid arrows represent possible senses and dashed arrows represent hypernym relations. Note that the same set of concepts are used to ground the word ‘pool’ regardless of its context. Other WordNet senses for ‘pool’ were removed from the figure for simplicity.
as WordNet, but it does not require a tagger for word senses. We use the proposed embeddings to predict prepositional phrase (PP) attachments (see §3), a challenging problem which emphasizes the selectional preferences between words in the PP and each of the candidate head words. Our empirical results and detailed analysis (see §4) show that the proposed embeddings effectively use WordNet to improve the accuracy of PP attachment predictions.
2 WordNet-Grounded Context-Sensitive Token Embeddings
In this section, we focus on defining our contextsensitive token embeddings. We first describe our grounding of word types using WordNet concepts. Then, we describe our model of contextsensitive token-level embeddings as a weighted sum of WordNet concept embeddings.
2.1 WordNet Grounding
We use WordNet to map each word type to a set of synsets, including possible generalizations or abstractions. Among the labeled relations defined in WordNet between different synsets, we focus on the hypernymy relation to help model generaliza-
tion and selectional preferences between words, which is especially important for predicting PP attachments (Resnik, 1993). To ground a word type, we identify the the set of (direct and indirect) hypernyms of the WordNet senses of that word. A simplified grounding of the word ‘pool’ is illustrated in Figure 1. This grounding is key to our model of token embeddings, to be described in the following subsections.
2.2 Context-Sensitive Token Embeddings
Our goal is to define a context-sensitive model of token embeddings which can be used as a dropin replacement for traditional type-level word embeddings.
Notation. Let Senses(w) be the list of synsets defined as possible word senses of a given word type w in WordNet, and Hypernyms(s) be the list of hypernyms for a synset s.1 For example, according to Figure 1:
Senses(pool) = [pond.n.01, pool.n.09], and
Hypernyms(pond.n.01) = [pond.n.01, lake.n.01,
body of water.n.01, entity.n.01] (1)
Each WordNet synset s is associated with a set of parameters vs ∈ Rn which represent its embedding. This parameterization is similar to that of Rothe and Schütze (2015).
Embedding model. Given a sequence of tokens t and their corresponding word types w, let ui ∈ Rn be the embedding of the word token ti at position i. Unlike most embedding models, the token embeddings ui are not parameters. Rather, ui is computed as the expected value of concept embeddings used to ground the word type wi corresponding to the token ti: ui = ∑
s∈Senses(wi) ∑ s′∈Hypernyms(s) p(s, s′ | t,w, i) vs′
(2)
such that∑ s∈Senses(wi) ∑ s′∈Hypernyms(s) p(s′, s | t,w, i) = 1
(3)
1For notational convenience, we assume that s ∈ Hypernyms(s).
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Figure 2: Steps for computing the contextsensitive token embedding for the word ‘pool’, as described in §2.2.
The distribution which governs the expectation over synset embeddings factorizes into two components:
p(s, s′ | t,w, i) ∝λwi exp−λwi rank(s,wi)× MLP([vs′ ; context(i, t)]) (4)
The first component, λwi exp −λwi rank(s,wi), is a sense prior which reflects the prominence of each word sense for a given word type. Note that this is defined for each word type (wi), and is shared across all tokens which have the same word type. The parameterization of the sense prior is similar to an exponential distribution since WordNet senses are organized in descending order of their frequency. The scalar parameter (λwi) controls the decay of the probability mass.
The second component, MLP([vs′ ; context(i, t)]) is what makes the token representations context-sensitive. It scores each concept in the WordNet grounding of wi by feeding the concatenation of the concept embedding and a dense vector that summarizes the textual context into a multilayer perceptron (MLP) with two tanh layers followed by a softmax layer. This component is inspired by the soft attention often used in neural machine translation (Bahdanau et al., 2014).2 The definition of the
2Although soft attention mechanism is typically used to
context function is dependent on the encoder used to encode the context. We describe a specific instantiation of this function in §3.
To summarize, Figure 2 illustrates how to compute the embedding of a word token ti = ‘pool’ in a given context:
1. compute a summary of the context context(i, t),
2. enumerate related concepts for ti,
3. compute p(s, s′ | t,w, i) for each pair (s, s′), and
4. compute ui = E[vs′ ].
In the following section, we describe our model for predicting PP attachments, including our definition for context.
3 PP Attachment
Disambiguating PP attachments is an important and challenging NLP problem, and is a good fit for evaluating our WordNet-grounded contextsensitive embeddings since modeling hypernymy and selectional preferences is critical for successful prediction of PP attachments (Resnik, 1993).
Figure 3, reproduced from Belinkov et al. (2014), illustrates an example of the PP attachment prediction problem. The accuracy of a
explicitly represent the importance of each item in a sequence, it can also be applied to non-sequential items.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
competitive English dependency parser at predicting the head word of an ambiguous prepositional phrase is 88.5%, significantly lower than the overall unlabeled attachment accuracy of the same parser (94.2%).3
This section formally defines the problem of PP attachment disambiguation, describes our baseline model, then shows how to integrate the token-level embeddings in the model.
3.1 Problem definition
We follow Belinkov et al. (2014)’s definition of the PP attachment problem. Given a preposition p and its direct dependent d in the prepositional phrase (PP), our goal is to predict the correct head word for the PP among an ordered list of candidate head words h. Each example in the train, validation, and test sets consists of an input tuple 〈h, p, d〉 and an output index k to identify the correct head among the candidates in h.
3.2 Model definition
Both our proposed and baseline models for PP attachment use bidirectional RNN with LSTM cells (bi-LSTM) to encode the sequence t = 〈h1, . . . , hK , p, d〉.
We score each candidate head by feeding the concatenation of the output bi-LSTM vectors for the head hk, the preposition p and the direct dependent d through an MLP, with a fully connected tanh layer to obtain a non-linear projection of the concatenation, followed by a fully-connected softmax layer:
p(hkis head) = MLPattach([lstm out(hk);
lstm out(p);
lstm out(d)]) (5)
To train the model, we use cross-entropy loss at the output layer for each candidate head in the training set. At test time, we predict the candidate head with the highest probability according to the model in Eq. 5, i.e.,
k̂ = argmax k p(hkis head = 1). (6)
This model is inspired by the Head-Prep-ChildTernary model of Belinkov et al. (2014). The main difference is that we replace the input features for each token with the output bi-RNN vectors.
3See Table 2 in §4 for detailed results.
We now describe the difference between the proposed model and the baseline. Generally, let lstm in(ti) and lstm out(ti) represent the input and output vectors of the bi-LSTM for each token ti ∈ t in the sequence.
Baseline model. In the baseline model, we use type-level word embeddings to represent the input vector lstm in(ti) for a token ti in the sequence. The word embedding parameters are initialized with pre-trained vectors, then tuned along with the parameters of the bi-LSTM and MLPattach. We call this model LSTM-PP.
Proposed model. In the proposed model, we use token level word embedding as described in §2 as the input to the bi-LSTM, i.e., lstm in(ti) = ui. The context used for the attention component is simply the hidden state from the previous timestep. However, since we use a bi-LSTM, the model essentially has two RNNs, and accordingly we have two context vectors, and associated attentions. That is, contextf (i, t) = lstm in(ti−1) for the forward RNN and contextb(i, t) = lstm in(ti+1) for the backward RNN. The synset embedding parameters are initialized with pretrained vectors and tuned along with the sense decay (λw) and MLP parameters from Eq. 4, the parameters of the bi-LSTM and those of MLPattach. We call this model OntoLSTM-PP.
4 Experiments
Dataset and evaluation. We used the English PP attachment dataset created and made available by Belinkov et al. (2014). The training and test splits contain 33,359 and 1951 labeled examples respectively. As explained in §3.1, the input for each example is 1) an ordered list of candidate head words, 2) the preposition, and 3) the direct dependent of the preposition. The head words are either nouns or verbs and the dependent is always a noun. All examples in this dataset have at least two candidate head words. As discussed in Belinkov et al. (2014), this dataset is a more realistic PP attachment task than the RRR dataset (Ratnaparkhi et al., 1994). The RRR dataset is a binary classification task with exactly two head word candidates in all examples. The context for each example in the RRR dataset is also limited which defeats the purpose of our context-sensitive embeddings.
Model specifications and hyperparameters. For efficient implementation, we use mini-batch
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
updates with the same number of senses and hypernyms for all examples, padding zeros and truncating senses and hypernyms as needed. For each word type, we use a maximum of S senses and H indirect hypernyms from WordNet. In our initial experiments on a held-out development set (10% of the training data), we found that values greater than S = 3 and H = 5 did not improve performance. We also used the development set to tune the number of layers in MLPattach separately for the OntoLSTM-PP and LSTM-PP, and the number of layers in the attention MLP in OntoLSTM-PP. When a synset has multiple hypernym paths, we use the shortest one. Finally, words types which do not appear in WordNet are assumed to have one unique sense per word type with no hypernyms. Since the POS tag for each word is included in the dataset, we exclude WordNet synsets which are incompatible with the POS tag. The synset embedding parameters are initilized using the synset vectors obtained by running AutoExtend (Rothe and Schütze, 2015) on 100-dimensional GloVe (Pennington et al., 2014) vectors for WordNet 3.1. Representation for the OOV word types in LSTMPP and OOV synset types in OntoLSTM-PP were randomly drawn from a uniform 100-d distribution. Initial sense prior parameters (λw) were also drawn from a uniform 1-d distribution.
Baselines. In our experiments, we compare our proposed model, OntoLSTM-PP with three baselines – LSTM-PP initialized with GloVe embedding, LSTM-PP initialized with GloVe vectors retrofitted to WordNet using the approach of Faruqui et al. (2015), and finally the best performing standalone PP attachment system from Belinkov et al. (2014), referred to as “HPCD (full)” in the paper. “HPCD (full)” is a neural network model that learns to compose the vector representations of each of the candidate heads with those of the preposition and the dependent, and predict attachments. The input representations are enriched using syntactic context information, POS, WordNet and VerbNet (Kipper et al., 2008) information and the distance of the head word from the PP is explicitly encoded in composition architecture. In contrast, we do not use syntactic context, VerbNet and distance information, and do not explicitly encode POS information.
4.1 PP Attachment Results
Table 1 shows that our proposed token level embedding scheme “OntoLSTM-PP” outperforms the better variant of our baseline “LSTM-PP” (with GloVe-retro intialization) by an absolute accuracy difference of 4.9%, or a relative error reduction of 32%. “OntoLSTM-PP’ also outperforms “HPCD (full)”, the previous best result on this dataset.
Initializing the word embeddings with GloVeretro (which uses WordNet as described in Faruqui et al. (2015)) instead of “textitGloVe” amounts to a small improvement, compared to the improvements obtained using “OntoLSTM-PP”. This result illustrates that our approach of dynamically choosing a context sensitive distribution over synsets is a more effective way of making use of WordNet.
Effect on dependency parsing. Following Belinkov et al. (2014), we used RBG parser (Lei et al., 2014), and modified it by adding a binary feature indicating the PP attachment predictions from our model.
We compare four ways to compute the additional binary features: 1) the predictions of the best standalone system “HPCD (full)” in Belinkov et al. (2014), 2) the predictions of our baseline model “LSTM-PP”, 3) the predictions of our improved model “OntoLSTM-PP”, and 4) the gold labels “Oracle PP”.
Table 2 shows the effect of using the PP attachment predictions as features within a dependency parser. We note there is a relatively small difference in unlabeled attachment accuracy for all dependencies (not only PP attachments), even when gold PP attachments are used as additional features to the parser. However, when gold PP attachment are used, we note a large potential improvement of 10.46 points (between the PPA accuracy for “RBG” and “RBG + Oracle PP”), which confirms that adding PP predictions as features is
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
System Full UAS PPA Acc. RBG 94.17 88.51 RBG + HPCD (full) 94.19 89.59 RBG + LSTM-PP 94.14 86.35 RBG + OntoLSTM-PP 94.30 90.11 RBG + Oracle PP 94.60 98.97
Table 2: Results from RBG dependency parser with features coming from various PP attachment predictors and oracle attachments.
an effective approach. Our proposed model “RBG + OntoLSTM-PP” recovers 15% of this potential improvement, while “RBG + HPCD (full)” recovers 10%, which illustrates that PP attachment remains a difficult problem with plenty of room for improvements even when using a dedicated model to predict PP attachments and using its predictions in a dependency parser.
We also note that, although we use the same predictions of the “HPCD (full)” model in Belinkov et al. (2014),4 we report different results than Belinkov et al. (2014). For example, the unlabeled attachment score (UAS) of the baselines “RBG” and “RBG + HPCD (full)” are 94.17 and 94.19, respectively, in Table 2, compared to 93.96 and 94.05, respectively, in Belinkov et al. (2014). This is due to the use of different versions of the RBG parser.5
4.2 Analysis
In this subsection, we analyze different aspects of our model in order to develop a better understanding of its behavior.
Effect of context sensitivity and sense priors. We now show some results that indicate the relative strengths of two components of our contextsensitive token embedding model. The second row in Table 3 shows the test accuracy of a system trained without sense priors (that is, making p(s|wi) from Eq. 2 a uniform distribution), and the third row shows the effect of making the token representations context-insensitive by giving a similar attention score to all related concepts, essentially making them type level representations,
4The authors kindly provided their predictions for 1942 test examples (out of 1951 examples in the full test set). In Table 2, we use the same subset of 1942 test examples and will include a link to the subset in the final draft.
5We use the latest commit (SHA: e07f74dd2ba47348fd548935155ded38eea20809) on the GitHub repository of the RGB parser.
but still grounded in WordNet. As it can be seen, removing context sensitivity has an adverse effect on the results. This illustrates the importance of the sense priors and the attention mechanism.
It is interesting that, even without sense priors and attention, the results with WordNet grounding is still higher than that of the two LSTM-PP systems in Table 1. This result illustrates the regularization behavior of sharing concept embeddings across multiple words, which is especially important for rare words.
Effect of training data size. Since “OntoLSTM-PP” uses external information, the gap between the model and “LSTM-PP” is expected to be more pronounced when the training data sizes are smaller. To test this hypothesis, we trained the two models with different amounts of training data and measured their accuracies on the test set. The plot is shown in Figure 4. As expected, the gap tends to be larger at lower data sizes. Surprisingly, even with 2000 sentences in the training data set, OntoLSTM-PP outperforms LSTM-PP trained with the full data set. When both the models are trained with the fill dataset, LSTM-PP reaches a training accuracy of 95.3%, whereas OntoLSTM-PP reaches 93.5%. The fact that LSTM-PP is overfitting the training data more, indicates the regularization capability of OntoLSTM-PP.
Qualitative analysis. To better understand the effect of WordNet grounding, we took a sample of sentences from the test set whose PP attachments were correctly predicted by OntoLSTM-PP but not by LSTM-PP. A common pattern observed was
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Figure 5: Two examples from the test set where OntoLSTM-PP predicts the head correctly and LSTM-PP does not, along with weights by OntoLSTM-PP to synsets that contribute to token representations of infrequent word types. The prepositions are shown in bold, LSTM-PP’s predictions in red and OntoLSTMPP’s predictions in green. Words that are not candidate heads or dependents are shown in brackets.
Model PPA Acc. full 89.7 - sense priors 88.4 - attention 87.5
Table 3: Effect of removing sense priors and context sensitivity (attention) from the model
that those sentences contained words not seen frequently in the training data. Figure 5 shows two such cases. In both cases, the weights assigned by OntoLSTM-PP to infrequent words are also shown. The word types soapsuds and buoyancy do not occur in the training data, but OntoLSTMPP was able to leverage the parameters learned for the synsets that contributed to their token representations. Another important observation is that the word type buoyancy has four senses in WordNet (we consider the first three), none of which is the metaphorical sense that is applicable to markets as shown in the example here. Selecting a combination of relevant hypernyms from various senses may have helped OntoLSTM-PP make the right prediction. This shows the value of using hypernymy information from WordNet. Moreover, this indicates the strength of the hybrid nature of the model, that lets it augment ontological information with distributional information.
Parameter space We note that the vocabulary sizes in OntoLSTM-PP and LSTM-PP are comparable as the synset types are shared across word types. In our experiments with the full PP attachment dataset, we learned embeddings for 18k synset types with OntoLSTM-PP and 11k word types with LSTM-PP. Since the biggest contribution to the parameter space comes from the embedding layer, the complexities of both the models are comparable.
Implementation and code availability. The models are implemented using Keras (Chollet, 2015), and the functionality is available in the form of Keras layers to make it easier for other researchers to use the proposed embeddingn model.
Future work. This approach may be extended to other NLP tasks that can benefit from using encoders that can access WordNet information. WordNet also has some drawbacks, and may not always have sufficient coverage given the task at hand. As we have shown in §4.2, our model can deal with missing WordNet information by augmenting it with distributional information. Moreover, the methods described in this paper can be extended to other kinds of structured knowledge sources like Freebase which may be more suitable for tasks like question answering.
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
5 Related Work
This work is related to various lines of research within the NLP community: dealing with synonymy and homonymy in word representations both in the context of distributed embeddings and more traditional vector spaces; hybrid models of distributional and knowledge based semantics; and selectional preferences and their relation with syntactic and semantic relations.
The need for going beyond a single vector per word-type has been well established for a while, and many efforts were focused on building multi-prototype vector space models of meaning (Reisinger and Mooney, 2010; Huang et al., 2012; Chen et al., 2014; Jauhar et al., 2015; Neelakantan et al., 2015, etc.). However, the target of all these approaches is obtaining multi-sense word vector spaces, either by incorporating sense tagged information or other kinds of external context. The number of vectors learned is still fixed, based on the preset number of senses. In contrast, our focus is on learning a context dependent distribution over those concept representations. Other work not necessarily related to multi-sense vectors, but still related to our work includes Belanger and Kakade (2015)’s work which proposed a Gaussian linear dynamical system for estimating token-level word embeddings, and Vilnis and McCallum (2014)’s work which proposed mapping each word type to a density instead of a point in a space to account for uncertainty in meaning. These approaches do not make use of lexical ontologies and is not amenable for joint training with a downstream NLP task.
Related to the idea of concept embeddings is Rothe and Schütze (2015) who estimated WordNet synset representations, given pre-trained typelevel word embeddings. In contrast, our work focuses on estimating token-level word embeddings as context sensitive distributions of concept embeddings.
There is a large body of work that tried to improve word embeddings using external resources. Yu and Dredze (2014) extended the CBOW model (Mikolov et al., 2013) by adding an extra term in the training objective for generating words conditioned on similar words according to a lexicon. Jauhar et al. (2015) extended the skipgram model (Mikolov et al., 2013) by representing word senses as latent variables in the generation process, and used a structured prior based on the ontology.
Faruqui et al. (2015) used belief propagation to update pre-trained word embeddings on a graph that encodes lexical relationships in the ontology. In contrast to previous work that was aimed at improving type level word representations, we propose an approach for obtaining context-sensitive embeddings at the token level, while jointly optimizing the model parameters for the NLP task of interest.
Resnik (1993) showed the applicability of semantic classes and selectional preferences to resolving syntactic ambiguity. Zapirain et al. (2013) applied models of selectional preferences automatically learned from WordNet and distributional information, to the problem of semantic role labeling. Resnik (1993); Brill and Resnik (1994); Agirre (2008) and others have used WordNet information towards improving prepositional phrase attachment predictions.
6 Conclusion
In this paper, we proposed a grounding of lexical items which acknowledges the semantic ambiguity of word types using WordNet and a method to learn a context-sensitive distribution over their representations. We also showed how to integrate the proposed representation with recurrent neural networks for disambiguating prepositional phrase attachments, showing that the proposed WordNetgrounded context-sensitive token embeddings outperforms standard type-level embeddings for predicting PP attachments. We provided a detailed qualitative and quantitative analysis of the proposed model.

- Overview:  The paper proposes a new model for training sense embeddings grounded in a lexical-semantic resource (in this case WordNet). There is no direct evaluation that the learned sense vectors are meaningful; instead, the sense vectors are combined back into word embeddings, which are evaluated in a downstream task: PP attachment prediction.  - Strengths:  PP attachment results seem solid.  - Weaknesses:  Whether the sense embeddings are meaningful remains uninvestigated.   The probabilistic model has some details that are hard to understand. Are the   lambda_w_i hyperparameters or trained? Where does  rank d come from, is this taken from the sense ranks in WordNet?  Related work: the idea of expressing embeddings of words as a convex combination of sense embeddings has been proposed a number of times previously. For instance, Johansson and Nieto Pi u00f1a  Embedding a semantic network in a word space d (NAACL, 2015) decomposed word embeddings into ontology-grounded sense embeddings based on this idea. Also in unsupervised sense vector training this idea has been used, for instance by Arora et al  Linear Algebraic Structure of Word Senses, with Applications to Polysemy d.  Minor comments:  no need to define types and tokens, this is standard terminology  why is the first   lamba_w_i in equation 4 needed if the probability is unnormalized?  - General Discussion:


Rare Entity Prediction: Language Understanding with External Knowledge using Hierarchical LSTMs
Reading comprehension in NLP refers to the ability of models to answer any question about a passage accurately. An important open problem is how to effectively use external knowledge to answer such questions. In this paper, we introduce a new task and derive new models to drive progress towards this goal. In particular, we propose the task of rare entity prediction: given a web document with several entities removed, models are tasked with predicting the correct missing entities conditioned on the document context and the lexical resources. This task is challenging due to the diversity of language styles and the extremely large number of rare entities. Our experiments show that models that make use of external knowledge in the form of lexical resources, particularly our model using hierarchical LSTMs, perform significantly better at rare entity prediction than those that do not.
1 Introduction
One of the preeminent problems in natural language processing is the design of models that can acquire knowledge of the world through language. Such knowledge can be obtained in at least two ways: through unstructured texts such as news articles or blogs, and through structured knowledge bases such as WordNet (Miller, 1995) and Freebase (Bollacker et al., 2008).
A natural setting for testing the abilities of models to acquire knowledge through natural language is reading comprehension, where models must answer questions about a given document. Many existing reading comprehension tasks, however, can be solved using basic language modelling, and re-
Context [...] , who lived from 1757 to 1827, was admired by a small group of intellectuals and artists in his day, but never gained general recognition as either a poet or painter. [...]
Candidate Entities Peter Ackroyd: Peter Ackroyd is an English biographer, novelist and critic with a particular interest in the history and culture of London. [...] William Blake: William Blake was an English poet, painter, and printmaker. [...] Emanuel Swedenborg: Emanuel Swedenborg was a Swedish scientist, philosopher, theologian, revelator, and mystic. [...]
quire little reasoning. This is the case for the Daily Mail/ CNN dataset (Hermann et al., 2015), for example, which has been shown to have a small gap between machine and human performance (Chen et al., 2016). Existing reading comprehension tasks target reasoning about generic concepts, while accounting for syntax, lexical semantics, and/or discourse. In this work, we aim to move towards reasoning about specific instances of entities in context. This is a very difficult problem, as we have very few training samples per instance; thus, we demonstrate that we cannot simply rely on language modelling, and must leverage external sources of knowledge.
Recent efforts have been made to integrate different sources of knowledge, for example com-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
bining distributional and relational semantics for building word embeddings (Fried and Duh, 2014; Long et al., 2016). These models show performance improvements on tasks such as knowledge base completion. However, such ideas are rarely examined for reading comprehension tasks.
In this paper, we propose a novel task called rare entity prediction, where models must predict missing entities (e.g. proper names or places) in web articles by leveraging the available lexical resources. Using lexical resources is necessary as there are a large number of rare entities in the dataset (see Table 2); thus, it is much more difficult to obtain information about these entities using the unstructured texts alone. For this task, we provide a significantly enhanced version of the Wikilinks dataset, with entity descriptions extracted from Freebase serving as the lexical resources, which we call the Wikilinks Entity Prediction dataset. An example from the Wikilinks Entity Prediction dataset is shown in Table 1.
In addition to the task and dataset, we introduce several models that can be used to solve this task, using distributional semantics and recurrent neural networks. We show that language modelling baseline models that do not consider the Freebase descriptions are unable to achieve good performance on the task. RNN models leveraging the external knowledge perform much better; in particular, our hierarchical double encoder model (HIERENC) achieves a 17% increase in accuracy over the language model baseline.
Rare entity prediction task is unique because it is difficult for models that heavily rely on word cooccurrence statistics to achieve good performance. To solve the task, models need to have the abilities to incorporate external a priori knowledge with an understanding of the unstructured natural language. Some questions can be challenging to answer even for humans without proper prior knowledge about the candidate entities. Table 1 provides a good example of such scenarios. We believe that being able to integrate multiple sources of knowledge is crucial for not only reading comprehension, but also other NLP systems.
2 Related Work
Related to our work is the task of entity prediction, also called link prediction or knowledge base completion, in the domain of multi-relational data. Multi-relational datasets like WordNet (Miller,
1995) and Freebase (Bollacker et al., 2008) consist of entity-relation triples of the form (head, relation, tail), indicating a relationship between the head and tail entity as described by the relation. In entity prediction, either the head or tail entity is removed, and the model has to predict the missing entity. While this task necessitates understanding structured information in the form of relationships between objects, it does not require the processing of natural language text, which is crucial in rare entity prediction.
Also related to our approach is the task of script learning or script induction, in which a system is asked to predict an event, or a (verb, dependency) pair, that is held out from a document (Chambers and Jurafsky, 2008, 2009; Jans et al., 2012). For example, the event ‘X orders Y ’ could predict the event ‘X eats Y ’. Approaches to this problem have used relational atoms similar to relation triples to represent events (Pichotta and Mooney, 2014), or have used recurrent neural networks (Pichotta and Mooney, 2016). Similarly, the task of discourse coherence modeling involves determining text relatedness at different levels of granularity (Grosz et al., 1995). While these tasks involve understanding unstructured natural language, and there are entity-based approaches for solving them (Barzilay and Lapata, 2008), they differ from our task as they do not explicitly require the use of structured knowledge sources such as Freebase. Events can be represented, for example, as relational atoms; for example, the event ‘X eats Y ’ could be represented as eat(X,Y ).
Rare entity prediction is clearly distinct from tasks such as entity tagging and recognition (Ritter et al., 2011), as models are provided with the actual name of the entity in question, and only have to match the entity with related concepts and tags. It is more closely related to the machine reading literature from e.g. Etzioni et al. (2006); however, the authors define machine reading as primarily unsupervised, whereas our task is supervised.
A similar supervised reading comprehension task was proposed by Hermann et al. (2015) using news articles from CNN and the Daily Mail. Given an article, models are tasked with filling in blanks of one-sentence summaries of the article. The original dataset was found to have a low ceiling for machine improvement (Chen et al., 2016); thus, alternative datasets have been proposed that consist of more difficult questions (Trischler et al.,
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
2016; Rajpurkar et al., 2016). A dataset with a similar task was also proposed by Hill et al. (2015a), where models must answer questions about short children’s stories. While these tasks require the understanding of unstructured natural language, it does not require integration with external knowledge sources.
Hill et al. (2015b) propose a method of combining distributional semantics with an external knowledge source in the form of dictionary definitions. The purpose of their model is to obtain more accurate word and phrase embeddings by combining lexical and phrasal semantics, and they achieve fairly good performance on a reverse dictionary and crossword QA tasks.
Perhaps the most related approach to our work is the one concurrently developed by Ahn et al. (2016). The authors propose a WikiFacts dataset where Wikipedia descriptions are aligned with Freebase facts. While they also aim to integrate external knowledge with unstructured natural language, their task differs from ours in that it is primarily a language modeling problem.
Finally, methods that address factoid question answering (QA) using Freebase (Bordes et al., 2014a; Yao and Van Durme, 2014; Serban et al., 2016) must also combine knowledge from a structured knowledge base with an understanding of the question that’s being asked. However, the nature of the unstructured data is different: in factoid QA, questions are often short and relate directly to the relations of a given entity, while rare entity prediction requires learning much longer-term dependencies in natural language, where relations between entities are not specifically referred to.
3 Rare Entity Prediction
3.1 The Wikilinks Dataset
The Wikilinks dataset (Singh et al., 2012) is a large dataset originally designed for cross-document coreference resolution, the task of grouping entity mentions from a set of documents into clusters that represent a single entity. The dataset consists of a list of non-Wikipedia web pages (discovered using the Google search index) that contain hyperlinks to Wikipedia, such as random blog posts or news articles. Every token with a hyperlink to Wikipedia is then marked and considered an entity mention in the dataset. Each entity mention is also linked back to a knowledge base through their corresponding Freebase IDs.
In order to ensure the hyperlinks refer to the correct Wikipedia pages, additional filtering is conducted to ensure that either (1) at least one token in the hyperlink (or anchor) matches a token in the title of the Wikipedia page, or (2) the anchor text matches exactly an anchor from the Wikipedia page text, which can be considered an alias of the page. As many near-duplicate copies of Wikipedia pages can be found online, any web pages where more than 70% of the sentences match those from their linked Wikipedia pages are discarded. To ensure the quality of data, the authors performed manual inspection on a small subset of the web pages. No incorrect hyperlinks were found.
3.2 The Wikilinks Rare Entity Prediction Dataset
We provide a significantly pre-processed and augmented version of the Wikilinks dataset for the purpose of entity prediction, which we called the Wikilinks Rare Entity Prediction dataset1. In particular, we parse the HTML texts of the web pages and extract their page contents to form our corpus. Entity mentions with hyperlinks to Wikipedia are marked and replaced by a special token (**blank**), serving as the placeholder for missing entities that we would like the models to predict. The correct missing entity ẽ is preserved as a target. Additionally, we extract the lexical definitions of all entities that are marked in the corpus from Freebase using their Freebase IDs, which are available for all entities in the Wikilinks dataset. These lexical definitions will serve as the external knowledge to our models.
Table 2 above shows some basic statistics of our dataset. As we can see, unlike the Children’s
1We will make this dataset publicly available upon acceptance of the paper.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Book dataset, which has 50k candidate entities for almost 700k context and query pairs (Hill et al., 2015a), the number of unique entities found in our dataset is in the same order of magnitude as the number of documents available.
Moreover, the majority of entities appears a relatively small number of times, with 92.8% observed less than or equal to 10 times across the entire corpus. This suggests that models that only rely on the surrounding context information may not be able to correctly predict the missing entities. This further motivates us to incorporate additional information into the decision process to improve the performance. In the experiments section, we show that the lexical resources are indeed necessary to achieve better results.
3.3 Task Definition2
Here, we formalize the task definition of the entity prediction problem. Given a document D in the corpus, we split it into an ordered list of contexts C = {C1, ...,Cn} where each context Ci (1 ≤ i ≤ n) is a single sentence (w1, ..., wm) where the special token **blank** is found.3 Let E be the set of candidate entities. For each missing entity, we want the model to select the correct entity e ∈ E to fill the blank slot. In our problem
2 On notation; we use A to denote sequences, A to denote sets, a to denote words/entities, a to denote vectors, A to denote matrices.
3Note that we restrict each context to only have one **blank** token.
setting, the model also has access to the lexical resource L = {Le | e ∈ E} where Le = (le1, ..., lek) is the lexical definition of entity e extracted from the knowledge base. Thus, the task of the model is to predict the correct missing entities for each empty slot in D.
There are several possible ways to specify the candidate set E . For instance, we could define E so that it includes all entities found in the corpus. However, given the extremely large amount of unique entities found in the dataset, this would render the task difficult to solve from both a practical and computational perspective. We present a simpler version of the task where E is the set of entities that are present in the document D. Note that we can make the task arbitrarily more difficult by randomly sampling other entities from the entity vocabulary and adding them to the candidate set.
We show an example from the Wikilinks Entity Prediction dataset, along with a visual guide to the notation from this section, in Figure 1.
4 Model Architectures
In this section, we present two models that use the lexical definitions of entities to solve the proposed rare entity prediction problem. The basic building blocks of our models are recurrent neural networks (RNN) with long short-term memory (LSTM) units. An RNN is a neural network with feedback connections that allows information from the past to be encoded in the hidden layer
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
e→ Le :
Ci: de
hei
w1 ... ... wm
le1 ... lek
P (e = ẽ|Ci, Le) = σ((hei )TWde + b)
Figure 2: Our double encoder architecture. Each entity e has an associated lexical definition Le = (le1, le2..., lek), which is fed through the lexical encoder g (orange squares) to provide an encoding de. This definition embedding is then fed as the blank input token of context Ci to the context encoder f (blue circles), which provides a context embedding hei .
h̄i = 1 |E| ∑ e′∈E h e′ ih̄i−1h̄i−2
...
P (e = ẽ|C1..i, Le) = σ((hei )TWde + rTi V + b) ri
... de′ ... de
heiCi
Figure 3: Our hierarchical encoder architecture. Each entity e is encoded as de, at each timestep, hei is computed for each e. h̄i is the average encoding, which is fed as input to the temporal model r (green diamonds). The temporal model produces ri, which is used to compute P (e = ẽ|C1..i, Le).
representation, thus is ideal for modelling sequential data (Dietterich, 2002) and most language related problems.
LSTMs are an extension of RNNs which include a memory cell ct alongside their hidden state representation ht. Reads and writes to the memory cell are controlled by a set of three gates that allow the model to either keep or discard information from the past and update their state with the current input (Hochreiter and Schmidhuber, 1997). This allows LSTMs to model potentially longer dependencies and at the same time mitigate the vanishing and exploding gradient problems, which are quite common among regular RNNs (Hochreiter, 1991; Bengio et al., 1994). In our experiments, we use LSTMs augmented with peephole connections (Gers et al., 2002).
We denote the output (i.e. the last hidden state) of an RNN f operating on a sequence S as f(S), and subscript the t-th hidden state as ft(S).
4.1 Double Encoder (DOUBENC)
We use two jointly trained recurrent models, a lexical encoder g(.) and a context encoder f(.), and a
logistic predictor P (see Figure 2). The lexical encoder converts the definition of an entity into a vector embedding, while the context encoder repeats the same process for a given context to obtain its context embedding. These two embeddings are then used by P to predict if the given entity-context pair is correct. Additionally, the blank in the context sentence is replaced by the encoded definition embedding to provide more information to f .
For an entity e in the candidate set E of document D, we retrieve its corresponding lexical definition Le, itself a sequence of words, to compute its encoding g(Le) ≡ de.
For a given context Ci, we replace the embedding of the blank token with de. Thus Ci = (w1, ..., wblank, ..., wm) becomes Cei = (w1, ...,de, ..., wm)
4. We then compute the context embedding of the new Cei , f(C e i ) ≡ hei
After getting hei and de, we wish to compute the probability of candidate e being the correct entity
4We mix the word/vector notation here since each word w is replaced by its corresponding word embedding vector.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
ẽ missing in context Ci. This probability is the output of the predictor:
P (e = ẽ|Ci, Le) = σ((hei )TWde + b)
where σ is the sigmoid function, W and b are model parameters. The cross term (hei )
TWde is a dot product between hei and de that weighs the dimensions differently based on the learned parametersW . A similar prediction method has been used successfully for question answering (Bordes et al., 2014b; Yu et al., 2014) and dialogue response retrieval (Lowe et al., 2015). Note that while hei is a function of de, using de in the cross term (hei )
TWde provides a much shorter gradient path from the loss to the lexical encoder through de.
Given a context, the model outputs a probability for each entity e ∈ E . Entities in the candidate set are then ranked against each other according to their predicted probabilities. The entity with the highest probability is considered as the most plausible answer for the missing entity in the current context. We consider the model to make an error if that entity is not ẽ.
4.2 Hierarchical Double Encoder
(HIERENC)
The double encoder model above considers each context independently. However, since each document consists of a sequence of contexts, the knowledge carried by other contexts in C could also provide useful information for the decision process of Ci.
To that end, we propose a hierarchical model structure by adding a LSTM network, which we call the temporal network r (see Figure 3), on top of the double encoder architecture. Since a document is a sequence of Cis, each timestep of this network consists of one such context, and thus is indexed with i.
Since we already have a context encoder f , we reuse the output of f(Cei ) as the input of r at timestep i. More specifically, we combine the embeddings generated by f into a single one via averaging: h̄i = 1|E| ∑ e′∈E h e′ i , which then serves as the input to the temporal network for context Ci. Note that alternatively, one could aggregate information about the past predictions through other means like policies or soft attention. However, this would introduce extra complexities to the learning process. As such, we use averaging to that end.
Finally, at each timestep i, the temporal network outputs an embedding ri(C1, ...,Cn) ≡ ri. We use
this temporal embedding to predict the probability of the context-entity pair using slightly altered P :
P (e = ẽ|C1..i, Le) = σ((hei )TWde + rTi V + b)
where W , V and b are model parameters. The entities in candidate set are again ranked against each other based on their probabilities.
5 Experiments
5.1 Experiment Setup
We randomly partition our dataset into training, validation and test sets. The training set consists of approximately 80% of the total dataset, the validation and test sets comprise about 10% each.
In our experiments, the contexts are defined as the sentences where the special **blank** tokens are found. The lexical definitions for each entity are the first sentences of their Freebase descriptions. We have experimented with different configurations of defining contexts and entity definitions, such as expanding the context window by including the sentences before and after the blank, as well as taking more than one sentence out of the entity description. However, results on the validation set show that increasing the context window size and definition size has very little impact on the evaluation metrics and drastically increases the training time of all models.
To train our models, we use the correct missing entity for each blank as the positive sample and all other entities in the candidate set as the negative samples. During the testing phase, we present each entity in the candidate set to our models and record the probabilities output by the models. The entity with the highest probability is chosen as the model prediction. For all gradient-based methods, including both baseline models and our proposed models, the learning objective is to minimize the binary cross-entropy of the training data.
We measure the performance on our entity prediction task using the accuracy; that is, the number of correct entity predictions made by the model divided by the total number of predictions. This is equivalent to the metric of Recall@1 that is often used in information retrieval. All our models are implemented using Theano (Theano Development Team, 2016).
5.2 Baselines
In order to demonstrate the effects of using lexical resources as external knowledge for solving
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
the task, we present here three sets of baselines: one set of simple baselines (RANDOM and FREQENT), one LSTM-based model that only relies on the contexts and does not utilize the definitions (CONTENC), and another set of models that do make use of the entity definitions but in a relatively simple fashion (TF-IDF + COS and AVGEMB + COS).
RANDOM For each context in a given document, this baseline simply selects an entity from the candidate set uniformly at random as its prediction.
FREQENT Under this baseline, we rank all entities in the candidate set by the number of times that they appear in the document. For each blank in the document, we always choose the entity with the highest frequency in that document as the prediction. Note that this model has access to extra information compared to the other models, in particular the total number of times each entity appears in the document.
CONTENC Instead of using their definitions, entities are treated as regular tokens in our vocabulary. Thus for a particular entity e, the context sequence Ci = [w1, ..., wblank, ..., wm] becomes [w1, ..., we, ..., wm]. We feed the sequence Ci into the context encoder and as usual take the last hidden state as the context embedding hei . Thus given Ci and e ∈ E, the probability of e being the correct missing entity is:
P (e = ẽ|Ci) = σ((hei )TW + b)
where again σ is the sigmoid function, W and b are model parameters. This model is essentially a language model baseline, that does not make use of the external a priori knowledge.
TF-IDF + COS This method takes the term frequency-inverse document frequency (TF-IDF) vectors of the context and the entity definition as their corresponding embeddings. The aggregations of contexts and definitions are treated as their own corpora, and two separate TF-IDF transformers are fitted. Candidate entities are ranked by the cosine similarity between their definition vectors and the context vector. The entity with the highest cosine similarity score is chosen as the prediction.
AVGEMB + COS This baseline computes the context embedding by taking the average of some
Accuracy Model Valid Test Fixed baselines RANDOM 0.294 0.301 FREQENT 0.329 0.331 Without external knowledge CONTENC 0.393 0.396 With external knowledge TF-IDF + COS 0.292 0.300 AVGEMB + COS 0.355 0.359 DOUBENC 0.547 0.540 HIERENC 0.573 0.566
Table 3: Results on the Wikilinks Entity Prediction dataset for all proposed baselines and models.
pre-trained word embeddings. The entities’ embeddings are computed in the same way. In our experiments, we choose to use the published GloVe (Pennington et al., 2014) pre-trained word embeddings. Same as above, the prediction is made by considering the cosine similarity between the context embedding and the entity embeddings.
5.3 Results
Empirical results are shown in Table 3. We test our two proposed model architectures (detailed in Section 4), in addition to baselines described in Section 5.2.
For the CONTENC baseline, we choose 300 as the size of hidden state for the encoder. For the DOUBENC and the HIERENC models, the size of hidden state for both the context encoder and the lexical encoder is set to 300. Additionally, an RNN with 200 LSTM units is used as the temporal network in the hierarchical case. All three models are trained with stochastic gradient descent with Adam (Kingma and Ba, 2014) as our optimizer, with a learning rate of 0.0005 used for CONTENC and 0.0001 used for DOUBENC as well as HIERENC. Models with the best performance on validation set are saved and used to test on the test set.
It is clear from Table 3 that models that only use contextual knowledge give relatively poor performance compared to the ones that utilize lexical resources. The large discrepancy between the context encoder and the double encoder shows that lexical resources play a crucial role in solving the task. The best result is achieved by the hierarchical double encoder, which confirms that knowing about previous contexts is indeed beneficial to the prediction at the current timestep.
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Context & Prediction [...] We heard from Audrey Bomse, who is with the Free Gaza movement. She was in , Cyprus. [...]
CONTENC: Istanbul HIERENC: Larnaca
Candidate Set Istanbul: Istanbul is the most populous city in Turkey, and the country’s economic, cultural, and historical center. Larnaca: Larnaca is a city on the southern coast of Cyprus and the capital of the eponymous district. Ben Macintyre: Ben Macintyre is a British author, historian, reviewer and columnist writing for The Times newspaper. (Other candidates......)
Table 4: An example from the test set, with the predictions made by CONTENC and HIERENC. HIERENC was able to successfully predict the correct missing entity, Larnaca.
However, even our best performing model, the hierarchical double encoder, is still a long way from solving this task, achieving an accuracy of 56.6%. This reflects the challenging nature of the task, as most of the predictions are about entities that have been seen fewer than 5 times in the entire corpus. Note that, since Singh et al. (2012) manually verified the quality of the original Wikilinks dataset, there are virtually no entities that are incorrectly labeled, and thus this is not a significant source of error.
6 Discussion
Table 4 shows an example found in the test set, along with the predictions from CONTENC and HIERENC. Although the context encoder was able to identify that the missing entity should be a city, it incorrectly predicted Istanbul. This is likely because Istanbul appears 86 times in the dataset, whereas Larnaca appears only twice. It seems that, although the context encoder was able to derive a strong association between Istanbul and Middle Eastern geolocations, such knowledge was not learned for Larnaca because of the lack of examples. Conversely, the hierarchical double encoder was able to take both the context and the external knowledge into account and successfully predicted the correct missing city.
One interesting observation is the margin of difference in accuracy between the context encoder
and the embedding average baseline. The context encoder, which is a relatively sophisticated context-only model, only slightly outperforms the simple embedding average baseline that has no learning component. This suggests that the lexical definitions are valuable in solving such tasks even when it is used in a rather simplistic way.
Another interesting observation is that, in our experiments we found that using a large context window size (including the sentences before and after the sentence where the blank is found) does not have any significant positive impact on the results. This implies that the words that are most informative about the missing entity in the blank are generally found in the vicinity of the blank. It is likely that more sophisticated models will be able to use the surrounding context information more effectively, leading to greater performance increases.
7 Conclusions
In this paper, we examine the use of external knowledge in the form of lexical resources for solving reading comprehension problems. Specifically, we propose the problem of rare entity prediction. In our Wikilinks Rare Entity Prediction dataset, the majority of the entities have very low frequencies across the entire corpus; thus, models that solely rely on co-occurrence statistics tend to underperform. We show that models leveraging the Freebase descriptions achieve large performance increases, particularly when this information is incorporated intelligently as in our double encoder-based models.
For future work, it would be interesting to examine the effects of other knowledge sources. In this paper, we use entity definitions as the source of external knowledge. However, Freebase also contains other types of valuable information, such as relational information between entities. Thus, one potential direction for future work would be to incorporate the relational information, alongside the lexical definitions, to achieve better results.
We have seen the crucial role that external knowledge plays in solving tasks with many rare entities. Thus we believe that incorporating external knowledge into other NLP systems, such as dialogue agents, should also see similar positive results. We plan to explore the idea of external knowledge integration further in our future research.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths: The paper empirically verifies that using external knowledge is a benefit.  - Weaknesses: Real world NLP applications should utilize external knowledge for making better predictions. The authors propose Rare Entity prediction task to demonstrate this is the case. However, the motivation of the task is not fully justified. Why is this task important? How would real world NLP applications benefit from this task? The paper lacks a convincing argument for proposing a new task. For current reading comprehension task, the evidence for a correct answer can be found in a given text, thus we are interested in learning a model of the world (i.e causality for example), or a basic reasoning model. Comparing to reading comprehension, rare entity prediction is rather unrealistic as humans are terrible with remembering name. The authors mentioned that the task is difficult due to the large number of rare entities, however challenging tasks with the same or even more difficult level exist, such as predicting correct morphological form of a word in morphologically rich languages. Such tasks have obvious applications in machine translation for example.  - General Discussion: It would be helpful if the authors characterize the dataset in more details. From figure 1 and table 4, it seems to me that overlapping entities is an important feature. There is noway i can predict the **blank** in figure 1 if I don't see the word London in Peter Ackoyd description. That's being said, before brutalizing neural networks, it is essential to understand the characteristic of the data and the cognitive process that searches for the right answer.  Given the lack of characteristic of the dataset, I find that the baselines are inappropriate. First of all, the CONTENC is a natural choice at the first sigh. However as the authors mentioned that candidate entities are rare, the embeddings of those entities are unrealizable. As a consequence, it is expected that CONTENC doesn't work well. Would it is fairer if the embeddings are initialized from pre-trained vectors on massive dataset? One would expect some sort of similarity between Larnaca and Cyprus in the embedding space and CONTENC would make a correct prediction in Table 4. What would be the performance of TF-IDF+COS and AVGEMB+COS if only entities are used to compute those vectors?  From modeling perspective, I appreciate that the authors chose a sigmoid predictor that output a numerical score between (0,1). This would help avoiding normalization over the list of candidates, which are rare and is difficult to learn reliable weights for those. However, a sidestep technique does exist, such as Pointer Network. A representation h_i for C_i (*blank* included) can be computed by an LSTM or BiLSTM, then Pointer Network would give a probabilistic interpretation p(e_k|C_i)   propto exp(dot(d_{e_k}, h_i)). In my opinion, Pointer Network would be an appropriate baseline. Another related note: Does the unbalanced set of negative positive labels affect the training? During training, the models make 1 positive prediction while number of negative predictions is at least 4 times higher?  While I find the task of Rare Entity prediction is unrealistic, having the dataset, it would be more interesting to learn about the reasoning process that leads to the right answer such as which set of words the model attends to when making prediction.