A Weakly-Supervised Method for Jointly Embedding Concepts, Phrases, and Words
Recent work on embedding ontology concepts has relied on either expensive manual annotation or automated concept tagging methods that ignore the textual contexts around concepts. We propose a novel method for jointly learning concept, phrase, and word embeddings from an unlabeled text corpus, by using the representative phrases for ontology concepts as distant supervision. We learn embeddings for medical concepts in the Unified Medical Language System and generaldomain concepts in YAGO, using a variety of corpora. Our embeddings show performance competitive with existing methods on concept similarity and relatedness tasks, while requiring no human corpus annotation and demonstrating more than 3x coverage in the vocabulary size.
1 Introduction
Recent vector space models of semantics are primarily built on words. Word embeddings are either used directly as features in downstream modeling tasks (Chiu et al., 2016b), or are composed to produce representations of more complex linguistic units, such as phrases, sentences (Socher et al., 2011; Le and Mikolov, 2014), or even documents (Yang et al., 2016). However, there are many cases in which words are not atomic semantic units, and multi-word expressions cannot be expressed as a composition of their member words: for example, “the Big Apple” is not a fruit, and “Lou Gehrig’s disease” has little to do with baseball. Furthermore, such concepts often multiple textual forms: “the Big Apple” and “New York City” both refer to the same location, and “Lou Gehrig’s disease” and “amyotrophic lateral scle-
rosis” are the same medical condition. Despite the lack of lexical overlap between these phrases, we would like a semantic model that can represent the underlying concept, regardless of the specific textual form used.
Our novel approach to this task combines structured knowledge with proven techniques for learning word embeddings to train context-based representations for contexts, phrases, and words. The model requires no human annotations: we use known phrases as distant supervision in distributional similarity training over an unannotated corpus. As in the formulation of Mintz et al. (2009) in extracting textual relationships, we assume that any occurrence of a known phrase signifies an occurrence of each concept1 it can refer to, and train our concept embeddings using the contexts of their phrase forms. We experimentally validate our approach by learning embeddings for biomedical concepts and real-world entities. An evaluation on concept similarity and relatedness tasks shows that our embeddings are competitive with prior models that required human annotations for concepts. We also present a novel dataset of similarity and relatedness of real-world entities, identified both by Wikipedia page and text phrase.
Furthermore, analysis suggests the source ontology structure is reflected in the organization of our embeddings: concepts and their representative phrases are embedded close to one another, and embedded concepts of the same semantic type cluster together with some regularity.
2 Related Work
Single word embeddings have been directly used as entity/concept models in prior work, includ-
1The term “concept” has been used in prior literature to mean both a specific concept in an ontology and a specific sense of a word. In this study, “concept” refers to a canonical abstract concept or real-world entity.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
ing in analogy completion tasks (Mikolov et al., 2013b; Linzen, 2016; Gladkova et al., 2016) and in similarity/relatedness tasks (Muneeb et al., 2015; Chiu et al., 2016a). Multi-word concepts have also been explored by approaches such as lexicalization and word averaging (Mikolov et al., 2013a; Socher et al., 2013). Fan et al. (2015), Yu et al. (2016), and Hill and Korhonen (2014) modify embedding training with additional information, such as named entity mentions and associated perceptions.
Additionally, a number of models have been proposed for the related task of separately embedding different word senses. Camacho-Collados et al. (2015) use a semantic network to train word sense representations using relevant subsets of Wikipedia. Nieta Pina and Johansson (2016) use WordNet graph structure as context for embedding word senses; a similar approach was taken by Hu et al. (2015) for embedding entities, using the hierarchy of the containing ontology augmented with textual information.
In all of these cases, however, words are used as atomic units, and can be composed in order to model concepts. However, there is also a rich literature on learning embeddings of knowledge base entities and ontology concepts directly from the structure, relationships, and attributes encoded in the knowledge base (Bordes et al., 2011, 2013; Wang et al., 2014b; Lin et al., 2015). Toutanova et al. (2015) decomposed textual relations into single words to improve generalization, but still relied on knowledge graph structure for embedding entities; Socher et al. (2013) and Yang et al. (2015), however, incorporated pre-trained word embeddings as an initialization for entity representations, using word averaging.
Several alternative methods for embedding concepts atomically have been proposed in the biomedical domain. De Vine et al. (2014) automatically tag a corpus for biomedical concepts and use the sequence of concept identifiers as training context for word2vec, while Mencia et al. (2016) use the text content of documents tagged with MeSH headers to learn representations for the headers. Choi et al. (2016a) in contrast, use sequences of human-annotated medical codes to learn embeddings; Choi et al. (2016b) take a similar approach, and also learn concept embeddings from concept identifier cooccurrence statistics.
Wang et al. (2014a) is the closest approach to ours, as they propose methods for jointly embedding KB entities and words into the same space. However, their embedding method relies on the binary relations between entities in the knowledge base, which have been shown to be highly incomplete (Min et al., 2013). Their best results also rely on human annotation via Wikipedia links.
3 Method
We propose a method to jointly embed concepts, phrases, and words into a real-valued space, using structured knowledge from an ontology and an unannotated training corpus.
Let C denote the set of canonical concepts in an ontology. For any specific concept c ∈ C, let Pc denote the set of phrases that can be used to represent it; P denotes the union of all of these sets of phrases. Figure 1 illustrates this many-tomany relationship between concepts and phrases.
Additionally, let T denote the sequence of tokens in a training corpus, and W be the word vocabulary used in it.
3.1 Preprocessing
After extracting the mapping between concepts and phrases from the ontology, each phrase is assigned a unique identifier. Then, all occurrences of the mapped phrases in the training corpus are replaced with the unique identifiers, converting them to unigrams. For example, in the sentence
Patient was diagnosed with chronic
obstructive lung disease in December.
the phrase “chronic obstructive lung disease” will be replaced with its unique identifer, yielding
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Patient was diagnosed with PHRASE 1337
in December.
This replacement is done greedily over the number of tokens matched, producing two parallel versions of the training corpus: the original, untagged text (TU ), and the text tagged for mapped phrases (TT ). One token in TT may correspond to multiple tokens in TU .
3.2 Proposed model
Our model adapts the skip-gram with negative sampling variant of word2vec for joint embedding of concepts, phrases, and words. Embeddings of dimensionality d utilize three matrices:
• EW = |W | × d; word embeddings
• EP = |P | × d; phrase embeddings
• EC = |C| × d; concept embeddings
Additionally, we use a single negative sampling matrix ENS = |W | × k. The relationships between these matrices are illustrated in Figure 2.
To train the embeddings, we iterate over the tagged and untagged versions of the training corpus in parallel, using a sliding context window of k untagged words. Word embeddings are trained using all words around them, and phrase embeddings are trained using the word contexts around the complete phrase. Concept embeddings are updated using the contexts of phrases that can represent them; for a mention of phrase p, these updates are normalized by the number of concepts p can represent. In all cases, the embeddings are trained to maximize the log-likelihood of their observed contexts, and to minimize the log-likelihood of randomly-selected negative samples.
3.2.1 Training objective
To calculate the loss for a single (w, p, o) observation, then, where w is the observed word, p is
EC
EP
EW
ENS
η
β
Figure 2: Schematic of proposed model; the dashed ENS box is discarded after training.
the observed phrase just completed, and o is the observed context word, we do as follows. Let Cp be the set of concepts that p can represent, and N be the set of negative samples for this observation. We follow the notation of (Levy and Goldberg, 2014) in representing the probability of the pair (w, o) coming from the true data distribution:
P (D = 1|w, c) = σ(~w · ~c) = 1 1 + e−~w·~c (1)
Then, the word embedding loss `w, phrase embedding loss `p, and concept embedding loss `c can be calculated as follows:
`w = logσ(~w · ~o)− ∑ n∈N logσ(~w · ~n) (2)
`p = logσ(~p · ~o)− ∑ n∈N logσ(~p · ~n) (3)
`c∈Cs = [ logσ(~c · ~o)− ∑ n∈N logσ(~c · ~n) ] 1 |Cs|
(4)
3.2.2 Enforcing embedding similarity
The training algorithm described in Section 3.2 shares information between words, phrases, and concepts only via the shared negative sample embedding matrix. This can be beneficial, as some phrases are not easily decomposable; in the case of “Lou Gehrig’s disease”, knowing that Lou Gehrig played baseball says little about ALS. However, many phrases have some compositionality (e.g., “high blood pressure”), and many phrases directly encode useful information about the concept (e.g., “amyotrophic lateral sclerosis”).
To enable a tradeoff between independence and compositionality in our training, we introduce two hyperparameters to our model: β controls the compositionality of phrases by words, and η controls the compositionality of concepts by phrases.
To construct the updated loss function, let Wp be the list of words in phrase p, andWp be the average of their embeddings. For a given concept c, let Pc denote the set of phrases that can represent it, with Pc being the average of their embeddings.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Now, the updated objective functions look like:
`w =logσ(~w · ~o)− ∑ n∈N logσ(~w · ~n) (5)
`p = ( 1− β )( logσ(~p · ~o)− ∑ n∈N logσ(~p · ~n) )
+ β ( logσ(~p · Wp) ) (6)
`c∈Cs = [( 1− η )( logσ(~c · ~o)− ∑ n∈N logσ(~c · ~n) )
+ η ( logσ(~c · Pc) )] 1 |Cs|
(7)
4 Materials
4.1 Training corpora
We train our embeddings on four corpora. Prior work has noted the effects of training corpus choice on a variety of tasks in the biomedical domain (Pakhomov et al., 2016; Griffis et al., 2016; Garla and Brandt, 2012), so we experiment with literature abstracts, clinical narratives, and their combination. Table 1 lists statistics about concepts and phrases embedded from each corpus.
PUBMED All abstracts from the 2016 PubMed baseline of biomedical literaure (i.e., all abstracts present in PubMed at the beginning of 2016).
CLINICAL A set of clinical notes from the Ohio State Wexner Medical Center, regarding patients diagnosed with a variety of chronic conditions.
PM+CL Concatenation of the above corpora. WIKINYT Combination of all articles in a 2015 dump of English Wikipedia and the New York Times portions of Gigaword (Parker et al., 2011).
4.2 Ontologies
Our canonical concepts and their representative strings are sourced from two ontologies: the Unified Medical Language System (UMLS) (Bodenreider, 2004), and YAGO (Suchanek et al., 2007; Mahdisoltani et al., 2015). The UMLS Metathesaurus provides mappings between more than 100 biomedical vocabularies, enumerates the medical concepts described, and includes canonical forms (Concept Unique Identifiers, or CUIs) for each concept. CUIs are further mapped to one or more textual forms, along with semantic type information. YAGO is a knowledge base encoding information about real-world entities and the relationships between them, including information from
WordNet (Fellbaum, 1998) and Wikipedia. As with the UMLS, YAGO maps between canonical entity identifiers and one or more textual forms.
In both cases, we restrict ourselves to text forms in English, and entities with at least one text form that appears ≥ 25 times in our corpora.
4.3 Evaluation datasets
4.3.1 Biomedical similarity and relatedness
To evaluate our biomedical concepts, we used the MayoSRS and Mini Mayo SRS datasets from (Pedersen et al., 2007) and the UMNSRS Similarity and Relatedness datasets of (Pakhomov et al., 2010). MayoSRS consists of 101 pairs of concepts, identified by both UMLS CUI and text form, ranked on a scale of 1-4; the Mini Mayo SRS set includes 29 of these pairs, selected for high inter-rater agreement. The UMNSRS Similarity and Relatedness datasets are composed of 566 and 587 concept pairs, respectively, ranked on a scale of 1-1600 for similarity or relatedness; CUI and text forms are provided for all concepts.
4.3.2 General-domain similarity and relatedness
While there are well-studied datasets of word similarity and relatedness available, such as WordSim-353 (Finkelstein et al., 2001), SimLex999 (Hill et al., 2015), as well as datasets for evaluating compositionality (Marelli et al., 2014) and typed relations between nouns (Baroni and Lenci, 2011), we are not aware of comparable similarity or relatedness datasets for real-world entities. We therefore present two novel datasets composed of human judgments of similarity and relatedness between pairs of people, places, and organizations.
We used Amazon’s Mechanical Turk crowdsourcing platform for our data collection. Participants were asked to assign a similarity or relatedness score from 0-100 to each of 34 pairs of entities, with a higher score indicating higher similarity/relatedness. We collected judgments for 24 such sets of entity pairs, getting separate judgments for similarity and for relatedness. 4 entity pairs in each set were used as validation questions, as they were determined to tend strongly towards either high or low similarity/relatedness. These pairs were not included in the final dataset; filtering for participants who self-reported high English reading proficiency and who gave reasonable responses to the validation questions pro-
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Corpus Onto # Docs # Tokens # Concepts Max Phrases Avg
Phrases PUBMED
UMLS 14.7m 2.7b 188,486 390 5.4
CLINICAL 377k 160.3m 37,219 160 7.2 PM+CL 15.1m 2.8b 193,064 390 5.4 WIKINYT YAGO 6.2m 3.5b 748,752 7,191 3.2
Table 1: Number of documents and tokens in each of the corpora used for training concept embeddings, along with the number of concepts (from ontology Onto) found in each. Max Phrases is the maximum number of representative phrases for a single concept that appear at least 25 times in the corpus; Avg Phrases is the average number of mapped phrases over all concepts.
duced a final dataset of 688 pairs for each task (658 pairs present in both filtered datasets). Participants were paid at least $0.75 for each survey, with the rate increased as needed to guarantee state minimum wage for the time spent on the survey. Overall Fleiss’ κ for our annotations was 0.24 for both datasets, indicating fair agreement between annotators. For further details about our data collection and analysis of response, please see the supplemental material.
5 Experiments
5.1 Model comparisons
To evaluate the benefit of embedding concepts directly, we compare with several approximations using embeddings learned on our corpora:
• APPROXPHRco We approximate a concept’s embedding as the average of the embeddings of its representative strings.
• APPROXWORD We approximate a concept’s embedding as the average embedding of the words used in it representative strings.
• PHRASEco We use the learned embedding of a particular string directly.
• WORD We approximate a particular string as the average embedding of its words.
Domain Dataset # of pairs
Medical
Mayo SRS 101 Mini Mayo SRS 29
UMNSRS Relatedness 587 UMNSRS Similarity 566
General Relatedness 688
Similarity 688
Table 2: Similarity and relatedness datasets used
In all settings, we use subscript co to denote jointly-trained embeddings; WORDind indicates word embeddings trained alone using the skipgram model of word2vec.
In addition, we compare with a variety of pretrained embeddings from prior research:
• DEVINE Embeddings of 52k UMLS CUIs trained over sequences of CUIs automatically tagged from EHR data and medical abstracts (De Vine et al., 2014).
• CHOI Embeddings of 15k UMLS CUIs trained via hierarchical sampling over sequences of manually annotated claims codes from EHR data (Choi et al., 2016b).
• MENCIA Embeddings of 26k MESH headings trained over manually-tagged PubMed abstracts to maximize similarity of headings to relevant documents (Mencia et al., 2016).
• CHIU-2 Word embeddings trained on PubMed abstracts with a context window size of 2 (Chiu et al., 2016a).
• CHIU-30 Word embeddings trained on PubMed abstracts with a context window size of 30 (Chiu et al., 2016a).
5.2 Hyperparameter tuning
Recent findings highlight the importance of hyperparameter selection for performance of distributed representations on downstream tasks (Levy et al., 2015; Chiu et al., 2016a). We therefore evaluate the impact of various hyperparameters on the similarity and relatedness task, and use the results to tune optimal settings for further evaluations.2 In
2(Chiu et al., 2016a) point out that in the biomedical domain, hyperparameter settings that improve performance on intrinsic evaluations of the vector space may degrade perfor-
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Corpus β η NS WS Dim PUBMED 0.75 0.0 5 30 400 CLINICAL 0.25 0.25 10 5 600 PM+CL 0.0 0.0 5 30 100 WIKINYT 0.0 0.0 5 5 300
Table 3: Tuned hyperparameters for each corpus; NS is the number of negative samples used, WS is the size of the context window, and Dim is the dimensionality of the vector space.
addition to the β and η tradeoff parameters of our model, we experiment with number of negative samples, vector dimensionality, and context window size. In all cases, we trained for 10 iterations. Table 3 lists the final tuned settings for each corpus; for more details on our hyperparameter tuning, please see the supplementary material.
5.3 Similarity and relatedness
Given a pair of ontology entities 〈e1, e2〉, we calculate both their similarity and relatedness via the cosine similarity of their embeddings. We then rank each list of entity pairs in order of decreasing cosine similarity, and compare our ranking against the ranked list of human similarity or relatedness judgments. We report Spearman’s rank correlation coefficient (Spearman’s ρ), which ranges from -1 (reverse ranking) to 1 (same ranking); Table 4 shows our results. On the two smaller biomedical datasets, our method is outperformed by CHOI; however, on the larger datasets, our concepts embeddings have the highest correlation with human judgments on the similarity task, and competitive performance on the relatedness task (jointly embedded phrases are negligibly better than CHIU-2). On our real-world entity datasets, the phrase-based concept approximation outperforms the direct concept embeddings.
5.4 Concepts, phrases, and words
Since our model jointly embeds concepts, phrases, and words into the same space, we assess how well the ontological links between concepts and phrases and between phrases and words are preserved. We first approximate each concept and phrase by averaging the embeddings of their representative phrases and words, and calculate co-
mance in extrinsic downstream applications. Our analysis is restricted to evaluations relying on the affine organization of the vector space; thus, we note that these optimized hyperparameters will likely change for downstream tasks.
sine similarity with the learned concept or phrase embedding. Figure 3a shows that almost every concept has very high similarity to its mean phrasal embedding, but we see higher variance when comparing phrases to the average of their words (Figure 3b). This is further reflected in Figure 3c, where we see that most concepts has cosine similarity near 0.5 to the average of the words of its representative phrases.
To assess if this pattern holds for individual cases, we examine the phrases and words in the neighborhood of a parent concept or phrase. In particular, given a parent concept or phrase p, we calculate the mean average precision (MAP) of Cp, the child phrases or words connected to it, within the full phrase or word vocabulary V .
We first rank all v ∈ V in order of descending cosine similarity to p; this produces the ranked list V̂ . Then, for each child term c in the “correct” subset Cp ⊂ V , we calculate its precision as:
Pr(c|Cp, V ) = 1V̂<c
V̂c (8)
where V̂c denotes the index of c in V̂ , and 1V̂<c is an indicator function over the elements of V̂ up to c that is 1 if the element is in Cp and 0 otherwise. The MAP calculation is then
MAP(p|V ) = 1 |P | ∑ p∈P
∑ c∈Cp Pr(c|Cp, V )
|Cp| (9)
Figure 4a shows that the resulting MAP scores for concepts and their phrases are skewed towards 0, with clear peaks at 0.5 and 1. This suggests that most concepts are not especially near their individual representative phrases in the vector space, in comparison to the full phrase vocabulary. In contrast, the MAP scores for phrases and their words are strikingly bi-modal near 0 and 1, indicating that most phrases are either quite close or very distant from their component words in the embedding space. Taken together, these results indicate that our model seems to be distributing component phrases or words fairly equally around their parents, though at varying distance.
5.5 Semantic type clustering
Choi et al. (2016b) propose the Medical Conceptual Similarity Measure, which evaluates how well embedded concepts of the same semantic
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Corpus Setting Mayo SRS Mini Mayo SRS
UMNSRS Relatedness
UMNSRS Similarity
PM+CL
WORDind 0.237 0.400 0.546 0.567 WORDco 0.370 0.505 0.579 0.615
PHRASEco 0.618 0.477 0.591 0.628 APPROXWORDind 0.376 0.606 0.375 0.445 APPROXWORDco 0.493 0.600 0.400 0.495
APPROXPHRco 0.609 0.623 0.543 0.637 CONCEPTco 0.593 0.591 0.559 0.653
Concept baselines DEVINE(CUIs only) 0.559 0.434 0.422 0.455
CHOI(+ICD9,etc.) 0.817 0.726 0.384 0.552 MENCIA(MeSH headers) 0.646 0.572 0.534 0.565
Word baselines CHIU-2(words only) 0.368 0.565 0.496 0.560CHIU-30(words only) 0.472 0.630 0.590 0.653
Table 4: Similarity/relatedness results for tuned embeddings on biomedical datasets. Reported scores are Spearman’s ρ, which ranges [−1, 1]. We only report PM+CL results here; PUBMED and CLINICAL are similar. Best-performing settings for each dataset are marked in bold.
Figure 3: Compositionality evaluation
Setting Relatedness Similarity WORDind 0.602 0.589 WORDco 0.645 0.636
PHRASEco 0.784 0.743 APPROXWORDind 0.578 0.588 APPROXWORDco 0.615 0.624
APPROXPHRco 0.810 0.766 CONCEPTco 0.789 0.750
Table 5: Similarity/relatedness results for tuned embeddings on real-world entity datasets.
type cluster in a fixed neighborhood of k nearest neighbors. To obtain a more complete picture of the space, we adapt their methodology to find all concepts of a given semantic type within the entire space; as this is a ranking problem, we report mean average precision. We follow Choi et al. (2016b) in selecting 6 common semantic types to evaluate: disease or syndrome, pharmacologic substance, neoplastic process, finding, injury and poisoning, and sign or symptom (this replaces clinical drug, as not all concept embedding sets contained multiple concepts of that type).
We evaluated our three biomedical concept embeddings, and compared against the four medical concept embeddings from prior work; the
results are shown in Table 6. The medical code embeddings of Choi et al. (2016b) demonstrate superior clustering in aggregate,3 but clustering performance is mixed among the semantic types, with our embeddings clustering competitively with prior work. In particular, “sign or symptom,” which has many exemplars in each embedding set, is uniformly poorly clustered, suggesting that its semantics are difficult to capture.
6 Discussion
Despite the noisiness of our distantly-supervised training method, our results indicate that we capture certain semantic characteristics of ontology concepts nearly as well as prior methods that require significantly more annotation. Furthermore, we can scale to a much larger vocabulary without additional expense. However, there are a number of points that bear further examination.
Our results comparing APPROXWORDco to APPROXWORDind align with prior work indicating that including structured knowledge about concepts in training word embeddings improves
3Choi et al. (2016b) propose another embedding technique that has superior MAP (0.20) on finding, which we omit for brevity.
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Figure 4: Mean average precision distributions for concept-phrase and phrase-word comparisons.
STY PM CL PM
+CL DV Choi ME
pharmacologic
substance 0.26 0.14 0.20 0.17 0.21 0.13
disease or syndrome 0.20 0.12 0.15 0.21 0.33 0.16
injury or poisoning 0.04 0.02 0.03 0.04 0.36 0.05
sign or
symptom 0.09 0.06 0.05 0.06 0.02 0.02
neoplastic
process 0.20 0.04 0.18 0.14 0.25 0.10
finding 0.12 0.12 0.11 0.09 0.12 0.01 Overall 0.15 0.08 0.12 0.12 0.22 0.08
Table 6: MAP for semantic type clustering with selected biomedical semantic types (STY). PM=PUBMED, CL=CLINICAL, DV=DeVine, ME=Mencia. Highest MAP per semantic type is marked in bold.
their performance in some semantic tasks (Wang et al., 2014a; Faruqui et al., 2015). However, the extreme similarity observed in our compositionality analysis, and the high performance of concept approximations, suggests that some non-lexical information about concepts is not being captured in our method. Leveraging an approach similar to Wang et al. (2014a) and directly including ontology structure as a training criterion may be one way to model some of this additional information.
Chiu et al. (2016b,a); Faruqui et al. (2016) describe several issues with evaluating linguistic embeddings with similarity and relatedness tasks, and poor generalization of results to downstream applications. While we address some of their concerns (e.g., our concept embeddings rule out polysemy), several of the issues they describe remain. One of these is the lack of train/test splits on the similarity and relatedness tasks; though we do not
train our model directly on these data, we are still tuning on the test set. We attempt to mitigate this issue by an adapted 5-fold cross-validation, and our other analyses of the space indicate that our embeddings are capturing some semantic properties, but applications to other intrinsic tasks such as analogy completion or downstream NLP tasks would provide additional strong evidence of these properties. While analogies at the entity/concept level are limited, we suggest that applications such as entity linking be pursued in future work.
Finally, we note a slight mismatch between the ontologies we used. The UMLS encodes information about abstract biomedical concepts, while YAGO is focused on concrete, real-world entities. We demonstrate that our method is able to model both kinds of information, but it would benefit from an application to more abstract data in the general domain, similar to previous work with BabelNet (Camacho-Collados et al., 2015).
7 Conclusions
We propose a novel model to jointly embed canonical concepts, the phrases that represent them, and words into a shared vector space. Our method only uses distant supervision from phrases linked to concepts in an ontology, but we experimentally demonstrate that our embeddings preserve several semantic properties comparably to recent methods that require human-annotated data. In particular, our concept embeddings maintain similarity and relatedness, as evaluated by cosine similarity, and preserve the links between concepts of the same type and between concepts and their representative phrases.
With this paper, we include cui2vec, a software implementation of our method, as well as our novel dataset of similarity and relatedness of realworld entities.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
A Supplemental Material
A.1 Hyperparameter tuning We used an adapted 5-fold cross-validation for our hyperparameter tuning. First, each dataset was partitioned into 5 portions. Then, each hyperparameter setting was evaluated on each of the other 4 partitions, and the average of those results was considered to be that setting’s error for the target partition. The overall error of each setting is then the average of its 5 per-fold errors.4
We evaluated the following settings: β, η ∈ {0, 0.25, 0.5, 0.75, 1}, number of negative samples ∈ {5, 10, 15}, vector dimension ∈ {50, 100, 200, 300, 400, 500, 600}, and context window size ∈ {2, 5, 10, 30}. Default hyperparameter settings were 5 negative samples, window size of 2, and vector dimension of 100. Chiu et al. (2016a) found that a learning rate of α = 0.05 performed better than the standard 0.025 with biomedical tasks; accordingly, we use α = 0.05 in our biomedical corpora. However, the traditional learning rate of α = 0.025 gave better performance on our general-domain corpus. In all cases, we restricted ourselves to a minimum frequency threshold for words and phrases of 255.
In general, we find that while enforcing some small amount of concept-phrase or phrase-word with the β and η parameters can be helpful, similarity and relatedness performance trend negative as β and η increase. This negative trend is more pronounced with the concept-phrase similarity, controlled by the η parameter; a non-zero η is only optimal in a single corpus setting.
With regards to the other hyperparameters we evaluated, no clear trends emerged, as each hyperparameter behaved differently in each corpus. However, the differences between best and worstperforming hyperparameter settings were relatively small (the largest was a difference in Spearman’s ρ of .075, from vector dimensionality in
4Since the similarity and relatedness evaluation is a ranking task, evaluated by Spearman’s ρ, this is not equivalent to simply ranking the full dataset; in fact, the validation numbers skew worse than the overall numbers, as the smaller dataset size in the cross-fold evaluations penalizes ranking errors more highly.
5This was done primarily for memory constraints. However, given the small size of our Clinical corpus, we experimented with smaller thresholds; we found overall decreases in performance and only small gains in the number of concepts modeled.
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
the CLINICAL corpus). The most consistentlyperforming hyperparameter settings across the corpora were negative samples=5, vector dimensionality=100, and context window size=5.
A.2 Similarity/Relatedness dataset
We followed a similar process to Pakhomov et al. (2010) in selecting the entity pairs to be used in our dataset. We first filtered all entities in YAGO to the subset that we learned embeddings for (≈ 700k); we then filtered to only entities labeled with WordNet types organization or person, or with the YAGO type geoEntity. For each pairing of these categories (Organization-Organization, Organization-Place, Organization-Person, PlacePlace, Place-Person, and Person-Person), we manually selected 30 pairs of entities for each of the following relatedness categories: Completely Unrelated, Somewhat Unrelated, Somewhat Related, and Highly Related. These produced the list of 720 entity pairs we used for our Mechanical Turk surveys.
We augmented each survey of 30 questions with 4 manually-created validation pairs using common entities (e.g., London, New York), each of which was categorized as Highly Related or Completely Unrelated. We included these validation questions at random indices in our surveys. To evaluate if participants were reading the questions, we binned their ratings on these validation questions into 0-25 (Completely Unrelated), 26- 50 (Somewhat Unrelated), 51-75 (Somewhat Related), and 76-100 (Highly Related). If a participant’s ratings disagreed with ours on multiple validation questions, we discarded their data (we allowed disagreement on a single question, as some validation questions had high variance in responses among reliable annotators).
We recruited 6 participants for each survey, for a total of 34 unique participants across the 48 HITs. The majority responded to a single HIT, while 3 completed more than 20. We discarded all submissions from 3 participants, as they did not report English reading proficiency (1) or did not satisfy the validation questions (2). All participants were paid, regardless of if we used their data or not.
To generate the final dataset, we assessed each participant’s responses to the validation questions in each survey. We kept surveys for which we had at least 4 participants with satisfactory answers to
Figure 5: Distribution of Fleiss’ kappa for the HITs in our similarity and relatedness datasets.
the validation questions; this resulted in discarding 1 of the 24 HITs for each task. Due to 2 repeated pairs, this gave us final dataset sizes of 688 pairs for each of similarity and relatedness, 658 of which were shared between the tasks.
In assessing inter-annotator agreement (IAA), we considered each HIT individually, as we had neither the same participants nor the same number of accepted responses for all HITs. Figure 5 shows the distribution of Fleiss’ κ for the HITs in each dataset; the average for each case was 0.24.

The authors presents a method to jointly embed words, phrases and concepts, based on plain text corpora and a manually-constructed ontology, in which concepts are represented by one or more phrases. They apply their method in the medical domain using the UMLS ontology, and in the general domain using the YAGO ontology. To evaluate their approach, the authors compare it to simpler baselines and prior work, mostly on intrinsic similarity and relatedness benchmarks. They use existing benchmarks in the medical domain, and use mechanical turkers to generate a new general-domain concept similarity and relatedness dataset, which they also intend to release. They report results that are comparable to prior work.  Strengths:  - The proposed joint embedding model is straightforward and makes reasonable sense to me. Its main value in my mind is in reaching a (configurable) middle ground between treating phrases as atomic units on one hand to considering their compositionallity on the other. The same approach is applied to concepts being  composed  of several representative phrases.  -  The paper describes a decent volume of work, including model development, an additional contribution in the form of a new evaluation dataset, and several evaluations and analyses performed.  Weaknesses:  - The evaluation reported in this paper includes only intrinsic tasks, mainly on similarity relatedness datasets. As the authors note, such evaluations are known to have very limited power in predicting the utility of embeddings in extrinsic tasks. Accordingly, it has become recently much more common to include at least one or two extrinsic tasks as part of the evaluation of embedding models.  - The similarity relatedness evaluation datasets used in the paper are presented as datasets recording human judgements of similarity between concepts. However, if I understand correctly, the actual judgements were made based on presenting phrases to the human annotators, and therefore they should be considered as phrase similarity datasets, and analyzed as such.  - The medical concept evaluation dataset,  mini MayoSRS  is extremely small (29 pairs), and its larger superset  MayoSRS  is only a little larger (101 pairs) and was reported to have a relatively low human annotator agreement. The other medical concept evaluation dataset,  UMNSRS , is more reasonable in size, but is based only on concepts that can be represented as single words, and were represented as such to the human annotators. This should be mentioned in the paper and makes the relevance of this dataset questionable with respect to representations of phrases and general concepts.   - As the authors themselves note, they (quite extensively) fine tune their hyperparameters on the very same datasets for which they report their results and compare them with prior work. This makes all the reported results and analyses questionable.  - The authors suggest that their method is superb to prior work, as it achieved comparable results while prior work required much more manual annotation. I don't think this argument is very strong because the authors also use large manually-constructed ontologies, and also because the manually annotated dataset used in prior work comes from existing clinical records that did not require dedicated annotations.  - In general, I was missing more useful insights into what is going on behind the reported numbers. The authors try to treat the relation between a phrase and its component words on one hand, and a concept and its alternative phrases on the other, as similar types of a compositional relation. However, they are different in nature and in my mind each deserves a dedicated analysis. For example, around line 588, I would expect an NLP analysis specific to the relation between phrases and their component words. Perhaps the reason for the reported behavior is dominant phrase headwords, etc. Another aspect that was absent but could strengthen the work, is an investigation of the effect of the hyperparameters that control the tradeoff between the atomic and compositional views of phrases and concepts.  General Discussion:  Due to the above mentioned weaknesses, I recommend to reject this submission. I encourage the authors to consider improving their evaluation datasets and methodology before re-submitting this paper.  Minor comments:  - Line 069: contexts to concepts  - Line 202: how are phrase overlaps handled?  - Line 220: I believe the dimensions should be |W| x d. Also, the terminology  negative sampling matrix  is confusing as the model uses these embeddings to represent contexts in positive instances as well.  - Line 250: regarding  the observed phrase just completed , it not clear to me how words are trained in the joint model. The text may imply that only the last words of a phrase are considered as target words, but that doesn t make sense.   - Notation in Equation 1 is confusing (using c instead of o)  - Line 361: Pedersen et al 2007 is missing in the reference section.  - Line 388: I find it odd to use such a fine-grained similarity scale (1-100)  for human annotations.  - Line 430: The newly introduced term  strings  here is confusing. I suggest to keep using  phrases  instead.  - Line 496: Which task exactly was used for the hyper-parameter tuning? That  important. I couldn t find that even in the appendix.  - Table 3: It  hard to see trends here, for instance PM+CL behaves rather differently than either PM or CL alone. It would be interesting to see development set trends with respect to these hyper-parameters.  - Line 535: missing reference to Table 5.


A Comparison of Robust Parsing Methods for HPSG
This paper explores several techniques for enhancing coverage when parsing with HPSG grammars, determines appropriate evaluation methods, and uses them to compare performance. Depending on the dataset, baseline coverage gaps can be reduced by between 75% and 100%, while simultaneously improving EDM F1 scores.
1 Introduction
Head-driven Phrase Structure Grammar (Pollard and Sag, 1994) is a constraint-based grammar formalism that combines linguistically motivated descriptive mechanisms with precise and computationally tractable logical and combinatorial properties. The design goals for computational language systems generally include the following (doubtless in addition to many others):
1. coverage — all naturally-occurring inputs can be analyzed 2. precision — only naturally-occurring inputs can be analyzed1 3. efficiency — analysis happens quickly 4. semantic competence — meaning2 is pre-
served and made explicit 5. succinctness — knowledge embodied by the
system is well-organized The relative priority of these goals tends to differ from one system to the next, however. Whereas many system designers will rank coverage and perhaps efficiency as the most important of these five desiderata, most HPSG grammarians would rank precision, semantic competence, and possibly succinctness higher (Flickinger, 2011).
1Or inputs that could reasonably be expected to occur in some natural context.
2By this is meant that portion of the meaning of an input which is of specific interest to the system in question.
While this is not the place to expound at length on the motivation for a precision grammar, it is perhaps instructive to mention two important benefits: first, by excluding analyses of strings that the grammarian judges to be outside of the language in question, the problem of spurious ambiguity is typically greatly reduced for strings that are grammatical. Second, precision is a prerequisite to bidirectionality, enabling both parsing and generation. A system that cannot differentiate between strings that are in a language and those that are not cannot be used in this way.
The relatively low priority assigned to coverage in many HPSG-based computational linguistics projects is a natural corollary of the premium placed on precision, together with the Zipfian distribution of language. However, given that precision grammars are typically hand-crafted resources embodying substantial amounts of human effort, restricted coverage can at times be quite frustrating. In particular, the inability of a precision grammar to produce any analysis whatsoever for some proportion of inputs means an application built on such a grammar has two options:
1. Ignore such inputs 2. Rely upon evidence from a second analysis
component, resulting in a non-homogeneous data flow
Both options are viable in certain circumstances; however, neither option is very appealing. Having to depend on two different types of signal makes the design of downstream components more complicated, and depending on the rate of coverage gaps, ignoring unparseable inputs entirely could be disastrous. A better solution is needed.
This paper examines the issue of incomplete coverage in HPSG grammars, and explores several mitigation techniques, collectively dubbed robust parsing. Specifically, the English Resource Grammar (Flickinger, 2000) is augmented with
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
extended coverage by two techniques involving adding low-precision HPSG rules, and two techniques involving automatic constraint relaxation. The techniques are described in detail in Section 3, and the results are presented in Section 4. Finally, the effectiveness of the techniques is examined and commented on in Section 5.
A secondary contribution of this paper involves methodology for the evaluation of such robust parsing techniques, which is somewhat nonobvious due to the fact that gold standard evaluation data for the English Resource Grammar (henceforth ERG) is only readily available for incoverage inputs.
2 Related Work
The desire for increased coverage is at least as old as the field of computational linguistics. In the context of parsing with PCFGs, a time-honored solution is to assume that any two nonterminals can combine to form any other nonterminal (e.g. PP→VP+DET), but only with extremely low probability. This technique is referred to as smoothing. The result of smoothing is that any input string can be assigned a sentential analysis. A major drawback is the resulting explosion of ambiguity and processing cost: any binary tree with labels drawn from the nonterminals is a legal analysis for every input of the corresponding number of words. However, if the extra probability mass added when smoothing the grammar is distributed judiciously, and if suitable efficiency methods (such as pruning) are employed, it is often still possible to arrive at useful analyses in reasonable amounts of time.
The idea behind smoothing can, in principle, be applied to parsing with HPSG as well. Instead of having an enumerated set of nonterminal symbols, these grammars use feature structures to represent categories. The translation is roughly that any pair of adjacent sub-analyses with feature structures X and Y can license a larger analysis with an arbitrary feature structure Z. There are two major flies in this ointment. First, since HPSG does not posit a notion of probabilities,3 there is no native way to document these licentious edges as suspect. Second, the inventory of possible feature structures Z to be licensed in each case is not necessarily bounded: while in some grammars the collection
3Rather, ranking of competing analyses of ambiguous sentences is considered a separate operation.
of possible feature structures may be enumerable, it is likely to be very large, and in other grammars it may not even be finite, since recursive structures are possible. Nonetheless, this avenue of attack has been explored in some depth by Fouvry (2003b). Instead of licensing completely arbitrary feature structures Z, Fouvry gives a technique whereby existing grammar rules can be used even in circumstances where they formally cannot apply, to produce a more constrained inventory of “robust” subanalyses Z. Fouvry also addresses how to keep track of the degree of badness of these robust analyses, by associating to each of them a score indicating how many paths within daughter feature structures violated constraints on the rule. In a refinement, Fouvry (2003a) also shows how to use a more articulated score, where violating some constraints is considered more egregious than violating others.
Unfortunately, Fouvry does not give an empirical evaluation of his work. Indeed, he notes that the evaluation of robust parsing methods for HPSG is challenging, in part because gold standard evaluation data tends to be missing for inputs that cannot be parsed.4 Even with Fouvry’s careful limiting of the space of robust subanalyses, the search space is likely to increase to the point that parsing of longer sentences is unlikely to be practical within reasonable resource limits. However, Fouvry’s work may deserve the title of the most principled approach to robust HPSG parsing.
A number of considerably less general but arguably more practical approaches have also been deployed. Among these is the unknown word handling mechanism used in the ERG (Adolphs et al., 2008). A part-of-speech tagger with its own robustness mechanisms (including access to a larger lexicon, a smoothed HMM, and suffix analysis) provides guesses about the coarse-grained part of speech of each word. In cases where the ERG has no native lexical coverage for a word, the part of speech tags are used to trigger “generic” lexical entries. Although these generic lexical entries are somewhat underspecified relative to native entries in terms of information about the syntax and semantics of the word in question, their presence usually enables the grammar to find an analysis for sentences that would otherwise have been un-
4This is because the gold standard treebanks are typically prepared by manually disambiguating parse forests prepared with the grammar’s help (Oepen et al., 2004).
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
parseable.5
Another relatively common (although unglamorous) reason for failing to produce results in the ERG is resource exhaustion:6 no result was found within the amount of memory or time the user allowed, even though a result might well have been found if more resources were available to devote to the search. Dridan (2013) presents an effective extension of a familiar technique called supertagging to reduce resource consumption: a trigram sequence tagger is used to discard unlikely lexical categories, resulting in (depending on the threshold used) vastly reduced search space. Dridan reports not only an increase in coverage resulting from this reduced resource exhaustion, but also an increase in EDM F1 score and speed.7
Sometimes parsing fails due to reasons related to the structure of the input, rather than missing lexical entries or resource limits. Flickinger and Packard (2015) present a technique they dub bridging, in which two highly generic rules are added to an HPSG grammar in such a way as to allow any two constituents to combine, but yielding a highly constrained result feature structure which cannot then combine with anything other than more of its own type. Their technique is one of the ones evaluated in this paper, and is discussed in more depth in Section 3.4.2.
Zhang and Krieger (2011) show how to use treebanks built using an HPSG grammar to induce a PCFG that can produce analyses which look superficially like HPSG derivations. With suitable choice of parameters, such a PCFG can be quite robust. Unfortunately, the PCFG-produced analyses are frequently not fully coherent with respect to the constraints stipulated in the original HPSG grammar, preventing their use for extracting semantic information. Zhang et al. (In prep) subsequently proposed a robust unification technique which, combined with Zhang and Krieger’s parser, allows nearly-coherent semantic analyses to be built using the original HPSG grammar, guided by the PCFG-produced trees. This technique is also
5This mechanism is enabled in all of the experiments described in this paper.
6The rate of occurrence is highly dependent on sentence length. Allowing up to 1GB of memory and 60 seconds of time on Wikipedia text with a 2.6GHz Opteron 6344 CPU, resource exhaustion accounts for about 6% of inputs but more than 50% of parse failures.
7Dridan’s mechanism is not used in our experiments, except with the pacnv system, where resource exhaustion without it was judged too excessive for useful results.
one of the ones evaluated in this paper, and is discussed in more depth in Section 3.4.4.
The next section will give more details about the components used, techniques evaluated, and experiments performed in this comparison of robust parsing methods.
3 Methods
3.1 English Resource Grammar
Development on the ERG began in 1994 at Stanford as a component of the German-English machine translation project Verbmobil, with the initial focus on parsing and generating relatively short English sentences and phrases in spoken dialogues about meeting scheduling. This HPSG grammar has been under continuous expansion since that rather modest beginning, and has been used in research and/or commercial applications in automated customer email response, online education for teaching English grammar as well as predicate logic, information extraction over the English Wikipedia, and shared tasks in simulated robot control and in grammatical error detection in scientific writing. Alongside the grammar, which can by now successfully analyze at least 9 in every 10 sentences across a wide variety of text domains (Flickinger, 2011), a manually constructed treebank of some 1.5 million words parsed by the ERG has been built both for regression testing and as training material for automatic parse selection. This resource, dubbed the Redwoods treebank (Oepen et al., 2004), stores for each parsed sentence both the correct syntactic derivation (if available in the parse forest) and the associated meaning representation expressed in Minimal Recursion Semantics, henceforth MRS (Copestake et al., 2005). Of the roughly 10% of sentences in an arbitrary English text that the ERG fails to parse, half typically fail due to the parser hitting resource limits (due to too much ambiguity present in exhaustive parsing), and half fail due to remaining linguistic shortcomings of the ERG for phenomena in the long tail of less frequent constructions in English.
3.2 Evaluation methodology
It seems clear that the underlying goal of robustness techniques like the ones explored herein is to provide a larger quantity of useful analyses to a downstream process than could previously be produced. To that end, the most interesting quantity to
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
evaluate would be improvement or regression on some collection of tasks for which the ERG’s parsing capability is an important component, i.e. an extrinsic evaluation. To do a good job of extrinsic evaluation, however, two things are required: first, a large enough annotated dataset to acquire meaningful statistics for a particular task, and second, a sufficiently varied collection of tasks to exercise all interesting aspects of the analyses. Unfortunately such work is beyond the scope of this paper. A lesser attempt at extrinsic evaluation seems tempting, but prone to errors.
Therefore, in this work we content ourselves with intrinsic evaluations of robust parsing techniques. Intrinsic evaluation is not without its own difficulties. The primary problem is that the sentences of greatest interest for evaluation, i.e. the ones that the grammar cannot by itself produce an analysis for, are exactly the ones for which no gold standard annotations are available.8 This is because the ERG’s gold standard annotations are built by manually disambiguating the forest of (often millions of) analyses licensed by the grammar.
3.3 Datasets
There are two ways to solve this problem. First, the missing gold standard data can be fabricated by a labor-intensive process we playfully dub “alchemy”. This involves making minor edits to the unparseable sentence and/or the grammar (for instance, declaring a new multi-word lexeme in the grammar, or fixing a typo in the input) in order to produce the desired analysis.9 Once the desired “gold” analysis has been recorded, the edits to the sentence and grammar can be discarded. The alchemy45 dataset used in our evaluations was produced this way, from a collection of Wall Street Journal sentences that the 1212 release of the ERG was unable to analyze within reasonable resource limits (8 of the 45 items were actually parseable with more generous resource limits; the rest were properly outside of the grammar’s coverage). Our ws13r dataset was produced in the same way, from ERG-unparseable Wikipedia sentences, but in this case, relative to a more recent grammar version.
8In fact, they are a subset; there are some inputs that the grammar is capable of analyzing, but not with the intended interpretation.
9It would also of course be possible to manually type out the desired analysis, but this is even more labor intensive and extremely error-prone, given the degree of connectivity and detail recorded in MRS.
The second approach is to perform robust parsing with respect to a slightly out-of-date version of the ERG, and evaluate the results only against inputs that were out of scope in that version but are in scope in a more recent grammar version. Our semcor and wsj00ab datasets follow this methodology; they consist of items from the Brown Corpus (Kucera and Francis, 1967) and Wall Street Journal, respectively, that had no gold standard analysis in the 1214 release of the grammar, but do have gold standard analyses in the current SVN trunk version of the grammar. This approach is much cheaper in terms of manual labor. However, it has the disadvantage (shared with our ws13r dataset) that comparison between system results and the gold standard necessarily involves two distinct grammar versions, and hence differences can be expected even for a perfect parser. We attempt to minimize the effect of these differences for the specific grammar pair of 1214 vs trunk by taking the following measures.
1. Rather than only judging exact matches, we give partial credit for partial MRS match, using the Elementary Dependency Matching (EDM) evaluation metric presented by Zhang et al. (In prep). 2. Internal properties of semantic variables, such as number or tense, are not included in evaluation, since there were significant changes in how they are computed between 1214 and trunk. 3. Information-structure predicates (bearing the suffix d rel) are ignored, since these were relocated in trunk. 4. Only the lemma field within predicate names is compared, as many subsense fields were changed. 5. Some slippage is allowed in aligning character positions recorded within each MRS.
The datasets used for evaluation are as follows:
Dataset Items #words #items In 1212? Version alchemy45 45 28.5 no parse 1212
ws13r 207 27.7 no parse trunk semcor 241 24.8 no gold10 trunk wsj00ab 76 25.6 no gold trunk
3.4 Parsing systems
We compared the performance of five approaches to increasing the robustness of parsing with the ERG. These techniques are described below.
10Some semcor and wsj00ab items are parseable under 1214, but not with the intended analysis.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
3.4.1 Baseline
The ERG by itself cannot, by definition, provide analyses for any inputs that it considers ungrammatical.11 However, in many cases analyses can be found for inputs which exhausted default resource limits, simply by extending those resource limits. In this sense, it is possible to consider the ERG itself as a baseline contender for robust parsing: by assigning more generous resource limits, some previously unparseable inputs receive analyses. The quality of these analyses (as measured for instance by EDM precision) can in general be expected to be in line with the quality of other native ERG analyses, but their quantity (as measured by EDM recall) will be limited. The system referred to as baseline in the results below is the ERG with no extensions, but with resource limits of 8GB of RAM and unlimited time per item.
3.4.2 Bridging
This system, first described by Flickinger and Packard (2015), is one of two “self-help” methods, meaning it is implemented within the same formalism used by the rest of the grammar. The goal of the method is to allow the construction of a spanning analysis from analyses of shorter spans within an input that normally would not be able to combine with each other. For an input like the following, the ERG is normally unable to assemble the pieces into a coherent whole:
(1) The dog, come quickly, the dog bit me!
Bridging allows an NP analysis of “The dog,” an imperative analysis of “come quickly,” and a declarative analysis of “the dog bit me!” to all be glued together into a single analysis that, while failing to say anything interesting about the relationships between those three parts, still makes the analyses of those parts accessible. This is implemented by two rules:
1. a unary rule promoting any ordinary analysis into a “bridge” analysis with the same span 2. a binary rule allowing the combination of two bridge analyses, where the lefthand daughter must itself be the unary bridging rule. Although the system could in principle also be implemented as a single binary rule, the above design has the advantage of greatly reducing the cost of
11Though some applications of the ERG for language learning make use of an extension to the grammar consisting of a set of mal-rules described in Bender et al. (2004), which explicitly license certain ungrammatical structures.
parsing, since all of the results of the unary rule for a given span are combinatorially equivalent (i.e. they pack: see Oepen and Carroll (2000)), leading to just a single application of the binary rule per span, rather than one for each analysis of the left-hand side. Finally, to mitigate the possibility of lexical gaps not covered by unknown word handling, the implementation also posits a “bridge” lexical analysis of every input token.
The wealth of robust analyses spawned by the bridging mechanism gives rise to an important question: which one is the most useful? This problem of ambiguity is of course isomorphic to the problem of ambiguity experienced during ordinary parsing, and is amenable to the same solution, namely a statistical ranking model trained on a manually annotated treebank. Unfortunately, whereas the ERG comes with curated treebanks of roughly 100,000 trees falling within the scope of its ordinary grammatical rules, there are only very minimal curated treebanks of bridged analyses. As a result, the statistical model we use to select a parse contains no preferences regarding the bridging rules. With future research, it may be possible to significantly improve the performance of this disambiguation task.
Formally, bridging ensures that any input is analyzable. However, even with aggressive packing techniques, the sheer ambiguity presented by the bridging rules makes the unpacking problem quite challenging. In fact, the parser used in these experiments (namely ACE 0.9.24: sweaglesw.org/linguistics/ace/) was not able to solve the unpacking problem for a significant number of inputs, resulting in somewhat disappointing actual coverage.
3.4.3 Pacman
One clear disadvantage of the bridging approach is that once a bridged analysis has been invoked for a portion of an input sentence, any larger portion of the sentence containing that bridged element must also be bridged (to keep the approach at all tractable), and hence fails to produce informative semantics for the larger portions. So we explored a second “self-help” method, again adding robustness rules to the grammar, but instead of categoryagnostic bridging rules, we provided rules that combine certain frequently appearing phrases such as NPs or VPs with an arbitrary word or phrase to their immediate left or right. The guiding intuition is that failure to parse a sentence often results
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
from some minor intrusion of an unexpected word or phrase, where if the parser could just skip over that obstacle, it could proceed to construct a quite serviceable derivation for the sentence. For example, the standard ERG will not succeed in analyzing the sentence They might try to buy a cheap car, I suppose, or an old truck because of the phrase I suppose appearing in the middle of a coordination of two NPs. But a rule that consumed I suppose along with the preceding NP a cheap car to produce an NP covering both phrases would then enable a usable analysis of the full sentence.
We term the use of such cheerfully greedy rules the “Pacman” approach, and for the present experiment, we limit ourselves to just two rules, one for building robust nominal phrases, and one for verb phrases; hence the shorthand label pacnv. The resulting grammar should be considerably more robust than the regular ERG, and possibly more successful in preserving meaning content than with bridging rules, but it cannot aspire to full coverage, since it crucially depends on finding a host nominal or verbal phrase next to the site of any parsing obstacle. We could expand the inventory of host phrase types to close this remaining coverage gap, but experiments to date indicate that increased ambiguity can quickly make parsing with a larger set of Pacman rules intractable.
3.4.4 PCFG approximation
Our third approach, referred to as csaw in the results below, is a straightforward reimplementation combining the methods of the JigSaw parser of Zhang and Krieger (2011) and the robust unification technique of Zhang et al. (In prep). The idea of combining these techniques belongs to the authors of those papers, and the details of how the systems work are given there as well. A summary will suffice to give readers the basic idea. First, a collection of derivation trees consistent with the underlying precision grammar is procured for use as training data. These trees can come either from gold standard treebanks or from parsing. The internal nodes of the trees are labeled with the rule name licensing the corresponding constituent (for example, the head specifier rule). Leaves correspond to lexemes, and are labeled with the name of the lexical type to which the lexeme belongs. In addition, some modifications are made to the shape of the tree: most lexical rules are collapsed onto the yield of the tree, and punctuation is split into binary structures. Finally, the trees can op-
tionally be decorated with grandparent information to a configurable depth. A PCFG is induced by maximum likelihood estimation from the decorated trees, with no smoothing, lexicalization, hierarchical splitting, or other enhancements (although such techniques would likely be fruitful avenues for future experiments).
The resulting PCFG is used in conjunction with a straightforward CKY parser to analyze arbitrary inputs. Depending on the level of grandparenting used and the size of the training set, the speed and coverage of this system can be high or low. For the purposes of this paper, two configurations are used: csaw-tb, which is an ungrandparented PCFG trained exclusively on hand-annotated gold-standard trees, and csaw-ww, which is a doubly-grandparented PCFG trained exclusively on automatically-disambiguated Wikipedia sentences. The two configurations are compared below:
System Non-terminals Rules Training set csaw-tb 236 ∼36K ∼100K csaw-ww 155042 ∼5M ∼50M
Since the terminal symbols of these grammars are lexical types combined with sequences of lexical rules, these grammars are not directly useable for parsing plain text. Rather, they are dependent upon the preprocessing environment and lexical parsing capabilities of the ERG to prepare sequences or lattices of terminal symbols from which PCFG parsing can proceed.12
The decorations and shape modifications performed prior to inducing the PCFG are reverseable. Analyses found by the PCFG grammar can easily be converted13 into trees that look superficially very similar to the derivation trees licensed by the precision grammar. In some cases, the sequence of rules and lexemes stipulated by these pseudo-derivations are in fact compatible with the unification constraints that define those rules and lexemes. Under these circumstances, the precision grammar can be used to “replay” the derivation and a coherent semantic analysis can be read out of the resulting feature structure, as in the case of ordinary parsing. Unfortunately, this can only happen when a fully consistent derivation exists, i.e. when the precision grammar was already
12This tight coupling facilitates semantic recovery, and is a major reason that more powerful “off-the-shelf” PCFG parsers were not used.
13Grandparent information must be removed, collapsed chains of lexical rules separated, and binary punctuation converted back into the ERG’s lexical rule analysis.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
capable of analyzing the input.
Zhang et al. (In prep) had the key insight that the unification constraints that these pseudoderivations violate are usually in parts of the feature structure geometry that are isolated from the portion used for semantic composition. As a result, robust unification techniques such as the one they present, or older ones such as that of (Fouvry, 2003b), can be used to replay the pseudoderivation and produce a feature structure that, while unreliable in terms of its syntactic properties, is largely consistent in terms of the semantic analysis it embodies. The csaw systems do just this, resulting in a (possibly somewhat incoherent) semantic analysis for nearly any input.14
3.4.5 A hybrid approach
The csaw approaches are “all-or-nothing”, in the sense that if there is any portion at all of input that the precision grammar is unable to analyze, the entirety of the syntactic constraints must be relaxed in favor of the PCFG. This feels a bit like throwing the baby out with the bath water: the precision grammar may have a great deal of useful expertise to contribute to the analysis of large portions of the sentence, even if it is overly prescriptive or incomplete in its treatment of some particular detail. The hybrid-tb and hybrid-ww techniques attempt to exploit the best of both worlds. This is achieved by first allowing the corresponding csaw-based system to propose a single complete analysis, and then allowing the rules of the precision grammar to find additional ways to use the pieces of that robust analysis. Specifically, the constituents postulated by csaw are converted into chart edges and allowed to participate in chart parsing along with the rest of the edges licensed by the grammar. This allows a robust analysis of some particularly troublesome construction to combine with all other possible analyses of the rest of the sentence (including portions nested within the robust analysis). The best analysis is subsequently chosen by the statistical parse ranking model used when performing ordinary parsing with the precision grammar.
14Some inputs may be unparseable by the PCFG, of course. The robust semantic readout process can also fail in the empirically very rare case that robust unification produces a cyclic structure in an infelicitous location, such as the list of semantic predications. The latter occurred just once for us.
4 Results
Table 1 shows the coverage of each technique on each dataset. As expected, the baseline ERG system had the lowest coverage of any of the techniques for the alchemy45 and wsj00ab datasets. The pacnv system yielded even lower coverage than the baseline system for ws13r and semcor, although it showed promise on the alchemy45 where native coverage is very low. The reason that coverage for the bridging and pacnv systems is relatively low is that these techniques introduce very challenging ambiguity management problems for the parser, and as a result the parser frequently exhausts the resource allocation (which was the same as used for the baseline method). The PCFG-based methods all showed higher coverage than the ERG, across all datasets. The hybrid-ww method achieved the highest coverage on all datasets except semcor, where it was edged out by csaw-ww and hybrid-tb.
Table 2 shows parsing speeds. While the speeds may seem slow and the scores may seem low, it is important to bear in mind that these datasets are selected specifically to be hard to parse. As the parsing was performed on a cluster where CPU speed and load varied somewhat from experiment to experiment, the speed measurements are only comparable on a broad scale. Nonetheless, we clearly see that csaw-tb is by far the fastest system: its grammar is simpler, and the expense of unification-based parsing is entirely avoided.
The slowest systems are csaw-ww and hybrid-ww; the former because the corresponding PCFG is very large, and the latter because it includes the former as a component, and has to perform unification-based parsing as well. These patterns held across all four datasets.
Table 2 also shows EDM F1 scores, a finegrained evaluation of the semantic dependencies encoded in the MRSes recovered by each parser. Three datasets pattern the same: ws13r, semcor, and wsj00ab. On these datasets, hybrid-ww produced the highest scores, while bridging, pacnv, and csaw-tb earned the lowest marks, edged out even by baseline.
On alchemy45, the story is different: baseline finds no competitors for the lowest position, while csaw-ww performs slightly better than hybrid-ww, taking the top score. The highly efficient csaw-tb also performs better on
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Method alchemy45 ws13r semcor wsj00ab baseline 17.78% 61.35% 88.74% 84.21% bridging 66.67% 74.40% 90.91% 90.79% pacnv+ut 53.33% 54.11% 87.45% 85.53% csaw-tb 97.78% 77.29% 98.27% 98.68% csaw-ww 100.00% 83.09% 100.00% 100.00%
hybrid-tb 97.78% 88.41% 100.00% 98.68% hybrid-ww 100.00% 90.82% 99.13% 100.00%
Table 1: Coverage for each technique.
Method alchemy45 ws13r semcor wsj00ab F1 T F1 T F1 T F1 T
baseline 28.83 44.1 52.28 31.4 79.60 7.7 72.78 10.6 bridging 42.07 41.2 44.31 48.3 69.61 22.4 66.11 21.9 pacnv+ut 43.85 29.0 42.27 89.7 72.70 16.6 66.16 27.6 csaw-tb 68.51 2.1 48.87 1.0 67.81 0.6 67.26 0.7 csaw-ww 77.11 247.0 60.98 224.3 78.51 135.8 74.48 132.2 hybrid-tb 69.76 39.2 63.73 20.4 78.76 13.4 76.17 14.1 hybrid-ww 75.56 219.2 68.47 228.2 81.52 122.0 78.61 119.8
Table 2: EDM F1 and wallclock seconds per item for each technique.
this dataset than it did on the others.
5 Discussion
The dichotomy between performance on alchemy45 and the other three datasets is interesting. There are two things about that dataset that are unique: first, it is the only dataset of the four that was annotated against the 1214 version of the ERG instead of against the trunk version. Second, it has by far the lowest proportion of sentences that the ERG can analyze unassisted. From that perspective, it is the best indicator of performance on inputs where robust techniques are needed the most. However, it is also the smallest of the four datasets, so results may not be as reliable.
One clear result is that the PCFG-based systems are capable of greatly enhancing the coverage of the ERG, while producing analyses whose quality is better than the other robust systems evaluated. While the speed of the best-performing csaw-ww and hybrid-ww methods is somewhat uninspiring, we console ourselves by noting that the method is at least tractable, and the performance is probably acceptable for some applications where coverage and accuracy are at a premium while time is not. When the utmost accuracy is not required for robust parses, the hybrid-tb system is not noticeably slower than
the baseline, yet has a much higher coverage, with scores somewhat lower than hybrid-ww but still much better than baseline for the alchemy45 dataset. The fact that the overall EDM F1 score for the hybrid-tb method was slightly lower than the very respectable baseline score on the semcor dataset indicates that successful deployment of this technique may be predicated upon restricting its use to cases where the ERG in isolation has already failed to produce an analysis.
In the present work, the parse ranking model is not adapted to the novel situations in which it is used, and hence likely does not perform as well as it could. New techniques are needed to train the ranking model to make informed decisions about these situations. Although coverage would be unaffected, accuracy could potentially be improved. Also, evaluating the difference in performance on data strictly outside of the coverage of the underlying precision grammar compared to data in scope would be profitable, as would exploring techniques for managing the decision about when to deploy robustness.
While it remains to be seen whether the promising coverage gains and accuracy figures reported here can be borne out in extrinsic evaluation, we believe there is reason for optimism, as downstream applications may be able to mitigate coverage issues using these techniques.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths:  Well-written.  - Weaknesses:  Although the title and abstract of the paper suggest that robust parsing methods for HPSG are being compared, the actual comparison is limited to only a few techniques applied to a single grammar, the ERG (where in the past the  choice has been made to create a treebank for only those sentences that are in the coverage of the grammar). Since the ERG is quite idiosyncratic in this respect, I fear that the paper is not interesting for researchers working in other precision grammar frameworks.  The paper lacks comparison with robustness techniques that are routinely applied for systems based on other precision grammars such as various systems based on CCG, LFG, the Alpage system for French, Alpino for Dutch and there is probably more. In the same spirit, there is a reference for supertagging to Dridan 2013 which is about supertagging for ERG whereas supertagging for other precision grammar systems has been proposed at least a decade earlier.  The paper lacks enough detail to make the results replicable. Not only are various details not spelled out (e.g. what are those limits on resource allocation), but perhaps more importantly, for some of the techniques that are being compared (eg the robust unification), and for the actual evaluation metric, the paper refers to another paper that is still in preparation.  The actual results of the various techniques are somewhat disappointing. With the exception of the csaw-tb method, the resulting parsing speed is extreme - sometimes much slower than the baseline method - where the baseline method is a method in which the standard resource limitations do not apply. The csaw-tb method is faster but not very accurate, and in any case it is not a method introduced in this paper but an existing PCFG approximation technique.  It would be (more) interesting to have an idea of the results on a representative dataset (consisting of both sentences that are in the coverage of the grammar and those that are not). In that case, a comparison with the  real  baseline system (ERG with standard settings) could be obtained.  Methodological issue: the datasets semcor and wsj00ab consist of sentences which an older version of ERG could not parse, but a newer version could. For this reason, the problems in these two datasets are clearly very much biased. It is no suprise therefore that the various techniques obtain much better results on those datasets. But to this reviewer, those results are somewhat meaningless.   minor:  EDM is used before explained   reverseability   - General Discussion: