Universal Dependencies (UD) provides a cross-linguistically uniform syntactic representation, with the aim of advancing multilingual applications of parsing and natural language understanding. Reddy et al. (2016) recently developed a semantic interface for (English) Stanford Dependencies, based on the lambda calculus. In this work, we introduce UDEPLAMBDA, a similar semantic interface for UD, which allows mapping natural language to logical forms in an almost language-independent framework. We evaluate our approach on semantic parsing for the task of question answering against Freebase. To facilitate multilingual evaluation, we provide German and Spanish translations of the WebQuestions and GraphQuestions datasets. Results show that UDEPLAMBDA outperforms strong baselines across languages and datasets. For English, it achieves the strongest result to date on GraphQuestions, with competitive results on WebQuestions.
1 Introduction
The Universal Dependencies (UD) initiative seeks to develop cross-linguistically consistent annotation guidelines as well as a large number of uniformly annotated treebanks for many languages.1 Such resources could advance multilingual applications of parsing, improve comparability of evaluation results, enable cross-lingual learning, and more generally support natural language understanding.
Seeking to exploit the benefits of UD for natural language understanding, we introduce UDEPLAMBDA, a semantic interface for UD that maps
1http://www.universaldependencies.org/.
natural language to logical forms, representing underlying predicate-argument structures, in an almost language-independent manner. Our framework is based on DEPLAMBDA (Reddy et al., 2016) a recently developed method that converts English Stanford Dependencies (SD) to logical forms. The conversion process is illustrated in Figure 1 and discussed in more detail in Section 2. Whereas DEPLAMBDA works only for English, UDEPLAMBDA applies to any language for which UD annotations are available.2 In this paper, we describe the rationale behind UDEPLAMBDA and highlight important differences from DEPLAMBDA, some of which stem from the different treatment of various linguistic constructions in UD.
Our experiments focus on semantic parsing as a testbed for evaluating the framework’s multilingual appeal. We address the task of learning to map natural language to machine interpretable formal meaning representations, specifically retrieving answers to questions from Freebase. To facilitate multilingual evaluation, we provide translations of the English WebQuestions (Berant et al., 2013) and GraphQuestions (Su et al., 2016) datasets to German and Spanish. We demonstrate that UDEPLAMBDA can be used to derive logical forms for these languages using a minimal amount of language-specific knowledge. Aside from developing the first multilingual semantic parsing tool for Freebase, we also experimentally show that UDEPLAMBDA outperforms strong baselines across languages and datasets. For English, it achieves the strongest result to date on GraphQuestions, with competitive results on WebQuestions. Beyond semantic parsing, we believe that the logical forms produced by our framework will be of use in various natural understanding tasks including entailment (Beltagy et al., 2016), text-based question
2As of v1.3, UD annotations are available for 47 languages.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
answering (Lewis and Steedman, 2013), sentence simplification (Narayan and Gardent, 2014), summarization (Liu et al., 2015), paraphrasing (Pavlick et al., 2015), and relation extraction (Rocktäschel et al., 2015). Our implementation and translated datasets will be made publicly available.
2 DEPLAMBDA
Before describing UDEPLAMBDA, we provide an overview of DEPLAMBDA (Reddy et al., 2016) on which our approach is based. DEPLAMBDA converts a dependency tree to its logical form in three steps: binarization, substitution, and composition, each of which is briefly outlined below.
Binarization A dependency tree is first mapped to a Lisp-style s-expression indicating the order of semantic composition. Figure 1(b) shows the s-expression for the sentence Disney won an Oscar for the movie Frozen, derived from the dependency tree in Figure 1(a). Here, the sub-expression (dobj won (det Oscar an)) indicates that the logical form of the phrase won an Oscar is derived by composing the logical form of the label dobj with the logical form of the word won and the logical form of the phrase an Oscar, derived analogously.
An obliqueness hierarchy is employed to impose a strict ordering on the modifiers to each head in the dependency tree. As an example, won has three modifiers in Figure 1(a), which according to the obliqueness hierarchy are composed in the order dobj> nmod> nsubj. In constructions like coordination, this ordering is crucial to arrive at the correct semantics (see Section 3.3).
Substitution Each symbol in the s-expressions is substituted for a lambda expression encoding its semantics. Words and dependency labels are assigned different types of expressions. In general, words have expressions of the following kind: ENTITY ⇒ λx.word(xa); e.g. Oscar⇒ λx.Oscar(xa) EVENT ⇒ λx.word(xe); e.g. won⇒ λx.won(xe) FUNCTIONAL⇒ λx.TRUE; e.g. an⇒ λx.TRUE
Here, the subscripts ·a and ·e denote the types of individuals (Ind) and events (Event), respectively, whereas x denotes a paired variable (xa,xe) of type Ind×Event. Roughly speaking, proper nouns and adjectives invoke ENTITY expressions, verbs and adverbs invoke EVENT expressions, and common nouns invoke both ENTITY and EVENT expressions (see Section 3.3), while remaining words invoke FUNCTIONAL expressions. As in DEPLAMBDA,
we enforce the constraint that every s-expression is of the type η = Ind×Event → Bool, which simplifies the type system considerably.
Expressions for dependency labels glue the semantics of heads and modifiers to articulate predicate-argument structure. These expressions in general take one of the following forms: COPY ⇒ λ f gx.∃y. f (x)∧g(y)∧ rel(x,y) e.g. nsubj, dobj, nmod, advmod INVERT ⇒ λ f gx.∃y. f (x)∧g(y)∧ reli(y,x) e.g. amod, acl MERGE ⇒ λ f gx. f (x)∧g(x) e.g. compound, appos, amod, acl HEAD ⇒ λ f gx. f (x) e.g. case, punct, aux, mark .
As an example of COPY, consider the lambda expression for dobj in (dobj won (det Oscar an)): λ f gx.∃y. f (x)∧ g(y)∧ arg2(xe,ya). This expression takes two functions f and g as input, where f represents the logical form of won and g represents the logical form of an Oscar. The predicateargument structure arg2(xe,ya) indicates that the arg2 of the event xe, i.e. won, is the individual ya, i.e. the entity Oscar. Since arg2(xe,ya) mimics the dependency structure dobj(won, Oscar), we refer to the expression kind evoked by dobj as COPY.
Expressions that invert the dependency direction are referred to as INVERT (e.g. amod in running horse); expressions that merge two subexpressions without introducing any relation predicates are referred to as MERGE (e.g. compound in movie Frozen); and expressions that simply return the parent expression semantics are referred to as HEAD
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
(e.g. case in for Frozen). While this generalization applies to most dependency labels, several labels take a different logical form not listed here, some of which are discussed in Section 3.3. Sometimes the mapping of dependency label to lambda expression may depend on surrounding part-of-speech tags or dependency labels. For example, amod acts as INVERT when the modifier is a verb (e.g. in running horse), and as MERGE when the modifier is an adjective (e.g. in beautiful horse).3
Composition The final logical form is computed by beta-reduction, treating expressions of the form (f x y) as the function f applied to the arguments x and y. For example, (dobj won (det Oscar an)) results in λx.∃z.won(xe)∧Oscar(za)∧ arg2(xe,za) when the expression for dobj is applied to those for won and (det Oscar an). Figure 1(c) shows the logical form for the s-expression in Figure 1(b).
3 UDEPLAMBDA
We now introduce UDEPLAMBDA, a semantic interface for Universal Dependencies.4 Whereas DEPLAMBDA only applies to English Stanford Dependencies, UDEPLAMBDA takes advantage of the cross-lingual nature of UD to facilitate an (almost) language independent semantic interface. This is accomplished by restricting the binarization, substitution, and composition steps described above to rely solely on information encoded in the UD representation. Importantly, UDEPLAMBDA is designed to not rely on lexical forms in a language to assign lambda expressions, but only on information contained in dependency labels and postags.
However, some linguistic phenomena are language specific (e.g. pronoun-dropping) or meaning specific (e.g. every and the in English have very different semantics, despite being both determiners) and are not encoded in the UD schema. Furthermore, some cross-linguistic phenomena, such as long-distance dependencies, are not part of the core UD representation. To circumvent this limitation, a simple enhancement step enriches the original UD representation before binarization takes place (Section 3.1). This step adds to the dependency tree missing syntactic information and long-distance dependencies, thereby creating a
3We use Tregex (Levy and Andrew, 2006) for substitution mappings and Cornell SPF (Artzi, 2013) as the lambdacalculus implementation. For example, in running horse, the tregex /label:amod/=target < /postag:verb/ matches amod to its INVERT expression λ f gx.∃y. f (x)∧g(y)∧ amodi(ye,xa).
4In what follows, all references to UD are to UD v1.3.
graph. Whereas DEPLAMBDA is not able to handle graph-structured input, UDEPLAMBDA is designed to work directly with dependency graphs (Section 3.2). Finally, the representation of several linguistic constructions differ between UD and SD, which requires different handling in the semantic interface (Section 3.3).
3.1 Enhancement
Both Schuster and Manning (2016) and Nivre et al. (2016) note the necessity of an enhanced UD representation to enable semantic applications. However, such enhancements are currently only available for a subset of languages in UD. Instead, we rely on a small number of enhancements for our main application—semantic parsing for questionanswering—with the hope that this step can be replaced by an enhanced UD representation in the future. Specifically, we define three kinds of enhancements: (1) long-distance dependencies; (2) types of coordination; and (3) refined question word tags.
First, we identify long-distance dependencies in relative clauses and control constructions. We follow Schuster and Manning (2016) and find these using the labels acl (relative) and xcomp (control). Figure 2(a) shows the long-distance dependency in the sentence Anna wants to marry Kristoff. Here, marry is provided with its missing nsubj (dashed arc). Second, UD conflates all coordinating constructions to a single dependency label, conj. To obtain the correct coordination scope, we refine conj to conj:verb, conj:vp, conj:sentence, conj:np, and conj:adj, similar to Reddy et al. (2016). Finally, unlike the PTB tags (Marcus et al., 1993) used by SD, the UD part-of-speech tags do
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
not distinguish question words. Since these are crucial to question-answering, we use a small lexicon to refine the tags for determiners (DET), adverbs (ADV) and pronouns (PRON) to DET:WH, ADV:WH and PRON:WH, respectively. Specifically, we use a list of 12 (English), 14 (Spanish) and 35 (German) words, respectively. This is the only part of UDEPLAMBDA that relies on language-specific information. We hope that, as the coverage of morphological features in UD improves, this refinement can be replaced by relying on morphological features, such as the interrogative feature (INT).
3.2 Graph Structures and BIND
To handle graph structures that may result from the enhancement step, such as those in Figure 2(a), we propose a variable-binding mechanism that differs from that of DEPLAMBDA. First, each longdistance dependency is split into independent arcs as shown in Figure 2(b). Here, Ω is a placeholder for the subject of marry, which in turn corresponds to Anna as indicated by the binding of Ω via the pseudo-label BIND. We treat BIND like an ordinary dependency label with semantics MERGE and process the resulting tree as usual, via the s-expression: (nsubj (xcomp wants (nsubj (mark
(dobj marry Kristoff) to) Ω) (BIND Anna Ω)) ,
with the lambda-expression substitutions: wants, marry ∈ EVENT; to ∈ FUNCTIONAL; Anna, Kristoff ∈ ENTITY; mark ∈ HEAD; BIND ∈ MERGE; xcomp = λ f gx.∃y. f (x)∧g(y)∧xcomp(xe,ye) .
These substitutions are based solely on unlexicalized context. For example, the part-of-speech tag PROPN of Anna invokes an ENTITY expression.
The placeholder Ω has semantics λx.EQ(x,ω), where EQ(u,ω) is true iff u and ω are equal (have the same denotation), which unifies the subject variable of wants with the subject variable of marry.
After substitution and composition, we get: λz.∃xywv.wants(ze)∧Anna(xa)∧ arg1(ze,xa)∧ EQ(x,ω) ∧marry(ye)∧xcomp(ze,ye)∧ arg1(ye,va)∧ EQ(v,ω) ∧ Kristoff(wa)∧ arg2(ye,wa) ,
This expression may be simplified further by replacing all occurrences of v with x and removing the unification predicates EQ, which results in: λz.∃xyw.wants(ze)∧Anna(xa)∧ arg1(ze,xa) ∧marry(ye)∧xcomp(ze,ye)∧ arg1(ye,xa) ∧ Kristoff(wa)∧ arg2(ye,wa) .
This expression encodes the fact that Anna is the arg1 of the marry event, as desired. DEPLAMBDA, in contrast, cannot handle graph-structured input,
since it lacks a principled way of generating sexpressions from graphs. Even given the above s-expression, BIND in DEPLAMBDA is defined in a way such that the composition fails to unify v and x, which is crucial for the correct semantics. Moreover, the definition of BIND in DEPLAMBDA does not have a formal interpretation within the lambda calculus, unlike ours.
3.3 Linguistic Constructions
Below, we highlight the most pertinent differences between UDEPLAMBDA and DEPLAMBDA, stemming from the different treatment of various linguistic constructions in UD versus SD.
Prepositional Phrases UD uses a content-head analysis, in contrast to SD, which treats function words as heads of prepositional phrases, Accordingly, the s-expression for the phrase president in 2009 is (nmod president (case 2009 in)) in UDEPLAMBDA and (prep president (pobj in 2009)) in DEPLAMBDA. To achieve the desired semantics, λx.∃y.president(xa)∧president event(xe)∧
arg1(xe,xa)∧2009(ya)∧prep.in(xe,ya) ,
DEPLAMBDA relies on an intermediate logical form that requires some post-processing, whereas UDEPLAMBDA obtains the desired logical form directly through the following entries: in ∈ FUNCTIONAL; 2009 ∈ ENTITY; case ∈ HEAD; president = λx.president(xa)∧president event(xe) ∧arg1(xe,xa) ; nmod = λ f gx.∃y. f (x)∧g(y)∧nmod.in(xe,ya) .
Other nmod constructions, such as possessives (nmod:poss), temporal modifiers (nmod:tmod) and adverbial modifiers (nmod:npmod), are handled similarly. Note how the common noun president, evokes both entity and event predicates above.
Passives DEPLAMBDA gives special treatment to passive verbs, identified by the fine-grained partof-speech tags in the PTB tag together with dependency context. For example, An Oscar was won is analyzed as λx.won.pass(xe)∧Oscar(ya)∧ arg1(xe,ya), where won.pass represents a passive event. However, UD does not distinguish between active and passive forms.5 While the labels nsubjpass or auxpass indicate passive constructions, such clues are sometimes missing, such as in reduced relatives. We therefore opt to not have separate entries for passives, but aim to produce identical logical forms for active and passive forms when
5UD encodes voice as a morphological feature, but most syntactic analyzers do not produce this information yet.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
language target people
x e1 y e2 Ghana speak .arg2 speak .arg1 people .arg1 people .nmod.in
typ e
typ e
(a) English
sprache target
x e1 Ghana gesprochen
.arg2 gesprochen .nmod.in
typ e
(b) German
lengua target
x e1 Ghana lengua .arg1 lengua .nmod.de
typ e
(c) Spanish
language .human language
target
x m Ghana location.country
.official language.2 location.country .official language.1
typ e
(d) Freebase
Figure 3: The ungrounded graphs for What language do the people in Ghana speak?, Welche Sprache wird in Ghana gesprochen? and Cuál es la lengua de Ghana?, and the corresponding grounded graph.
possible (for example, by treating nsubjpass as direct object). With the following entries, won ∈ EVENT; an, was ∈ FUNCTIONAL; auxpass ∈ HEAD; nsubjpass = λ f gx.∃y. f (x)∧g(y)∧ arg2(xe,ya) ,
the lambda expression for An Oscar was won becomes λx.won(xe)∧Oscar(ya)∧arg2(xe,ya), identical to that of its active form. However, not having a special entry for passive verbs may have undesirable side-effects. For example, in the reducedrelative construction Pixar claimed the Oscar won for Frozen, the phrase the Oscar won ... will receive the semantics λx.Oscar(ya)∧won(xe)∧ arg1(xe,ya), which differs from that of an Oscar was won. We leave it to the target application to disambiguate the interpretation in such cases.
Long-Distance Dependencies As discussed in Section 3.2, we handle long-distance dependencies evoked by clausal modifiers (acl) and control verbs (xcomp) with the BIND mechanism, whereas DEPLAMBDA cannot handle control constructions. For xcomp, as seen earlier, we use the mapping λ f gx.∃y. f (x)∧g(y)∧xcomp(xe,ye). For acl we use λ f gx.∃y. f (x)∧ g(y), to conjoin the main clause and the modifier clause. However, not all acl clauses evoke long-distance dependencies, e.g. in the news that Disney won an Oscar, the clause that Disney won an Oscar is a subordinating conjunction of news. In such cases, we instead assign acl the INVERT semantics.
Questions Question words are marked with the enhanced part-of-speech tags DET:WH, ADV:WH and PRON:WH, which are all assigned the semantics λx.${word}(xa)∧ TARGET(xa). The predicate TARGET indicates that xa represents the variable of interest, that is the answer to the question.
3.4 Limitations
In order to achieve language independence, UDEPLAMBDA has to sacrifice semantic specificity, since in many cases the semantics is carried by lexical information. Consider the sentences John broke the
window and The window broke. Although it is the window that broke in both cases, our inferred logical forms do not canonicalize the relation between broke and window. To achieve this, we would have to make the substitution of nsubj depend on lexical context, such that when window occurs as nsubj with broke, the predicate arg2 is invoked rather than arg1. We do not address this problem, and leave it to the target application to infer that arg2 and arg1 have the same semantic function in these cases. We anticipate that the ability to make such lexicalized semantic inferences in a task-agnostic cross-lingual framework would be highly useful and a crucial avenue for future work on universal semantics.
Other constructions that require lexical information are quantifiers like every, some and most, negation markers like no and not, and intentional verbs like believe and said. UD does not have special labels to indicate these. Although not currently implemented, we discuss how to handle quantifiers in this framework in the supplementary material.
4 Cross-lingual Semantic Parsing
To study the multilingual nature of UDEPLAMBDA, we conduct an empirical evaluation on question answering against Freebase in three different languages: English, Spanish, and German. Before discussing the details of this experiment, we briefly outline the semantic parsing framework employed.
4.1 Semantic Parsing as Graph Matching
UDEPLAMBDA generates ungrounded logical forms that are independent of any knowledge base, such as Freebase. We use GRAPHPARSER (Reddy et al., 2016) to map these logical forms to their grounded Freebase graphs, via corresponding ungrounded graphs. Figures 3(a) to 3(c) show the ungrounded graphs corresponding to logical forms from UDEPLAMBDA, each grounded to the same Freebase graph in Figure 3(d). Here, rectangles denote entities, circles denote events, rounded rectangles denote entity types, and edges between events
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
WebQuestions
en What language do the people in Ghana speak? de Welche Sprache wird in Ghana gesprochen? es ¿Cuál es la lengua de Ghana?
en Who was Vincent van Gogh inspired by? de Von wem wurde Vincent van Gogh inspiriert? es ¿Qué inspiró a Van Gogh?
GraphQuestions
en NASA has how many launch sites? de Wie viele Abschussbasen besitzt NASA? es ¿Cuántos sitios de despegue tiene NASA?
en Which loudspeakers are heavier than 82.0 kg? de Welche Lautsprecher sind schwerer als 82.0 kg? es ¿Qué altavoces pesan más de 82.0 kg?
Table 1: Example questions and their translations.
and entities denote predicates or Freebase relations. Finally, the TARGET node represents the set of values of x that are consistent with the Freebase graph, that is the answer to the question.
GRAPHPARSER treats semantic parsing as a graph-matching problem with the goal of finding the Freebase graphs that are structurally isomorphic to an ungrounded graph and rank them according to a model. To account for structural mismatches, GRAPHPARSER uses two graph transformations: CONTRACT and EXPAND. In Figure 3(a) there are two edges between x and Ghana. CONTRACT collapses one of these edges to create a graph isomorphic to Freebase. EXPAND, in contrast, adds edges to connect the graph in the case of disconnected components. The search space is explored by beam search and model parameters are estimated with the averaged structured perceptron (Collins, 2002) from training data consisting of question-answer pairs, using answer F1-score as the objective.
4.2 Datasets
We evaluate our approach on two public benchmarks of question answering against Freebase: WebQuestions (Berant et al., 2013), a widely used benchmark consisting of English questions and their answers, and GraphQuestions (Su et al., 2016), a recently released dataset of English questions with both their answers and grounded logical forms. While WebQuestions is dominated by simple entityattribute questions, GraphQuestions contains a large number of compositional questions involving aggregation (e.g. How many children of Eddard Stark were born in Winterfell? ) and comparison (e.g. In which month does the average rainfall of
New York City exceed 86 mm? ). The number of training, development and test questions is 2644, 1134, and 2032, respectively, for WebQuestions and 1794, 764, and 2608 for GraphQuestions.
To support multilingual evaluation, we created translations of WebQuestions and GraphQuestions to German and Spanish.6 For WebQuestions two professional annotators were hired per language, while for GraphQuestions we used a trusted pool of 20 annotators per language (with a single annotator per question). Examples of the original questions and their translations are provided in Table 1.
4.3 Implementation Details
Here we provide details on the syntactic analyzers employed, our entity resolution algorithm, and the features used by the grounding model.
Dependency Parsing The English, Spanish, and German Universal Dependencies (UD) treebanks (v1.3; Nivre et al 2016) were used to train part of speech taggers and dependency parsers. We used a bidirectional LSTM tagger (Plank et al., 2016) and a bidirectional LSTM shift-reduce parser (Kiperwasser and Goldberg, 2016). Both the tagger and parser require word embeddings. For English, we used GloVe embeddings (Pennington et al., 2014) trained on Wikipedia and the Gigaword corpus.7 For German and Spanish, we used SENNA embeddings (Collobert et al., 2011; Al-Rfou et al., 2013) trained on Wikipedia corpora (589M words German; 397M words Spanish).8 Measured on the UD test sets, the tagger accuracies are 94.5 (English), 92.2 (German), and 95.7 (Spanish), with corresponding labeled attachment parser scores of 81.8, 74.7, and 82.2.
Entity Resolution We follow Reddy et al. (2016) and resolve entities in three steps: (1) potential entity spans are identified using seven handcrafted part-of-speech patterns; (2) each span is associated
6Translations will be publicly released upon publication. 7http://nlp.stanford.edu/projects/glove/. 8https://sites.google.com/site/rmyeid/projects/polyglot.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
WebQuestions GraphQuestions Method en de es en de es
SINGLEEVENT 47.6 43.9 45.0 15.9 8.3 11.2 DEPTREE 47.8 43.9 44.5 15.8 7.9 11.0 CCGGRAPH 48.4 – – 15.9 – – UDEPLAMBDA 48.3 44.2 45.7 17.6 9.0 12.4
Table 3: F1-scores on the test for models trained on the training set (excluding the development set).
with potential Freebase entities according to the Freebase/KG API;9 and (3) the 10-best entity linking lattices, scored by a structured perceptron, are input to GRAPHPARSER, leaving the final disambiguation to the semantic parsing problem. Table 2 shows the 1-best and 10-best entity disambiguation F1-scores for each language and dataset.10
Features We use features similar to Reddy et al. (2016): basic features of words and Freebase relations, and graph features crossing ungrounded events with grounded relations, ungrounded types with grounded relations, and ungrounded answer type crossed with a binary feature indicating if the answer is a number. In addition, we add features encoding the semantic similarity of ungrounded events and Freebase relations. Specifically, we used the cosine similarity of the translation-invariant embeddings of Huang et al. (2015).11
4.4 Comparison Systems
We compared UDEPLAMBDA to prior work and three versions of GRAPHPARSER that operate on different representations: entity cliques, dependency trees, and CCG-based semantic derivations.
SINGLEEVENT This model resembles the learning-to-rank model of Bast and Haussmann (2015). An ungrounded graph is generated by connecting all entities in the question with the TARGET node, representing a single event. Note that this baseline cannot handle compositional questions, or those with aggregation or comparison.
DEPTREE An ungrounded graph is obtained directly from the original dependency tree. An event is created for each parent and its dependents in the tree. Each dependent is linked to this event with an edge labeled with its dependency relation, while the
9http://developers.google.com/freebase/. 10Due to the recent Freebase API shutdown, we used the KG API for GraphQuestions. We observed that this leads to inferior entity linking results compared to those of Freebase.
11http://128.2.220.95/multilingual/data/.
parent is linked to the event with an edge labeled arg0. If a word is a question word, an additional TARGET predicate is attached to its entity node.
CCGGRAPH This is the CCG-based semantic representation of Reddy et al. (2014). Note that this baseline exists only for English.
4.5 Results
Table 3 shows the performance of GRAPHPARSER with these different representations. Here and in what follows, we use average F1-score of predicted answers (Berant et al., 2013) as the evaluation metric. We first observe that UDEPLAMBDA consistently outperforms the SINGLEEVENT and DEPTREE representations in all languages.12
For English, performance is almost on par with CCGGRAPH, which suggests that UDEPLAMBDA does not sacrifice too much specificity for universality. With both datasets, results are lower for German compared to Spanish. This agrees with the lower performance of the syntactic parser on the German portion of the UD treebank. Finally, while these results confirm that GraphQuestions is much harder compared to WebQuestions, we note that both datasets predominantly contain single-hop questions, as indicated by the competitive performance of SINGLEEVENT on both datasets.
Table 4 compares UDEPLAMBDA with previ-
12For the DEPTREE model, we CONTRACT each multihop path between the question word and an entity to a single edge. Without this constraint, DEPTREE F1 results are 45.5 (en), 42.9 (de), and 44.2 (es) on WebQuestions, and 11.0 (en), 6.6 (de), and 2.6 (es) on GraphQuestions.
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
ously published models which exist only for English and have been mainly evaluated on WebQuestions. These are either symbolic like ours (first block) or employ neural networks (second block). Results for models using additional task-specific training resources, such as ClueWeb09, Wikipedia, or SimpleQuestions (Bordes et al., 2015) are shown in parentheses. On GraphQuestions, we achieve a new state-of-the-art result with a gain of 4.8 F1points over the previously reported best result. On WebQuestions we are 2.1 points below the best model using comparable resources, and 3.8 points below the state of the art. Most related to our work is the English-specific system of Reddy et al. (2016). We attribute the 0.8 point difference in F1score to their use of the more fine-grained PTB tag set and Stanford Dependencies.
5 Related Work
Our work continues the long tradition of building logical forms from syntactic representations initiated by Montague (1973). The literature is rife with attempts to develop semantic interfaces for HPSG (Copestake et al., 2005), LFG (Kaplan and Bresnan, 1982; Dalrymple et al., 1995; Crouch and King, 2006), TAG (Kallmeyer and Joshi, 2003; Gardent and Kallmeyer, 2003; Nesson and Shieber, 2006), and CCG (Steedman, 2000; Baldridge and Kruijff, 2002; Bos et al., 2004; Artzi et al., 2015). Unlike existing semantic interfaces, UDEPLAMBDA (like DEPLAMBDA) uses dependency syntax, taking advantage of recent advances in multilingual parsing (McDonald et al., 2013; Nivre et al, 2016).
A common trend in previous work on semantic interfaces is the reliance on rich typed feature structures or semantic types coupled with strong type constraints, which can be very informative but unavoidably language specific. Creating rich semantic types from dependency trees which lack a typing system would be labor intensive and brittle in the face of parsing errors. Instead, UDEPLAMBDA relies on generic unlexicalized information present in dependency treebanks and uses a simple type system (one type for dependency labels, and one for words) along with a combinatory mechanism, which avoids type collisions. Earlier attempts at extracting semantic representations from dependencies have mainly focused on language-specific dependency representations (Spreyer and Frank, 2005; Simov and Osenova, 2011; Hahn and Meurers, 2011; Reddy et al., 2016; Falke et al., 2016;
Beltagy, 2016), and multi-layered dependency annotations (Jakob et al., 2010; Bédaride and Gardent, 2011). In contrast, UDEPLAMBDA derives semantic representations for multiple languages in a common schema directly from Universal Dependencies. This work parallels a growing interest in creating other forms of multilingual semantic representations (Akbik et al., 2015; Vanderwende et al., 2015; White et al., 2016; Evang and Bos, 2016).
We evaluate UDEPLAMBDA on semantic parsing for question answering against a knowledge base. Here, the literature offers two main modeling paradigms: (1) learning of task-specific grammars that directly parse language to a grounded representation (Zelle and Mooney, 1996; Zettlemoyer and Collins, 2005; Wong and Mooney, 2007; Kwiatkowksi et al., 2010; Liang et al., 2011; Berant et al., 2013; Flanigan et al., 2014; Pasupat and Liang, 2015; Groschwitz et al., 2015); and (2) converting language to a linguistically motivated taskindependent representation that is then mapped to a grounded representation (Kwiatkowski et al., 2013; Reddy et al., 2014; Krishnamurthy and Mitchell, 2015; Gardner and Krishnamurthy, 2017). Our work belongs to the latter paradigm, as we map natural language to Freebase indirectly via logical forms. Capitalizing on natural-language syntax affords interpretability, scalability, and reduced duplication of effort across applications (Bender et al., 2015). Our work also relates to literature on parsing multiple languages to a common executable representation (Cimiano et al., 2013; Haas and Riezler, 2016). However, existing approaches (Kwiatkowksi et al., 2010; Jones et al., 2012; Jie and Lu, 2014) still map to the target meaning representations (more or less) directly.
6 Conclusions
We introduced UDEPLAMBDA, a semantic interface for Universal Dependencies, and showed that the resulting semantic representation can be used for question-answering against a knowledge base in multiple languages. We provided translations of benchmark datasets in German and Spanish, in the hope to stimulate further multilingual research on semantic parsing and question answering in general. We have only scratched the surface when it comes to applying UDEPLAMBDA to natural language understanding tasks. In the future, we would like to explore how this framework can benefit other tasks such as summarization and machine translation.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
This paper describes interesting and ambitious work: the automated conversion of Universal Dependency grammar structures into [what the paper calls] semantic logical form representations.  In essence, each UD construct is assigned a target construction in logical form, and a procedure is defined to effect the conversion, working  inside-out  using an intermediate form to ensure proper nesting of substructures into encapsulating ones.  Two evaluations are carried out: comparing the results to gold-standard lambda structures and measuring the effectiveness of the resulting lambda expressions in actually delivering the answers to questions from two QA sets.    It is impossible to describe all this adequately in the space provided.  The authors have taken some care to cover all principal parts, but there are still many missing details.  I would love to see a longer version of the paper!  Particularly the QA results are short-changed; it would have been nice to learn which types of question are not handled, and which are not answered correctly, and why not.  This information would have been useful to gaining better insight into the limitations of the logical form representations.    That leads to my main concern objection.  This logical form representation is not in fact a  real  semantic one.                          It is, essentially, a rather close rewrite of the dependency structure of the input, with some (good) steps toward  semanticization , including the insertion of lambda operators, the explicit inclusion of dropped arguments (via the enhancement operation), and the introduction of appropriate types units for such constructions as eventive adjectives and nouns like  running horse d and  president in 2009 d.  But many (even simple) aspects of semantic are either not present (at least, not in the paper) and or simply wrong.  Missing: quantification (as in  every d or  all d); numbers (as in  20 d or  just over 1000 d); various forms of reference (as in  he d,  that man d,  what I said before d); negation and modals, which change the semantics in interesting ways; inter-event relationships (as in the subevent relationship between the events in  the vacation was nice, but traveling was a pain d; etc. etc.  To add them one can easily cheat, by treating these items as if they were just unusual words and defining obvious and simple lambda formulas for them.  But they in fact require specific treatment; for example, a number requires the creation of a separate set object in the representation, with its own canonical variable (allowing later text to refer to  one of them d and bind the variable properly).  For another example, Person A  model of an event may differ from Person B , so one needs two representation symbols for the event, plus a coupling and mapping between them.  For another example, one has to be able to handle time, even if simply by temporally indexing events and states.  None of this is here, and it is not immediately obvious how this would be added.  In some cases, as DRT shows, quantifier and referential scoping is not trivial.    It is easy to point to missing things, and unfair to the paper in some sense; you can t be expected to do it all.  But you cannot be allowed to make obvious errors.  Very disturbing is the assignment of event relations strictly in parallel with the verb  (or noun ) syntactic roles.  No-one can claim seriously that  he broke the window d and  the window broke d has  he d and  the window d filling the same semantic role for  break d.  That  simply not correct, and one cannot dismiss the problem, as the paper does, to some nebulous subsequent semantic processing.                          This really needs adequate treatment, even in this paper.  This is to my mind the principal shortcoming of this work; for me this is the make-or-break point as to whether I would fight to have the paper accepted in the conference.  (I would have been far happier if the authors had simply acknowledged that this aspect is wrong and will be worked on in future, with a sketch saying how: perhaps by reference to FrameNet and semantic filler requirements.)                            Independent of the representation, the notation conversion procedure is reasonably clear.  I like the facts that it is rather cleaner and simpler than its predecessor (based on Stanford dependencies), and also that the authors have the courage of submitting non-neural work to the ACL in these days of unbridled and giddy enthusiasm for anything neural.