Universal Dependencies (UD) provides a cross-linguistically uniform syntactic representation, with the aim of advancing multilingual applications of parsing and natural language understanding. Reddy et al. (2016) recently developed a semantic interface for (English) Stanford Dependencies, based on the lambda calculus. In this work, we introduce UDEPLAMBDA, a similar semantic interface for UD, which allows mapping natural language to logical forms in an almost language-independent framework. We evaluate our approach on semantic parsing for the task of question answering against Freebase. To facilitate multilingual evaluation, we provide German and Spanish translations of the WebQuestions and GraphQuestions datasets. Results show that UDEPLAMBDA outperforms strong baselines across languages and datasets. For English, it achieves the strongest result to date on GraphQuestions, with competitive results on WebQuestions.
1 Introduction
The Universal Dependencies (UD) initiative seeks to develop cross-linguistically consistent annotation guidelines as well as a large number of uniformly annotated treebanks for many languages.1 Such resources could advance multilingual applications of parsing, improve comparability of evaluation results, enable cross-lingual learning, and more generally support natural language understanding.
Seeking to exploit the benefits of UD for natural language understanding, we introduce UDEPLAMBDA, a semantic interface for UD that maps
1http://www.universaldependencies.org/.
natural language to logical forms, representing underlying predicate-argument structures, in an almost language-independent manner. Our framework is based on DEPLAMBDA (Reddy et al., 2016) a recently developed method that converts English Stanford Dependencies (SD) to logical forms. The conversion process is illustrated in Figure 1 and discussed in more detail in Section 2. Whereas DEPLAMBDA works only for English, UDEPLAMBDA applies to any language for which UD annotations are available.2 In this paper, we describe the rationale behind UDEPLAMBDA and highlight important differences from DEPLAMBDA, some of which stem from the different treatment of various linguistic constructions in UD.
Our experiments focus on semantic parsing as a testbed for evaluating the framework’s multilingual appeal. We address the task of learning to map natural language to machine interpretable formal meaning representations, specifically retrieving answers to questions from Freebase. To facilitate multilingual evaluation, we provide translations of the English WebQuestions (Berant et al., 2013) and GraphQuestions (Su et al., 2016) datasets to German and Spanish. We demonstrate that UDEPLAMBDA can be used to derive logical forms for these languages using a minimal amount of language-specific knowledge. Aside from developing the first multilingual semantic parsing tool for Freebase, we also experimentally show that UDEPLAMBDA outperforms strong baselines across languages and datasets. For English, it achieves the strongest result to date on GraphQuestions, with competitive results on WebQuestions. Beyond semantic parsing, we believe that the logical forms produced by our framework will be of use in various natural understanding tasks including entailment (Beltagy et al., 2016), text-based question
2As of v1.3, UD annotations are available for 47 languages.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
answering (Lewis and Steedman, 2013), sentence simplification (Narayan and Gardent, 2014), summarization (Liu et al., 2015), paraphrasing (Pavlick et al., 2015), and relation extraction (Rocktäschel et al., 2015). Our implementation and translated datasets will be made publicly available.
2 DEPLAMBDA
Before describing UDEPLAMBDA, we provide an overview of DEPLAMBDA (Reddy et al., 2016) on which our approach is based. DEPLAMBDA converts a dependency tree to its logical form in three steps: binarization, substitution, and composition, each of which is briefly outlined below.
Binarization A dependency tree is first mapped to a Lisp-style s-expression indicating the order of semantic composition. Figure 1(b) shows the s-expression for the sentence Disney won an Oscar for the movie Frozen, derived from the dependency tree in Figure 1(a). Here, the sub-expression (dobj won (det Oscar an)) indicates that the logical form of the phrase won an Oscar is derived by composing the logical form of the label dobj with the logical form of the word won and the logical form of the phrase an Oscar, derived analogously.
An obliqueness hierarchy is employed to impose a strict ordering on the modifiers to each head in the dependency tree. As an example, won has three modifiers in Figure 1(a), which according to the obliqueness hierarchy are composed in the order dobj> nmod> nsubj. In constructions like coordination, this ordering is crucial to arrive at the correct semantics (see Section 3.3).
Substitution Each symbol in the s-expressions is substituted for a lambda expression encoding its semantics. Words and dependency labels are assigned different types of expressions. In general, words have expressions of the following kind: ENTITY ⇒ λx.word(xa); e.g. Oscar⇒ λx.Oscar(xa) EVENT ⇒ λx.word(xe); e.g. won⇒ λx.won(xe) FUNCTIONAL⇒ λx.TRUE; e.g. an⇒ λx.TRUE
Here, the subscripts ·a and ·e denote the types of individuals (Ind) and events (Event), respectively, whereas x denotes a paired variable (xa,xe) of type Ind×Event. Roughly speaking, proper nouns and adjectives invoke ENTITY expressions, verbs and adverbs invoke EVENT expressions, and common nouns invoke both ENTITY and EVENT expressions (see Section 3.3), while remaining words invoke FUNCTIONAL expressions. As in DEPLAMBDA,
we enforce the constraint that every s-expression is of the type η = Ind×Event → Bool, which simplifies the type system considerably.
Expressions for dependency labels glue the semantics of heads and modifiers to articulate predicate-argument structure. These expressions in general take one of the following forms: COPY ⇒ λ f gx.∃y. f (x)∧g(y)∧ rel(x,y) e.g. nsubj, dobj, nmod, advmod INVERT ⇒ λ f gx.∃y. f (x)∧g(y)∧ reli(y,x) e.g. amod, acl MERGE ⇒ λ f gx. f (x)∧g(x) e.g. compound, appos, amod, acl HEAD ⇒ λ f gx. f (x) e.g. case, punct, aux, mark .
As an example of COPY, consider the lambda expression for dobj in (dobj won (det Oscar an)): λ f gx.∃y. f (x)∧ g(y)∧ arg2(xe,ya). This expression takes two functions f and g as input, where f represents the logical form of won and g represents the logical form of an Oscar. The predicateargument structure arg2(xe,ya) indicates that the arg2 of the event xe, i.e. won, is the individual ya, i.e. the entity Oscar. Since arg2(xe,ya) mimics the dependency structure dobj(won, Oscar), we refer to the expression kind evoked by dobj as COPY.
Expressions that invert the dependency direction are referred to as INVERT (e.g. amod in running horse); expressions that merge two subexpressions without introducing any relation predicates are referred to as MERGE (e.g. compound in movie Frozen); and expressions that simply return the parent expression semantics are referred to as HEAD
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
(e.g. case in for Frozen). While this generalization applies to most dependency labels, several labels take a different logical form not listed here, some of which are discussed in Section 3.3. Sometimes the mapping of dependency label to lambda expression may depend on surrounding part-of-speech tags or dependency labels. For example, amod acts as INVERT when the modifier is a verb (e.g. in running horse), and as MERGE when the modifier is an adjective (e.g. in beautiful horse).3
Composition The final logical form is computed by beta-reduction, treating expressions of the form (f x y) as the function f applied to the arguments x and y. For example, (dobj won (det Oscar an)) results in λx.∃z.won(xe)∧Oscar(za)∧ arg2(xe,za) when the expression for dobj is applied to those for won and (det Oscar an). Figure 1(c) shows the logical form for the s-expression in Figure 1(b).
3 UDEPLAMBDA
We now introduce UDEPLAMBDA, a semantic interface for Universal Dependencies.4 Whereas DEPLAMBDA only applies to English Stanford Dependencies, UDEPLAMBDA takes advantage of the cross-lingual nature of UD to facilitate an (almost) language independent semantic interface. This is accomplished by restricting the binarization, substitution, and composition steps described above to rely solely on information encoded in the UD representation. Importantly, UDEPLAMBDA is designed to not rely on lexical forms in a language to assign lambda expressions, but only on information contained in dependency labels and postags.
However, some linguistic phenomena are language specific (e.g. pronoun-dropping) or meaning specific (e.g. every and the in English have very different semantics, despite being both determiners) and are not encoded in the UD schema. Furthermore, some cross-linguistic phenomena, such as long-distance dependencies, are not part of the core UD representation. To circumvent this limitation, a simple enhancement step enriches the original UD representation before binarization takes place (Section 3.1). This step adds to the dependency tree missing syntactic information and long-distance dependencies, thereby creating a
3We use Tregex (Levy and Andrew, 2006) for substitution mappings and Cornell SPF (Artzi, 2013) as the lambdacalculus implementation. For example, in running horse, the tregex /label:amod/=target < /postag:verb/ matches amod to its INVERT expression λ f gx.∃y. f (x)∧g(y)∧ amodi(ye,xa).
4In what follows, all references to UD are to UD v1.3.
graph. Whereas DEPLAMBDA is not able to handle graph-structured input, UDEPLAMBDA is designed to work directly with dependency graphs (Section 3.2). Finally, the representation of several linguistic constructions differ between UD and SD, which requires different handling in the semantic interface (Section 3.3).
3.1 Enhancement
Both Schuster and Manning (2016) and Nivre et al. (2016) note the necessity of an enhanced UD representation to enable semantic applications. However, such enhancements are currently only available for a subset of languages in UD. Instead, we rely on a small number of enhancements for our main application—semantic parsing for questionanswering—with the hope that this step can be replaced by an enhanced UD representation in the future. Specifically, we define three kinds of enhancements: (1) long-distance dependencies; (2) types of coordination; and (3) refined question word tags.
First, we identify long-distance dependencies in relative clauses and control constructions. We follow Schuster and Manning (2016) and find these using the labels acl (relative) and xcomp (control). Figure 2(a) shows the long-distance dependency in the sentence Anna wants to marry Kristoff. Here, marry is provided with its missing nsubj (dashed arc). Second, UD conflates all coordinating constructions to a single dependency label, conj. To obtain the correct coordination scope, we refine conj to conj:verb, conj:vp, conj:sentence, conj:np, and conj:adj, similar to Reddy et al. (2016). Finally, unlike the PTB tags (Marcus et al., 1993) used by SD, the UD part-of-speech tags do
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
not distinguish question words. Since these are crucial to question-answering, we use a small lexicon to refine the tags for determiners (DET), adverbs (ADV) and pronouns (PRON) to DET:WH, ADV:WH and PRON:WH, respectively. Specifically, we use a list of 12 (English), 14 (Spanish) and 35 (German) words, respectively. This is the only part of UDEPLAMBDA that relies on language-specific information. We hope that, as the coverage of morphological features in UD improves, this refinement can be replaced by relying on morphological features, such as the interrogative feature (INT).
3.2 Graph Structures and BIND
To handle graph structures that may result from the enhancement step, such as those in Figure 2(a), we propose a variable-binding mechanism that differs from that of DEPLAMBDA. First, each longdistance dependency is split into independent arcs as shown in Figure 2(b). Here, Ω is a placeholder for the subject of marry, which in turn corresponds to Anna as indicated by the binding of Ω via the pseudo-label BIND. We treat BIND like an ordinary dependency label with semantics MERGE and process the resulting tree as usual, via the s-expression: (nsubj (xcomp wants (nsubj (mark
(dobj marry Kristoff) to) Ω) (BIND Anna Ω)) ,
with the lambda-expression substitutions: wants, marry ∈ EVENT; to ∈ FUNCTIONAL; Anna, Kristoff ∈ ENTITY; mark ∈ HEAD; BIND ∈ MERGE; xcomp = λ f gx.∃y. f (x)∧g(y)∧xcomp(xe,ye) .
These substitutions are based solely on unlexicalized context. For example, the part-of-speech tag PROPN of Anna invokes an ENTITY expression.
The placeholder Ω has semantics λx.EQ(x,ω), where EQ(u,ω) is true iff u and ω are equal (have the same denotation), which unifies the subject variable of wants with the subject variable of marry.
After substitution and composition, we get: λz.∃xywv.wants(ze)∧Anna(xa)∧ arg1(ze,xa)∧ EQ(x,ω) ∧marry(ye)∧xcomp(ze,ye)∧ arg1(ye,va)∧ EQ(v,ω) ∧ Kristoff(wa)∧ arg2(ye,wa) ,
This expression may be simplified further by replacing all occurrences of v with x and removing the unification predicates EQ, which results in: λz.∃xyw.wants(ze)∧Anna(xa)∧ arg1(ze,xa) ∧marry(ye)∧xcomp(ze,ye)∧ arg1(ye,xa) ∧ Kristoff(wa)∧ arg2(ye,wa) .
This expression encodes the fact that Anna is the arg1 of the marry event, as desired. DEPLAMBDA, in contrast, cannot handle graph-structured input,
since it lacks a principled way of generating sexpressions from graphs. Even given the above s-expression, BIND in DEPLAMBDA is defined in a way such that the composition fails to unify v and x, which is crucial for the correct semantics. Moreover, the definition of BIND in DEPLAMBDA does not have a formal interpretation within the lambda calculus, unlike ours.
3.3 Linguistic Constructions
Below, we highlight the most pertinent differences between UDEPLAMBDA and DEPLAMBDA, stemming from the different treatment of various linguistic constructions in UD versus SD.
Prepositional Phrases UD uses a content-head analysis, in contrast to SD, which treats function words as heads of prepositional phrases, Accordingly, the s-expression for the phrase president in 2009 is (nmod president (case 2009 in)) in UDEPLAMBDA and (prep president (pobj in 2009)) in DEPLAMBDA. To achieve the desired semantics, λx.∃y.president(xa)∧president event(xe)∧
arg1(xe,xa)∧2009(ya)∧prep.in(xe,ya) ,
DEPLAMBDA relies on an intermediate logical form that requires some post-processing, whereas UDEPLAMBDA obtains the desired logical form directly through the following entries: in ∈ FUNCTIONAL; 2009 ∈ ENTITY; case ∈ HEAD; president = λx.president(xa)∧president event(xe) ∧arg1(xe,xa) ; nmod = λ f gx.∃y. f (x)∧g(y)∧nmod.in(xe,ya) .
Other nmod constructions, such as possessives (nmod:poss), temporal modifiers (nmod:tmod) and adverbial modifiers (nmod:npmod), are handled similarly. Note how the common noun president, evokes both entity and event predicates above.
Passives DEPLAMBDA gives special treatment to passive verbs, identified by the fine-grained partof-speech tags in the PTB tag together with dependency context. For example, An Oscar was won is analyzed as λx.won.pass(xe)∧Oscar(ya)∧ arg1(xe,ya), where won.pass represents a passive event. However, UD does not distinguish between active and passive forms.5 While the labels nsubjpass or auxpass indicate passive constructions, such clues are sometimes missing, such as in reduced relatives. We therefore opt to not have separate entries for passives, but aim to produce identical logical forms for active and passive forms when
5UD encodes voice as a morphological feature, but most syntactic analyzers do not produce this information yet.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
language target people
x e1 y e2 Ghana speak .arg2 speak .arg1 people .arg1 people .nmod.in
typ e
typ e
(a) English
sprache target
x e1 Ghana gesprochen
.arg2 gesprochen .nmod.in
typ e
(b) German
lengua target
x e1 Ghana lengua .arg1 lengua .nmod.de
typ e
(c) Spanish
language .human language
target
x m Ghana location.country
.official language.2 location.country .official language.1
typ e
(d) Freebase
Figure 3: The ungrounded graphs for What language do the people in Ghana speak?, Welche Sprache wird in Ghana gesprochen? and Cuál es la lengua de Ghana?, and the corresponding grounded graph.
possible (for example, by treating nsubjpass as direct object). With the following entries, won ∈ EVENT; an, was ∈ FUNCTIONAL; auxpass ∈ HEAD; nsubjpass = λ f gx.∃y. f (x)∧g(y)∧ arg2(xe,ya) ,
the lambda expression for An Oscar was won becomes λx.won(xe)∧Oscar(ya)∧arg2(xe,ya), identical to that of its active form. However, not having a special entry for passive verbs may have undesirable side-effects. For example, in the reducedrelative construction Pixar claimed the Oscar won for Frozen, the phrase the Oscar won ... will receive the semantics λx.Oscar(ya)∧won(xe)∧ arg1(xe,ya), which differs from that of an Oscar was won. We leave it to the target application to disambiguate the interpretation in such cases.
Long-Distance Dependencies As discussed in Section 3.2, we handle long-distance dependencies evoked by clausal modifiers (acl) and control verbs (xcomp) with the BIND mechanism, whereas DEPLAMBDA cannot handle control constructions. For xcomp, as seen earlier, we use the mapping λ f gx.∃y. f (x)∧g(y)∧xcomp(xe,ye). For acl we use λ f gx.∃y. f (x)∧ g(y), to conjoin the main clause and the modifier clause. However, not all acl clauses evoke long-distance dependencies, e.g. in the news that Disney won an Oscar, the clause that Disney won an Oscar is a subordinating conjunction of news. In such cases, we instead assign acl the INVERT semantics.
Questions Question words are marked with the enhanced part-of-speech tags DET:WH, ADV:WH and PRON:WH, which are all assigned the semantics λx.${word}(xa)∧ TARGET(xa). The predicate TARGET indicates that xa represents the variable of interest, that is the answer to the question.
3.4 Limitations
In order to achieve language independence, UDEPLAMBDA has to sacrifice semantic specificity, since in many cases the semantics is carried by lexical information. Consider the sentences John broke the
window and The window broke. Although it is the window that broke in both cases, our inferred logical forms do not canonicalize the relation between broke and window. To achieve this, we would have to make the substitution of nsubj depend on lexical context, such that when window occurs as nsubj with broke, the predicate arg2 is invoked rather than arg1. We do not address this problem, and leave it to the target application to infer that arg2 and arg1 have the same semantic function in these cases. We anticipate that the ability to make such lexicalized semantic inferences in a task-agnostic cross-lingual framework would be highly useful and a crucial avenue for future work on universal semantics.
Other constructions that require lexical information are quantifiers like every, some and most, negation markers like no and not, and intentional verbs like believe and said. UD does not have special labels to indicate these. Although not currently implemented, we discuss how to handle quantifiers in this framework in the supplementary material.
4 Cross-lingual Semantic Parsing
To study the multilingual nature of UDEPLAMBDA, we conduct an empirical evaluation on question answering against Freebase in three different languages: English, Spanish, and German. Before discussing the details of this experiment, we briefly outline the semantic parsing framework employed.
4.1 Semantic Parsing as Graph Matching
UDEPLAMBDA generates ungrounded logical forms that are independent of any knowledge base, such as Freebase. We use GRAPHPARSER (Reddy et al., 2016) to map these logical forms to their grounded Freebase graphs, via corresponding ungrounded graphs. Figures 3(a) to 3(c) show the ungrounded graphs corresponding to logical forms from UDEPLAMBDA, each grounded to the same Freebase graph in Figure 3(d). Here, rectangles denote entities, circles denote events, rounded rectangles denote entity types, and edges between events
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
WebQuestions
en What language do the people in Ghana speak? de Welche Sprache wird in Ghana gesprochen? es ¿Cuál es la lengua de Ghana?
en Who was Vincent van Gogh inspired by? de Von wem wurde Vincent van Gogh inspiriert? es ¿Qué inspiró a Van Gogh?
GraphQuestions
en NASA has how many launch sites? de Wie viele Abschussbasen besitzt NASA? es ¿Cuántos sitios de despegue tiene NASA?
en Which loudspeakers are heavier than 82.0 kg? de Welche Lautsprecher sind schwerer als 82.0 kg? es ¿Qué altavoces pesan más de 82.0 kg?
Table 1: Example questions and their translations.
and entities denote predicates or Freebase relations. Finally, the TARGET node represents the set of values of x that are consistent with the Freebase graph, that is the answer to the question.
GRAPHPARSER treats semantic parsing as a graph-matching problem with the goal of finding the Freebase graphs that are structurally isomorphic to an ungrounded graph and rank them according to a model. To account for structural mismatches, GRAPHPARSER uses two graph transformations: CONTRACT and EXPAND. In Figure 3(a) there are two edges between x and Ghana. CONTRACT collapses one of these edges to create a graph isomorphic to Freebase. EXPAND, in contrast, adds edges to connect the graph in the case of disconnected components. The search space is explored by beam search and model parameters are estimated with the averaged structured perceptron (Collins, 2002) from training data consisting of question-answer pairs, using answer F1-score as the objective.
4.2 Datasets
We evaluate our approach on two public benchmarks of question answering against Freebase: WebQuestions (Berant et al., 2013), a widely used benchmark consisting of English questions and their answers, and GraphQuestions (Su et al., 2016), a recently released dataset of English questions with both their answers and grounded logical forms. While WebQuestions is dominated by simple entityattribute questions, GraphQuestions contains a large number of compositional questions involving aggregation (e.g. How many children of Eddard Stark were born in Winterfell? ) and comparison (e.g. In which month does the average rainfall of
New York City exceed 86 mm? ). The number of training, development and test questions is 2644, 1134, and 2032, respectively, for WebQuestions and 1794, 764, and 2608 for GraphQuestions.
To support multilingual evaluation, we created translations of WebQuestions and GraphQuestions to German and Spanish.6 For WebQuestions two professional annotators were hired per language, while for GraphQuestions we used a trusted pool of 20 annotators per language (with a single annotator per question). Examples of the original questions and their translations are provided in Table 1.
4.3 Implementation Details
Here we provide details on the syntactic analyzers employed, our entity resolution algorithm, and the features used by the grounding model.
Dependency Parsing The English, Spanish, and German Universal Dependencies (UD) treebanks (v1.3; Nivre et al 2016) were used to train part of speech taggers and dependency parsers. We used a bidirectional LSTM tagger (Plank et al., 2016) and a bidirectional LSTM shift-reduce parser (Kiperwasser and Goldberg, 2016). Both the tagger and parser require word embeddings. For English, we used GloVe embeddings (Pennington et al., 2014) trained on Wikipedia and the Gigaword corpus.7 For German and Spanish, we used SENNA embeddings (Collobert et al., 2011; Al-Rfou et al., 2013) trained on Wikipedia corpora (589M words German; 397M words Spanish).8 Measured on the UD test sets, the tagger accuracies are 94.5 (English), 92.2 (German), and 95.7 (Spanish), with corresponding labeled attachment parser scores of 81.8, 74.7, and 82.2.
Entity Resolution We follow Reddy et al. (2016) and resolve entities in three steps: (1) potential entity spans are identified using seven handcrafted part-of-speech patterns; (2) each span is associated
6Translations will be publicly released upon publication. 7http://nlp.stanford.edu/projects/glove/. 8https://sites.google.com/site/rmyeid/projects/polyglot.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
WebQuestions GraphQuestions Method en de es en de es
SINGLEEVENT 47.6 43.9 45.0 15.9 8.3 11.2 DEPTREE 47.8 43.9 44.5 15.8 7.9 11.0 CCGGRAPH 48.4 – – 15.9 – – UDEPLAMBDA 48.3 44.2 45.7 17.6 9.0 12.4
Table 3: F1-scores on the test for models trained on the training set (excluding the development set).
with potential Freebase entities according to the Freebase/KG API;9 and (3) the 10-best entity linking lattices, scored by a structured perceptron, are input to GRAPHPARSER, leaving the final disambiguation to the semantic parsing problem. Table 2 shows the 1-best and 10-best entity disambiguation F1-scores for each language and dataset.10
Features We use features similar to Reddy et al. (2016): basic features of words and Freebase relations, and graph features crossing ungrounded events with grounded relations, ungrounded types with grounded relations, and ungrounded answer type crossed with a binary feature indicating if the answer is a number. In addition, we add features encoding the semantic similarity of ungrounded events and Freebase relations. Specifically, we used the cosine similarity of the translation-invariant embeddings of Huang et al. (2015).11
4.4 Comparison Systems
We compared UDEPLAMBDA to prior work and three versions of GRAPHPARSER that operate on different representations: entity cliques, dependency trees, and CCG-based semantic derivations.
SINGLEEVENT This model resembles the learning-to-rank model of Bast and Haussmann (2015). An ungrounded graph is generated by connecting all entities in the question with the TARGET node, representing a single event. Note that this baseline cannot handle compositional questions, or those with aggregation or comparison.
DEPTREE An ungrounded graph is obtained directly from the original dependency tree. An event is created for each parent and its dependents in the tree. Each dependent is linked to this event with an edge labeled with its dependency relation, while the
9http://developers.google.com/freebase/. 10Due to the recent Freebase API shutdown, we used the KG API for GraphQuestions. We observed that this leads to inferior entity linking results compared to those of Freebase.
11http://128.2.220.95/multilingual/data/.
parent is linked to the event with an edge labeled arg0. If a word is a question word, an additional TARGET predicate is attached to its entity node.
CCGGRAPH This is the CCG-based semantic representation of Reddy et al. (2014). Note that this baseline exists only for English.
4.5 Results
Table 3 shows the performance of GRAPHPARSER with these different representations. Here and in what follows, we use average F1-score of predicted answers (Berant et al., 2013) as the evaluation metric. We first observe that UDEPLAMBDA consistently outperforms the SINGLEEVENT and DEPTREE representations in all languages.12
For English, performance is almost on par with CCGGRAPH, which suggests that UDEPLAMBDA does not sacrifice too much specificity for universality. With both datasets, results are lower for German compared to Spanish. This agrees with the lower performance of the syntactic parser on the German portion of the UD treebank. Finally, while these results confirm that GraphQuestions is much harder compared to WebQuestions, we note that both datasets predominantly contain single-hop questions, as indicated by the competitive performance of SINGLEEVENT on both datasets.
Table 4 compares UDEPLAMBDA with previ-
12For the DEPTREE model, we CONTRACT each multihop path between the question word and an entity to a single edge. Without this constraint, DEPTREE F1 results are 45.5 (en), 42.9 (de), and 44.2 (es) on WebQuestions, and 11.0 (en), 6.6 (de), and 2.6 (es) on GraphQuestions.
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
ously published models which exist only for English and have been mainly evaluated on WebQuestions. These are either symbolic like ours (first block) or employ neural networks (second block). Results for models using additional task-specific training resources, such as ClueWeb09, Wikipedia, or SimpleQuestions (Bordes et al., 2015) are shown in parentheses. On GraphQuestions, we achieve a new state-of-the-art result with a gain of 4.8 F1points over the previously reported best result. On WebQuestions we are 2.1 points below the best model using comparable resources, and 3.8 points below the state of the art. Most related to our work is the English-specific system of Reddy et al. (2016). We attribute the 0.8 point difference in F1score to their use of the more fine-grained PTB tag set and Stanford Dependencies.
5 Related Work
Our work continues the long tradition of building logical forms from syntactic representations initiated by Montague (1973). The literature is rife with attempts to develop semantic interfaces for HPSG (Copestake et al., 2005), LFG (Kaplan and Bresnan, 1982; Dalrymple et al., 1995; Crouch and King, 2006), TAG (Kallmeyer and Joshi, 2003; Gardent and Kallmeyer, 2003; Nesson and Shieber, 2006), and CCG (Steedman, 2000; Baldridge and Kruijff, 2002; Bos et al., 2004; Artzi et al., 2015). Unlike existing semantic interfaces, UDEPLAMBDA (like DEPLAMBDA) uses dependency syntax, taking advantage of recent advances in multilingual parsing (McDonald et al., 2013; Nivre et al, 2016).
A common trend in previous work on semantic interfaces is the reliance on rich typed feature structures or semantic types coupled with strong type constraints, which can be very informative but unavoidably language specific. Creating rich semantic types from dependency trees which lack a typing system would be labor intensive and brittle in the face of parsing errors. Instead, UDEPLAMBDA relies on generic unlexicalized information present in dependency treebanks and uses a simple type system (one type for dependency labels, and one for words) along with a combinatory mechanism, which avoids type collisions. Earlier attempts at extracting semantic representations from dependencies have mainly focused on language-specific dependency representations (Spreyer and Frank, 2005; Simov and Osenova, 2011; Hahn and Meurers, 2011; Reddy et al., 2016; Falke et al., 2016;
Beltagy, 2016), and multi-layered dependency annotations (Jakob et al., 2010; Bédaride and Gardent, 2011). In contrast, UDEPLAMBDA derives semantic representations for multiple languages in a common schema directly from Universal Dependencies. This work parallels a growing interest in creating other forms of multilingual semantic representations (Akbik et al., 2015; Vanderwende et al., 2015; White et al., 2016; Evang and Bos, 2016).
We evaluate UDEPLAMBDA on semantic parsing for question answering against a knowledge base. Here, the literature offers two main modeling paradigms: (1) learning of task-specific grammars that directly parse language to a grounded representation (Zelle and Mooney, 1996; Zettlemoyer and Collins, 2005; Wong and Mooney, 2007; Kwiatkowksi et al., 2010; Liang et al., 2011; Berant et al., 2013; Flanigan et al., 2014; Pasupat and Liang, 2015; Groschwitz et al., 2015); and (2) converting language to a linguistically motivated taskindependent representation that is then mapped to a grounded representation (Kwiatkowski et al., 2013; Reddy et al., 2014; Krishnamurthy and Mitchell, 2015; Gardner and Krishnamurthy, 2017). Our work belongs to the latter paradigm, as we map natural language to Freebase indirectly via logical forms. Capitalizing on natural-language syntax affords interpretability, scalability, and reduced duplication of effort across applications (Bender et al., 2015). Our work also relates to literature on parsing multiple languages to a common executable representation (Cimiano et al., 2013; Haas and Riezler, 2016). However, existing approaches (Kwiatkowksi et al., 2010; Jones et al., 2012; Jie and Lu, 2014) still map to the target meaning representations (more or less) directly.
6 Conclusions
We introduced UDEPLAMBDA, a semantic interface for Universal Dependencies, and showed that the resulting semantic representation can be used for question-answering against a knowledge base in multiple languages. We provided translations of benchmark datasets in German and Spanish, in the hope to stimulate further multilingual research on semantic parsing and question answering in general. We have only scratched the surface when it comes to applying UDEPLAMBDA to natural language understanding tasks. In the future, we would like to explore how this framework can benefit other tasks such as summarization and machine translation.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

This paper describes interesting and ambitious work: the automated conversion of Universal Dependency grammar structures into [what the paper calls] semantic logical form representations.  In essence, each UD construct is assigned a target construction in logical form, and a procedure is defined to effect the conversion, working  inside-out  using an intermediate form to ensure proper nesting of substructures into encapsulating ones.  Two evaluations are carried out: comparing the results to gold-standard lambda structures and measuring the effectiveness of the resulting lambda expressions in actually delivering the answers to questions from two QA sets.    It is impossible to describe all this adequately in the space provided.  The authors have taken some care to cover all principal parts, but there are still many missing details.  I would love to see a longer version of the paper!  Particularly the QA results are short-changed; it would have been nice to learn which types of question are not handled, and which are not answered correctly, and why not.  This information would have been useful to gaining better insight into the limitations of the logical form representations.    That leads to my main concern objection.  This logical form representation is not in fact a  real  semantic one.                          It is, essentially, a rather close rewrite of the dependency structure of the input, with some (good) steps toward  semanticization , including the insertion of lambda operators, the explicit inclusion of dropped arguments (via the enhancement operation), and the introduction of appropriate types units for such constructions as eventive adjectives and nouns like  running horse d and  president in 2009 d.  But many (even simple) aspects of semantic are either not present (at least, not in the paper) and or simply wrong.  Missing: quantification (as in  every d or  all d); numbers (as in  20 d or  just over 1000 d); various forms of reference (as in  he d,  that man d,  what I said before d); negation and modals, which change the semantics in interesting ways; inter-event relationships (as in the subevent relationship between the events in  the vacation was nice, but traveling was a pain d; etc. etc.  To add them one can easily cheat, by treating these items as if they were just unusual words and defining obvious and simple lambda formulas for them.  But they in fact require specific treatment; for example, a number requires the creation of a separate set object in the representation, with its own canonical variable (allowing later text to refer to  one of them d and bind the variable properly).  For another example, Person A  model of an event may differ from Person B , so one needs two representation symbols for the event, plus a coupling and mapping between them.  For another example, one has to be able to handle time, even if simply by temporally indexing events and states.  None of this is here, and it is not immediately obvious how this would be added.  In some cases, as DRT shows, quantifier and referential scoping is not trivial.    It is easy to point to missing things, and unfair to the paper in some sense; you can t be expected to do it all.  But you cannot be allowed to make obvious errors.  Very disturbing is the assignment of event relations strictly in parallel with the verb  (or noun ) syntactic roles.  No-one can claim seriously that  he broke the window d and  the window broke d has  he d and  the window d filling the same semantic role for  break d.  That  simply not correct, and one cannot dismiss the problem, as the paper does, to some nebulous subsequent semantic processing.                          This really needs adequate treatment, even in this paper.  This is to my mind the principal shortcoming of this work; for me this is the make-or-break point as to whether I would fight to have the paper accepted in the conference.  (I would have been far happier if the authors had simply acknowledged that this aspect is wrong and will be worked on in future, with a sketch saying how: perhaps by reference to FrameNet and semantic filler requirements.)                            Independent of the representation, the notation conversion procedure is reasonably clear.  I like the facts that it is rather cleaner and simpler than its predecessor (based on Stanford dependencies), and also that the authors have the courage of submitting non-neural work to the ACL in these days of unbridled and giddy enthusiasm for anything neural.


We introduce a novel framework for evaluating multimodal deep learning models with respect to their language understanding and generalization abilities. In this approach, artificial data is automatically generated according to the experimenter’s specifications. The content of the data, both during training and evaluation, can be controlled in detail, which enables tasks to be created that require generalization abilities, in particular the combination of previously introduced concepts in novel ways. We demonstrate the potential of our methodology by evaluating a multimodal architecture on four different tasks, and show that our framework gives us insights into the model’s capabilities and limitations.
1 Introduction
Deep learning methods have had a major impact on research in natural language processing and raised performance substantially in many of the standard evaluations. Moreover, multimodal tasks like image captioning (Karpathy and Li, 2015) or visual question answering (Antol et al., 2015) can now be tackled with great success. Such systems seem to solve the problems entirely on a sub-symbolic level, based only on raw image (and text) input, whereas previous approaches required a hand-crafted combination of various higher-level components.
There is, however, concern about how deep neural networks learn to solve such tasks. Investigations for image recognition (Szegedy et al., 2014; Nguyen et al., 2015; Zhang et al., 2017) have shown surprising behavior very different from what would be expected of their “surpassing
human-level performance” (He et al., 2015). Deep networks for language tasks may exhibit similarly odd behavior (Sproat and Jaitly, 2016; Arthur et al., 2016). Such results cast doubt on whether deep learning systems reliably acquire appropriate generalizations. However, given the recursive nature of language and the potentially enormous problem space of image captioning and similar tasks, acquiring the ability for reliable generalization will eventually be essential.
A more theoretical issue is the capability of network architectures to be able, in principle, to learn certain classes of structure. For instance, it has been shown that LSTMs possess the ability to handle long-range dependencies (Hochreiter and Schmidhuber, 1997; Gers and Schmidhuber, 2001). However, the formal experiments that have been done along such lines are limited, particularly in the multimodal domain of vision and language. While recent work indicates that the information encoded in image embeddings is rich enough for good captioning results, it is an open question whether current multimodal architectures are able, in principle, to combine visual information effectively to handle the full range of linguistic constructions.
This paper introduces a new test methodology for multimodal deep learning models. SHAPEWORLD is a framework for specifying datasets, which differ from standard evaluation datasets in two main ways. Firstly, a SHAPEWORLD dataset defines a process for generating artificial data, which is randomly sampled during training/testing according to constraints specified by the experimenter. Secondly, the evaluation focus of the methodology is on linguistic understanding capabilities of the type investigated by formal semantics. Hence the visual complexity and the openclass vocabulary size is reduced to a minimum, while allowing indefinitely complex syntactic con-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
structions. Since it is possible to control data generation for both training and evaluation, we can introduce previously unseen configurations in the test data. This allows us to design tasks which require the system to recombine learned concepts to understand these novel instances, and is hence a form of zero-shot learning. We think of the SHAPEWORLD tasks as unit-testing multimodal systems for specific linguistic generalization capabilities, in a similar way to the bAbI task set of Weston et al. (2015) for text-only understanding.
We also present initial results on the performance of a generic multimodal architecture. The interest here is not obtaining high numbers, but rather analyzing the performance of a generic model with our evaluation methodology and demonstrating the potential of SHAPEWORLD in investigating multimodal deep neural networks. In fact, we have been surprised to see some poor performance even for simple tasks. We invite the community to explore the SHAPEWORLD tasks and use them to evaluate other successful image captioning or visual question answering systems. We want to emphasize, however, that the goal is not to achieve optimal performance by tuning a network architecture for one of these tasks. The central question is rather whether deep network architectures are able to successfully demonstrate the required understanding and generalization ability, and whether this will carry over to more complex tasks which can be defined using SHAPEWORLD.
2 Related work
With the increasing popularity of deep learning approaches, artificial data of various kinds is again seen as a valuable tool in experimentation. Recently, the simulation paradigm has been argued to be a promising driver for artificial intelligence research (Kiela et al., 2016). Various platforms following this paradigm have already been released, amongst others, OpenAI Gym (Brockman et al., 2016), DeepMind Lab (Beattie et al., 2016), Project Malmo (Johnson et al., 2016), to name a few of the most popular. An important advantage of simulated data is its infinite availability, particularly in light of the need of many deep learning models for huge amounts of data. Automatically generating data greatly reduces the cost, time and human effort. Moreover, it allows researchers to focus on particular problems, isolated from noisy
and complex environments. When focusing on language tasks, the simulation paradigm faces the problem that interesting language generation is a difficult task in its own right and that the difficulty increases with the complexity of the underlying world. The bAbI tasks (Weston et al., 2015) are generated by internally simulating a short scene and extracting a few simple sentences from it. A similar approach is taken by Narasimhan et al. (2015), but here the simulation is more complex, comprising a text-based role-playing game. The MazeBase game environment (Sukhbaatar et al., 2015) also uses language as a mean to represent the game world. However, the descriptions are in an abstract, formulaic format, and the focus of the simulation is much more on the planning than the language component. The long-term research proposal of Mikolov et al. (2015) also simulates a world where an agent learns to solve tasks by communication with a teacher module. At least for a start, this module is supposed to be scripted to automatically generate appropriate responses, given its internal knowledge of the world state.
Automatically generated data is common for tasks specifically focusing on the ability to efficiently process data of a certain formal structure. Here, data is deliberately stripped of any realworld connection to create an abstract capability check. Recent work in the context of deep learning has investigated sequence patterns (Joulin and Mikolov, 2015), combinatorial problems (Vinyals et al., 2015), or executing programming language code (Zaremba and Sutskever, 2014), amongst others. This kind of tasks is particularly common for neural network models (compare e.g., Bengio et al. (1994) more than twenty years ago). The reason for interest in abstract capability tests is that the learning process of deep neural networks is far more difficult to understand in a detailed way than shallower machine learning methods.
The multimodal tasks of image captioning and visual question answering are most closely related to our methodology, but usually consist of repurposed real-world photos and human-written descriptions1. However, there have been experiments in which parts of the data are artificial or generated automatically: for instance, automatic
1Although we use the term “real-world”, in contrast to artificial simulations, it should be clear that it is very unlike the visual input which would be experienced by an entity situated in the real world
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
question generation (Ren et al., 2015) or modification of captions (Hodosh and Hockenmaier, 2016). Abstract Clipart scenes have been used for image captioning (Zitnick et al., 2016; Zitnick and Parikh, 2013). Most similar to our work are the experiments done by Bowman et al. (2015) and Sorodoc et al. (2016). Both generate artificial data fully automatically, based on abstract models, for a task which is targeted at a specific linguistic aspect, logical semantics and quantifiers, respectively.
Our own work is based on automatically generated, fully artificial data. This data, however, is not designed to address only a single structural problem, but is able to cover a whole range of linguistic phenomena. In fact, our generation system closely resembles classical work in formal semantics, where a statement corresponds to a logical expression which can be evaluated against an abstract world model (Montague, 1970). We utilize broad-coverage realization driven by the English Resource Grammar (ERG) (Flickinger, 2000) processed with Packard’s Answer Constraint Engine (ACE, http://sweaglesw. org/linguistics/ace/). However, while SHAPEWORLD uses a logical representation internally, the external representation seen by the system under evaluation does not involve any abstract formalization of text or images. It nevertheless presents the intended problems clearly, without any uncontrolled noise, biases, or potentially hidden correlations, which can obfuscate results when using real-world images and text.
3 The SHAPEWORLD framework
The SHAPEWORLD framework2 is based on the concept of microworlds – small and selfcontained artificial scenarios – which guide the data creation process. The SHAPEWORLD microworlds simply consist of colored shapes. This closed-world domain allows for exhaustive coverage of the space of possible microworlds and associated captions, reducing the vocabulary size to (for now) far less than 100 words with an emphasis
2 The SHAPEWORLD code is written in Python 3 and is available on GitHub (https://git.io/vDWOX). The generated data is returned as NumPy arrays, so that it is possible to integrate it into Python-based deep learning projects based on common frameworks like TensorFlow, Theano, etc. In our experiments, we use Tensorflow and we provide example scripts as part of the package. The Python package pydmrs (Copestake et al., 2016) is required for the internal DMRS-based caption generation.
on closed-class words. In the following we explain the details of the data generation process inside the SHAPEWORLD framework. A schematic illustration of the process is shown in figure 1.
3.1 Image caption agreement task
In this paper we focus on the task of image caption agreement. The system to be evaluated is presented with an image and a natural language caption, and has to decide whether they are consistent with each other. Compared to the classic image captioning task, it emphasizes understanding rather than the synthesis part of language use. We therefore avoid the problem of evaluating the appropriateness of a caption. The setup allows us to control the content of both modalities and consequently force a system to cope with difficult types of captions while obtaining a clear indicator of successful understanding.
One further motivation for the task is that human performance could be measured using the same setup. We would expect close-to-perfect human performance on the tasks described here, assuming time is not tightly constrained. Interesting comparisons are potentially possible where human performance depends on presentation: e.g., quantifiers such as most (Pietroski et al., 2009). However, we will not discuss this further in the current paper.
3.2 World and image generation
At the core of each abstract microworld instance lies an abstract world model. In the SHAPEWORLD framework, the internal representation of a microworld is simply a list of entities given as records containing their primary attributes, such as position, shape, color, which are considered to be high-level semantic aspects reflected in captions. In addition, an entity has secondary attributes and methods which control, for instance, details of visual appearance, visual noise infusion, or the collision-free placement of entities.
Currently, the SHAPEWORLD framework provides eight shape types (square, rectangle, triangle, pentagon, cross, circle, semicircle and ellipse) and seven colors (red, green, blue, yellow, magenta, cyan and white). The location and the secondary attributes are sampled uniformly or according to a truncated normal distribution (object size: 10% to 15% of image size; rotation: random; shade: shifted up to 50% towards black or white; and general noise: up to 10% deviation
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
World Generator World Captioner
World Model {s : tr, c : gr, x : 0.9, y : 0.1, . . . }, {s : ci, c : bl, x : 0.2, y : 0.3, . . . }, {s : cr, c : re, x : 0.7, y : 0.5, . . . }, {s : sq, c : re, x : 0.8, y : 0.8, . . . }, {s : ci, c : gr, x : 0.3, y : 0.9, . . . }
Model for Caption {s : cr, c : bl, x : 0.9, y : 0.1, . . . }, {s : tr, c : re, x : 0.2, y : 0.3, . . . }, {s : cr, c : re, x : 0.7, y : 0.5, . . . }, {s : sq, c : gr, x : 0.8, y : 0.8, . . . }
DMRS Graph Pattern
Caption Object Image:
Agreement: False ≈ 0 ∈ [0, 1]
Caption: There is a red triangle.
(for negative instance)
(for positive
instance)
Figure 1: The generation process for image caption agreement data. Depending on whether a positive instance, i.e., a true statement about the world, or a negative instance is to be generated, either the first world model is used for caption generation or a new microworld is generated.
per pixel). Importantly, all these ways of infusing noise can be controlled, which is useful particularly since noise is often seen as important for successful training of deep models.
The generator module automatically generates a world model by randomly sampling all these attributes from a set of available values. Both the values and other aspects of the generation process can be specified and adjusted appropriately. The internal abstract representation is then used as a basis to extract a concrete microworld instance consisting of image and caption. An image (of size 64×64 in this work) is just a straightforward visualization of the world model. Note that the exact visual appearance of an entity with certain primary attributes varies from instance to instance. Caption generation is more complex, as discussed in the next section.
3.3 Caption generation
The caption interface specifies the methods a language generation module has to provide to integrate into the data generation process. We currently provide an implementation using a grammar-based approach. More specifically, Dependency Minimal Recursion Semantics (DMRS) is an abstract semantic graph representation designed for use with high-precision grammars, such as those distributed by the DELPH-IN consortium. A semantic representation like DMRS is particularly suited for the SHAPEWORLD framework, since it essentially mirrors parts of the internal world model and hence acts like a languagespecific annotation.3 For instance, noun nodes
3Although we currently use the English Resource Grammar (Flickinger, 2000), other DELPH-IN grammars use a
correspond to entities, adjective nodes add attributes, and verb phrase nodes or sub-graphs specify relations between entities. Below an example of a DMRS semantic graph with its compositional components colored:
There is a blue circle.
Compositionality of the semantic representation is a useful property and an important reason for our choice of using DMRS. Given compositionality, it is enough to specify the semantics of words – or, more precisely, of the linguistic atoms in the SHAPEWORLD context, which potentially are sub-graphs with multiple nodes and inner link structure – to be able to obtain the corresponding semantics of composed sub-graphs and so generate a wide range of combinatorially different captions. For instance, the semantics of words like square or red iteratively filter a subset of agreeing entities, transitive relations like to the left of act similarly on pairs of entity sets, and quantifiers compare the cardinality of two entity sets.
Figure 2 shows an example of a more complex compositional caption, which contains the DMRS graph above as sub-graph. It also illustrates how various details are automatically inferred by the ERG, including number-agreement between subject and verb, and between quantifier and noun, and realization of an adjective as relative clause.
compatible approach, so SHAPEWORLD could be ported to other languages relatively easily.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Most squares are green and there are some circles which are blue.
≈ [ #{s1∈World : square(s1.shape)∧ green(s1.color)}
#{s2∈World : square(s2.shape)} > 1 2
] ∧ [ ∃s3 ∈ World : circle(s3.shape) ∧ blue(s3.color) ]
Figure 2: An example of a DMRS graph corresponding to a more complex caption, with compositional components colored. The logical formula gives the formal semantic interpretation over a world model.
This greatly facilitates the generation of a combinatorially large amount of captions and makes the DMRS graph patterns reusable, as outlined in section 4. Figure 2 furthermore gives a formal semantic interpretation of the caption meaning as logical formula over a world model. In essentially the same way, the agreement of a caption with a microworld is computed in the SHAPEWORLD framework.
Similar to the generator module, the captioner module also randomly samples from a set of dataset-specific DMRS graph patterns, which are then applied to a world model to construct an agreeing caption object. Such a caption object can be turned into natural language. Its ability to check whether another world model would agree with the semantics of this caption is important for the generation of negative instances, i.e., captions that do not agree with the corresponding microworld they are supposed to describe. These captions are obtained by sampling a second, false world model, extracting a caption object from it, and ensuring that it does not accidentally also agree with the first, true microworld.
3.4 Training and testing on SHAPEWORLD datasets
Since SHAPEWORLD datasets are actually data generation processes, training and evaluation work differently from classic datasets. Where usually one has a fixed set of test instances, here models are trained and tested on a fixed set of more abstract configuration constraints. In particular, in our experiments, the constraints for testing (and validation) differ from the training constraints, hence requiring true generalization abilities. This means that, for instance, a certain shapecolor-combination, a specific number of objects, a spatial location or a caption type can be held-out and never generated during training. The model is
presented with instances of this configuration only when evaluated and is hence required to recombine concepts it acquired from the training data. It is thus possible for a system to achieve optimal performance during training, but completely fail the evaluation. Another parameter is the ratio of correct instances, which may differ between training and evaluation. This adds yet another level of control to allow us to analyze the behavior of a model.
4 Some SHAPEWORLD datasets
In this paper we look at four datasets, each designed to investigate an aspect of the capability to understand language in a multimodal setup and to generalize to new instances not seen during training. Two example images of microworlds for each dataset are shown in Figure 3.
OneShape The first and simplest dataset requires the evaluated system to learn to separate the concepts of shape and color, instead of treating each combination as an atomic object. For this, microworlds are generated which contain a single object, together with a simple existential statement of the form: There is a [color] [shape]. One color-shape-combination, red square, is reserved for evaluation, which leaves 55 combinations for training.
MultiShape This dataset generates worlds with up to four objects, with the same type of existential statement as before, describing one of these objects. The system is evaluated on worlds containing five objects, requiring the ability to focus attention on one object without being distracted by other objects.
Spatial Microworlds with two objects are generated in this dataset accompanied by a statement about their relative location: to the left of, to the
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
ONESHAPE MULTISHAPE SPATIAL QUANTIFIER
Figure 3: Two example microworlds generated by each of the four presented SHAPEWORLD datasets.
right of, above or below. The overall caption is of the form: The [color] [shape] is [spatial] the [color] [shape]. As in the first dataset, a previously unseen attribute combination is presented for evaluation.
Quantifier The quantifier dataset generates microworlds of three to six objects, with seven used for evaluation. The captions focus on the quantifiers no, a, the, some, two, most, all and every. Compatibility with the caption is analyzed according to the quantifier’s usual cardinality-based interpretation in formal semantics. For instance, most is considered to be true if the target attribute applies to more than half of the domain set of objects. There are a variety of possible caption patterns in this dataset with different degrees of underspecification in domain and target, as illustrated in the examples below:
• Some shapes are green. • Most shapes are rectangles. • No shape is a red triangle. • All triangles are green. • Two blue shapes are pentagons.
Since we first sample a microworld model and subsequently a caption, here we cannot easily control the sampling process to uniformly sample each possible caption, as is trivially the case for the other datasets. Consequently, we restrict the number of different shapes and colors possible in a world, so that situations for the quantifiers all, every and most become more likely. Moreover, we adapt the probabilities of the sampling process so that we observe a more uniform distribution over caption patterns when sampling a large number of instances.
An important property of the SHAPEWORLD datasets is their compositionality. Instead of having to define a dataset from scratch every time, we can specify atomic datasets like the ones described above, and then combine them in a mixer dataset, which tests for various different aspects
of multimodal language understanding simultaneously. Reusability in fact applies even further down in the component hierarchy. For instance, we use the same generic world generator module for all four datasets. This is even more important for caption generation where, for instance, a logical combinator dataset can reuse different world captioner modules to generate simple statements which then are merged by logical connectives.
5 Experiments
As discussed in the introduction, the aim of our experiments is to demonstrate that SHAPEWORLD allows detailed investigation of neural network architectures, rather than high performance as such, although naturally a certain minimum performance is required for the results to be interesting.
5.1 Network architecture
We evaluate a generic multimodal deep neural network architecture shown in figure 4, which is inspired by recent work on visual question answering (Antol et al., 2015). The visual input is processed by a convolutional neural network (CNN) module (LeCun et al., 1989) with three layers of convolutions (filter sizes: 5, 3, 3; number of filters: 16, 32, 64) followed by a 2×2 max-pooling step, or average-pooling in the last layer, respectively. With an image size of 64×64, the obtained world embedding has 4096 dimensions. The textual input is transformed to 32-dimensional embeddings and passed on to a long short-term memory (LSTM) module (Hochreiter and Schmidhuber, 1997) with state size 64. The final state is used as the caption embedding, which is scaled and fused with the world embedding via pointwise multiplication. After a fully-connected hidden layer of 512 dimensions, yield the agreement score. The entire architecture is trained end-toend on the task. Both the CNN module and the word embeddings are included in the training, as opposed to using pre-trained, general-purpose versions.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
CNN CNN CNN
world embedding
LSTM LSTM LSTM LSTM LSTM
there is a green circle
caption embedding
word embeddings lookup
?
agreement embedding hidden layer agreement ∈ [0, 1]
scale
Figure 4: A schematic depiction of the multimodal deep neural network which we evaluate on the SHAPEWORLD datasets.
ONESHAPE
0.5
0.6
0.7
0.8
0.9
1
0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000
training evaluation
ONESHAPE (*)
0.5
0.6
0.7
0.8
0.9
1
0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000
training evaluation
MULTISHAPE
0.5
0.6
0.7
0.8
0.9
1
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000
training evaluation
SPATIAL
0.5
0.6
0.7
0.8
0.9
1
0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000
training evaluation
SPATIAL (*)
0.5
0.6
0.7
0.8
0.9
1
0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000
training evaluation
QUANTIFIER
0.5
0.6
0.7
0.8
0.9
1
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000
training evaluation
Figure 5: Performance of the evaluated multimodal deep neural network on the four SHAPEWORLD datasets, measured as accuracy of correctly judged image caption agreements. .
5.2 Training and evaluation setup
In all experiments, we use a training batch size of 128 where the ratio of correct instances is 33%. To track the performance of the model throughout the learning process (as opposed to just at the end), we add an evaluation step every 100th training iteration, where we obtain the accuracy on both training and evaluation instances. These numbers are calculated on basis of a batch of 1024 instances each, with a correct instance ratio of 50%, and are used to visualize and investigate the learning performance in our experiments.
5.3 Results
Figure 4 shows the performance results of our experiments. We observe a high accuracy on the ONESHAPE dataset on training instances, but an unexpectedly low and unstable performance on the
held-back instances. This indicates that the system does not naturally learn to distinguish shape and color as two separate concepts, but rather learns to classify each shape-colour-combination individually – after all, the ONESHAPE dataset essentially resembles a classification task. In a followup experiment, we reduced the embedding size to 8 and the LSTM state size to 16 dimensions (marked with (*) in figure 4). Being given less memory to learn simple non-generalizing classification, the model indeed achieves similarly high accuracy numbers in evaluation as during training.
The training performance on the MULTISHAPE dataset is relatively low and plateaus at around 80-85% after roughly 5000 iterations. The evaluation accuracy, in comparison, is not that much lower, indicating that the generalization to an unseen number of objects is learned, to some degree.
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
On the SPATIAL dataset, unlike on ONESHAPE, the system successfully learns the shape-colordistinction from the start. Note that each caption here involves two object descriptions, so instead of simply learning classification, it is necessary to disentangle the words of the caption. The fact that the evaluation accuracy is higher than the training accuracy in the beginning is because the held-back combination red square is (unsurprisingly) easier to recognize than some others. Again, we also ran an experiment with reduced embedding and LSTM size. Contrary to ONESHAPE, the accuracy curves indicate that this impedes the learning of an appropriate generalization ability for the more complex SPATIAL task.
The QUANTIFIER dataset confirms the lower accuracy of captions focusing on parts of a world which contains multiple shapes. The training accuracy is around 5% lower than for MULTISHAPE and shows no substantial difference to the validation accuracy, implying that quantifiers, expectedly, are more difficult to learn than existential statements, but definitely possible to some degree.
5.4 Discussion
We want to emphasize the fact that the training accuracy here represents an interesting measure on its own. Contrary to the common evaluation methodology, where training performance is less expressive after multiple iterations over the training data, each instance is randomly generated anew in our setting. Consequently, even the training accuracy reflects a learning process, and overfitting happens on a conceptual/structural level instead of a numerical correlation level. Investigations of overfitting in this sense are not easily possible with non-artificial data.
Another type of investigation of what a model has learned involves manipulation of the evaluation data. For instance, we analyzed the fact that the network only achieves around 95% accuracy on the supposedly simple ONESHAPE dataset. Looking at a few misclassifications, we found that sometimes the negative instances are problematic where either only the shape or only the color attribute differed from what the caption stated. We thus created a modified version of the ONESHAPE dataset which generates only negative instances of this structure. Applying the trained model to this dataset indeed gave a training accuracy of around 80% and an evaluation accuracy of only 55-60%.
Changing the network parameters altered the performance for the ONESHAPE and SPATIAL datasets in interestingly different ways. In general, when we changed parameters in our experiments, we often observed either no learning at all, less (or no) improvement in evaluation accuracy compared to training accuracy, or similar curves for both numbers. Given our aim to set up capability unit-tests, such discrete, to some degree invariant outcomes are what we ideally expect – a behavior we intend to investigate further. Another aspect, which might turn out interesting in more complex experiments is the shape of the accuracy curves.
6 Conclusion and future work
We have presented a new test methodology and framework, SHAPEWORLD, for multimodal deep learning models with a focus on formal-semantic style generalization capabilities. In this framework, artificial data is automatically generated, and datasets specify parameters of this generation process to create appropriate data, which targets a specific multimodal understanding task. We evaluated a neural network architecture on four image caption agreement datasets, where the system has to decide whether a statement applies to an image. We show how the SHAPEWORLD framework can be used to investigate the learning process of deep neural networks, and that it clearly indicates whether certain abilities are acquired. There are clearly some deficiencies in the generalization ability of the evaluated network – one aim of this paper is to invite the research community to evaluate their architectures on our datasets.
The SHAPEWORLD framework is still under development. In particular, we plan to add new datasets addressing other aspects of language, as well as integrating options to enhance the language generation module, with the aim of providing more varied and natural image descriptions. One option would be to integrate a subsequent step applying paraphrase rules after caption generation – Copestake et al. (2016) describe how this can be implemented on the level of DMRS graphs. In addition, the framework can easily be extended to produce more complex worlds, e.g., based on Cliparts (Zitnick et al., 2016). However, we plan to stick to the simplicity and abstractness of our microworlds for now, since we think they offer several advantages and enough richness for various interesting investigations.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths:  The authors introduce a new software package called ShapeWorld for automatically generating data for image captioning problems. The microworld used to generate the image captions is simple enough to make the data being generated and errors by a model readily interpretable. However, the authors demonstrate that configurations of the packages produce data that is challenging enough to serve as a good benchmark for ongoing research.  - Weaknesses:  The primary weakness of this paper is that it does look a bit like a demo paper. The authors do provides experiments that evaluate a reasonable baseline image captioning system on the data generated by ShapeWorld. However, similar experiments are included in demo papers.  The paper includes a hyperlink to the software package on github that presumably unmasks the authors of the paper.  - General Discussion:  Scientific progress often involves some something analogous to vygotsky's zone of proximal development, whereby progress can be made more quickly if research focuses on problems with just the right level of difficulty (e.g., the use of tidigits for speech recognition research in the early 90s). This paper is exciting since it offers a simple microworld that is easy for researchers to completely comprehend but that also is just difficult enough for existing models.  The strengths of the work are multiplied by the fact that the software is opensource, is readily available on github and generates the data in a format that can be easily used with models built using modern deep learning libraries (e.g., TensorFlow).  The methods used by the software package to generate the artificial data are clearly explained. It is also great that the authors did experiments with different configurations of their software and a baseline image caption model in order to demonstrate the strengths and weakness of existing techniques.   My only real concern with this paper is whether the community would be better served by placing it in the demo section. Publishing it in the non-demo long paper track might cause confusion as well as be unfair to authors who correctly submitted similar papers to the ACL demo track.