Verb Physics: Relative Physical Knowledge of Actions and Objects
Learning commonsense knowledge from natural language text is nontrivial due to reporting bias: people rarely state the obvious, e.g., “my house is bigger than me”. However, while rarely stated explicitly, this trivial everyday knowledge does influence the way people talk about the world, which provides indirect clues to reason about the world. For example, a statement like “John entered his house” implies that his house is bigger than John. In this paper, we present an approach to infer relative physical knowledge of actions and objects along six dimensions (e.g., size, weight, and strength) from unstructured natural language text. We frame knowledge acquisition as joint inference over two closely related problems: learning (1) relative physical knowledge of object pairs and (2) physical implications of actions when applied to those object pairs. Empirical results demonstrate that it is possible to extract knowledge of actions and objects from language and that joint inference over different knowledge types improves performance.
1 000
011
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033
034
035
036
037
038
039
040
041
042
043
044
045
046
047
048
049
061
062
063
064
065
066
067
068
069
070
071
072
073
074
075
076
077
078
079
080
081
082
083
084
085
086
089
090
091
092
093
094
095
096
097
098
099
In this paper, we present an approach to infer relative physical knowledge of actions and objects along six dimensions (e.g., size, weight, and strength) from unstructured natural language text. We frame knowledge acquisition as joint inference over two closely related problems: learning (1) relative physical knowledge of object pairs and (2) physical implications of actions when applied to those object pairs. Empirical results demonstrate that it is possible to extract knowledge of actions and objects from language and that joint inference over different knowledge types improves performance.
1 Introduction
Reading and reasoning about natural language text often requires trivial knowledge about everyday physical actions and objects. For example, given a sentence “Martin could fit the trophy into the suitcase”, we can trivially infer that the trophy must be smaller than the suitcase, even though it’s not stated explicitly. This reasoning requires knowledge about the action “fit”, in particular, typical preconditions that need to be satisfied in order to perform the action. In addition, reasoning about
Natural language clues
Relative physical knowledge about objects
Physical implications of actions
“She barged into the stable.”
HUMAN STABLE
size: smaller
weight: lighter speed: faster
strength: n/a
rigidness: less rigid
x barged into y
⇒ x is smaller than y
⇒ x is lighter than y
⇒ x is faster than y
⇒ x is less rigid than y
the applicability of various physical actions in a given situation often requires background knowledge about objects in the world, for example, that people are usually smaller than houses, that cars generally move faster than humans walk, or that a brick probably is heavier than a feather.
In fact, the potential use of such knowledge about everyday actions and objects can go beyond language understanding and reasoning. Many open challenges in computer vision and robotics may also benefit from such knowledge, as shown
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
in recent work that requires visual reasoning and entailment (Izadinia et al., 2015; Zhu et al., 2014). Ideally, an AI system should acquire such knowledge through direct physical interactions with the world. However, such a physically interactive system does not seem feasible in a foreseeable future.
In this paper, we present an approach to acquire trivial physical knowledge from unstructured natural language text as an alternative knowledge source. In particular, we focus on acquiring relative physical knowledge of actions and objects organized along five dimensions: size, weight, strength, rigidness, and speed. Figure 1 illustrates example knowledge of (1) relative physical relations on object pairs and (2) physical implications of actions when applied to those object pairs.
While natural language text is a rich source to obtain broad knowledge about the world, compiling trivial commonsense knowledge from unstructured text is a nontrivial feat. The central challenge lies in reporting bias: people rarely states the obvious (Gordon and Van Durme, 2013; Sorower et al., 2011), since it goes against Grice’s conversational maxim on the quantity (Grice, 1975) of information.
In this work, we demonstrate that it is possible to overcome reporting bias and still extract the unspoken knowledge from language. The key insight is this: there is consistency in the way people describe how they interact with the world, which provides vital clues to reverse engineer the common knowledge shared among people. More concretely, we frame knowledge acquisition as joint inference over two closely related puzzles: inferring relative physical knowledge about object pairs while simultaneously reasoning about physical implications of actions.
Importantly, four of five dimensions of knowledge in our study — weight, strength, rigidness, speed — are either not visual or not easily recognizable from image recognition using currently available computer vision techniques. Thus, our work provides unique values to complement recent attempts to acquire commonsense knowledge from web images (Izadinia et al., 2015; Bagherinezhad et al., 2016; Sadeghi et al., 2015).
In sum, our contributions are threefold:
• We introduce a new task of commonsense knowledge extraction from language, focusing on physical implications of actions and relative physical relations among objects, or-
ganized along five dimensions. • We propose a model that can infer relations
over grounded object pairs together with first order relations implied by physical verbs. • We develop a new dataset VERBPHYSICS
that compiles crowd-sourced knowledge of actions and objects.1
The rest of the paper is organized as follows. We first provide the formal definition of knowledge we aim to learn in Section 2. We then describe our data collection in Section 3 and present our inference model in Section 4. Empirical results and related work follow in Sections 5 and Section 6. We conclude at Section 7.
2 Representation of Relative Physical Knowledge
2.1 Knowledge Dimensions
We consider five dimensions of relative physical knowledge in this work: size, weight, strength, rigidness, and speed. “Strength” in our work refers to the physical durability of an object (e.g., “diamond” is stronger than “glass”), while “rigidness” refers to the physical flexibility of an object (e.g., “glass” is more rigid than a “wire”). When considered in verb implications, size, weight, strength, rigidness generally concerns pre-conditions of the action, while speed concerns the post-condition of the action.
2.2 Relative physical knowledge
Let us first consider the problem of representing relative physical knowledge between two objects. We can write a single piece of knowledge like “A person is larger than a basketball” as
person >size basketball
Any propositional statement can have exceptions and counterexamples. Moreover, we need to cope with uncertainties involved in knowledge acquisition. Therefore, we assume each knowledge piece is associated with a probability distribution. More formally, given objects x and y, we define a random variable Oax,y whose range is {>, <,'} with respect to a knowledge dimension a ∈ {SIZE,WEIGHT,STRENGTH,RIGIDNESS,SPEED} so that:
P (Oax,y = r), r ∈ {>, <,'}. 1Will be publicly shared at Anonymized.URL.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
approx max width
action
theme Iagent,theme agent goal Itheme,goal
Iagent,goal
“He threw the ball”
Iagent,theme
x threw y
⇒ x is larger than y ⇒ x is heavier than y ⇒ x is slower than y
“We walked into the house”
x walked into y
⇒ x is smaller than y ⇒ x is lighter than y ⇒ x is faster than y
Iagent,goal “I squashed the bug with my boot”
squashed x with y
⇒ x is smaller than y ⇒ x is lighter than y
⇒ x is weaker than y
Itheme,goal
⇒ x is less rigid than y
⇒ x is slower than y
Figure 2: Example physical implications represented as frame relations between a pair of arguments.
This immediately provides two simple properties:
P (Ox,y = >) = P (Oy,x = <) P (Ox,x = ') = 1
2.3 Physical Implications of Verbs
Next we consider representing relative physical implications of actions applied over two objects. For example, consider an action frame “x threw y.” In general, following implications are likely to be true:
“x threw y” =⇒ x >size y “x threw y” =⇒ x >weight y
Again, in order to cope with exceptions and uncertainties, we assume a probability distribution associated with each implication. More formally, we define a random variable F av to denote the implication of the action verb v when applied over its arguments x and y with respect to a knowledge dimension a so that:
P (F sizethrew = >) := P (“x threw y”⇒ x >size y) P (Fwgtthrew = >) := P (“x threw y”⇒ x >wgt y)
where the range of F sizethrew is {>, <,'}. Intuitively, F sizethrew represents the likely first order relation implied by “throw” over ungrounded (i.e., variable) object pairs.
Above definition assumes that there is only a single implication relation for any given verb with
respect to a specific knowledge dimension. This is generally not true, since a verb, especially a common action verb, can often invoke a number of different frames according to frame semantics (Fillmore, 1976). Thus, given a number of different frame relations v1...vT associated with a verb v, we define random variables F with respect to a specific frame relation vt, i.e., F avt . We use this notation going forward.
Frame Perspective on Verb Implications: Figure 2 illustrates the frame centric view on physical implication knowledge we aim to learn. Importantly, the key insight of our work is inspired by Fillmore’s original manuscript on frame semantics (Fillmore, 1976). Fillmore has argued that “frames”—the contexts in which utterances are situated—should be considered as a third primitive of describing a language, along with a grammar and lexicon. While existing frame annotations such as FrameNet (Baker et al., 1998), PropBank (Palmer et al., 2005), and VerbNet (Schuler, 2005) provide a rich frame knowledge associated with a predicate, none of them provide the exact kind of physical implications we consider in our paper. Thus, our work can potentially contribute to these resources by investigating new approaches to automatically recover richer frame knowledge from language.
3 Data and Crowd-sourced Knowledge
Action Verbs: We pick 50 classes of Levin verbs from both “alternation classes” and “verb
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
classes” (Levin, 1993), which corresponds to about 1100 unique verbs. We sort this list by frequency of occurrence in our frame patterns in the Google Syntax Ngrams corpus (Goldberg and Orwant, 2013) and pick the top 100 verbs.
Action Frames: Figure 2 illustrates examples of action frame relations. Because we consider implications over pairwise argument relations for each frame, there are sometimes multiple frame relations we consider for a single frame. To enumerate action frame relations for each verb, we use syntactic patterns based on dependency parse by extracting the core components (subject, verb, direct object, prepositional object) of an action, then map the subject to an agent, the direct object to a theme, and the prepositional object to a goal.2 For those frames that involve an argument in a prepositional phrase, we create a separate frame for each preposition based on the statistics observed in the Google Syntax Ngram corpus.
Because the syntax ngram corpus provides only tree snippets without context, this way of enumerating potential frame patterns tend to overgenerate. Thus we refine our prepositions for each frame by taking either the intersection or union with the top 5 Google Surface Ngrams (Michel et al., 2011), depending on whether the frame was under- or over-generating. We also add an additional crowdsourcing step where we ask crowd workers to judge whether a frame pattern with a particular verb and preposition could plausibly be found in a sentence. This process results in 813 frame templates, an average of 8.13 per verb.
Object Pairs: To provide a source of ground truth relations between objects, we select the object pairs that occur in the 813 frame templates with positive pointwise mutual information (PMI) across the Google Syntax Ngram corpus. After replacing a small set of “human” nouns with a generic HUMAN object, filtering out nouns labeled as abstract by WordNet (Miller, 1995), and distilling all surface forms to their lemmas (also with WordNet), the result is 3656 object pairs.
3.1 Crowdsourcing Knowledge
We collect human judgements of the frame knowledge implications to use as a small set of seed
2Future research could use an SRL parser instead. We use dependency parse to benefit from the Google Syntax Ngram dataset that provides language statistics over an extremely large corpus, which does not exist for SRL.
knowledge (5%), a development set (45%), and a test set (50%). Crowd workers are given with a frame template such as “x threw y,” and then asked to list a few plausible objects (including people and animals) for the missing slots (e.g., x and y).3 We then ask them to rate general relationship that the arguments of the frame exhibit with respect to the knowledge dimension we choose (size, weight, etc.). For each knowledge dimension, or attribute, a, workers select an answer from (1) x >a y, (2) x <a y, (3) x 'a y, or (4) no general relation.
We conduct a similar crowdsourcing step for the set of object pairs. We ask crowd workers to compare each of the 3656 object pairs along the five knowledge dimensions we consider, selecting an answer from the same options above as with frames. We reserve 50% of the data as a test set, and split the remainder up either 5% / 45% or 20% / 30% (seed / development) to investigate the effects of different seed knowledge sizes on the model.
Statistics for the dataset are provided in Table 1. About 90% of the frames as well as object pairs had 2/3 agreement between workers. After removing frame/attribute combinations and object pairs that received less than 2/3 agreement, or were selected by at least 2/3 workers to have no relation,
3This step is to prime them for thinking about the particular template; we do not use the objects they provided.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
we end up with roughly 400–600 usable frames and 2100–2500 usable object pairs per attribute.
4 Model
We model knowledge acquisition as probabilistic inference using a factor graph. As shown in Figure 3, the graph consists of multiple substrates (page-wide boxes) corresponding to different knowledge dimensions (shown only three of them —strength, size, weight—for brevity). Each substrate consists of two types of sub-graphs: verb subgraphs and object subgraphs, which are connected through factors that quantify action–object compatibilities. Connecting across substrates are factors that model inter-dependencies across different knowledge dimensions. In what follows, we describe each graph component.
4.1 Nodes
In the factor graph, we have two types of nodes in order to capture both classes of knowledge. The object first type of nodes are object pair nodes. Each object pair node is a random variable Oax,y which captures the relative strength of attribute a between objects x and y.
The second type of nodes are frame nodes. Each frame node is a random variable F avt . This corresponds to the verb v used in a particular type of frame t, and captures the implied knowledge the frame vt holds along an attribute a.
All random variables take on the values {>, <,'}. For an object pair node Oax,y, the value represents the belief about the relation between x and y along attribute a. For a frame node F avt , the value represents the belief about the relation along attribute a between any two objects that might be used in the frame vt.
We denote the sets of all object pair and frame random variables O and F , respectively.
4.2 Action–Object Compatibility
The key aspect of our work is to reason about two types of knowledge simultaneously: relative knowledge on grounded object pairs, and implications of actions related to those objects. Thus we connect the verb subgraphs and object subgraphs through selectional preference factors ψs between two such nodes Oax,y and F a vt if we find evidence from text that suggests objects x and y are used in the frame vt. These factors encourage both random variables to agree on the same value.
As an example, consider a node Osizep,b which represents the relative size of a person and a basketball, and a node F sizethrewdobj which represents the relative size implied by an “x threw y” frame. If we find significant evidence in text that “[person] threw [basketball]” occurs, we would add a selectional preference factor to connect Osizep,b with F sizethrewdobj and encourage them towards the same value. This means that if it is discovered that people are larger than basketballs (>), then we would expect the frame “x threw y” to entail x >size y (also >).
4.3 Semantic Similarities
Some frames have relatively sparse text evidences to support their corresponding knowledge acquisition. Thus, we include several types of factors based on semantic similarities as described below.
Cross-Verb Frame Similarity We add a group of factors ψv between two verbs v and u (to connect a specific frame of v with a corresponding frame of u) based on the verb-level similarities.
Within-Verb Frame Similarity Within each verb v, which consists of a set of frame relations v1, ...vT , we also include frame-level similarity factors ψf between vi and vj . This gives us more evidence over a broader range of frames when textual evidence might be sparse.
Object Similarity As with verbs, we add factors ψo that encourage similar pairs of objects to take the same value. Given that each node represents a pair of objects, finding that x and y are similar yields two main cases in how to add factors (aside from the trivial case where the variable Oax,y is given a unary factor to encourage the value ').
1. If nodes Ox,z and Oy,z exist, we expect objects x and y to both have a similar relation to z. We add a factor that encourages Ox,z and Oy,z to take the same value. The same is true if nodes Oz,x and Oz,y exist.
2. On the other hand, if nodesOx,z andOz,y exist, we expect these two nodes to reach the opposite decision. In this case, we add a factor that encourages one node to take the value > if the other prefers the value <, and vice versa. (For the case of ', if one prefers that value, then both should.)
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
vsizesquish
vsizethrow
vsizewalk
vweightthrow
vweightwalk … … … F sizethrow1 F sizethrow2 F sizethrow3 F sizethrow4
f a o v
s
f
v v
a
s
s v
a
a
frames for vsizethrow
o
o
s
h a rd
n e ss
random variable (RV)
group of RVs
factor connects RV
factors connect subset of RVs
verb similarity
frame similarity
object similarity
attribute similarity
selectional preference
f
a
o
v
s attribute subgraphs
subgraphs object
verb subgraphs
si z e
w e ig
h t
Osizes,t
Osizep,q
Osizeq,r
Osizep,s
st re
n g th
vstrengthsquish
Ostrengthp,t Ostrengthp,q
Figure 3: High level view of the factor graph model. Performance on both learning relative knowledge about objects (right), as well as entailed knowledge from verbs (center) via realized frames (left), is improved by modeling their interplay (orange). Unary seed (ψseed) and embedding (ψemb) factors are omitted for clarity.
4.4 Cross-Knowledge Correlation
Some knowledge dimensions, such as size and weight, have a significant correlation in their implied relations. For two such attributes a and b, if the same frame F avi and F b vi exists in both graphs, we add a factor ψa between them to push them towards taking the same value.
4.5 Seed Knowledge
In order to kick off learning, we provide a small set of seed knowledge among the random variables in {O,F} with seed factors ψseed. These unary seed factors push the belief for its associated random variable strongly towards the seed label.
4.6 Potential Functions
Unary Factors For all frame and object pair random variables in the training set, we train a maximum entropy classifier to predict the value of the variable. We then use the probabilities of the classifier as potentials for seed factors given to all random variables in their class (frame or object pair). Each log-linear classifier is trained separately per attribute on a featurized vector of the variable:
P (r|Xa) ∝ ewa·f(Xa)
The feature function is defined differently according to the node type:
f(Oap,q) := 〈g(p), g(q)〉 f(F avt) := 〈h(t), g(v), g(t)〉
Here g(x) is the GloVe word embedding for the word x (t is the frame relation’s preposition, and g(t) is simply set to the zero vector if there is no preposition) and h(t) is a one-hot vector of the frame relation type. We use GloVe vectors of 100 dimensions for verbs and 50 dimensions for objects and prepositions (the dimensions picked based on development set).
Binary Factors In the case of all other factors, we use a “soft 1” agreement matrix with strong signal down the diagonals: > ' <> 0.7 0.1 0.2
' 0.15 0.7 0.15 < 0.2 0.1 0.7
4.7 Inference
After our full graph is constructed, we use belief propagation to infer the assignments of frames and object pairs not in our training data. Each message µ is a vector x with probabilities for each value
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
x ∈ {>, <,'}. A message passed from a random variable v to a neighboring factor f is the product of the messages from its other neighboring factors:
µv→f (x) ∝ ∏
f ′∈N(v)\{f}
µf ′→v(x)
A message passed from a factor f with potential ψ to a random variable v is a marginalized belief about the value of v of the other neighboring random variables combined with its potential:
µf→v(x) ∝ ∑
x′∈x′\x
ψ(x′) ∏
v′∈N(f)\{v}
µv′→f (x ′ v′)
5 Experimental Results
Factor Graph Construction We first need to pick a set of frames and objects to determine our set of random variables. The frames are simply the subset of the frames that were crowdsourced in the given configuration (e.g., seed + dev), with “soft 1” unary seed factors (the gold label indexed row of the binary factor matrix) given only to those in the seed set. The same selection criteria and seed factors are applied to the crowdsourced object pairs.
For lexical similarity factors (ψv, ψo), we pick connections based on the cosine similarity scores of GloVe vectors (Pennington et al., 2014) thresholded above a value (0.4) chosen based on development set performance. Attribute similarity factors (ψa) are chosen based on sets of frames that reach largely the same decisions on the seed data (95%). Frame similarity factors (ψf ) are added to pairs of frames with linguistically similar constructions. (Both attribute and frame similarity factors did not help on development set performance and thus were omitted from the final models; their effects are shown in the ablations in Table 3.)
Finally, selectional preference factors (ψs) are picked by using positive pointwise mutual information (PMI) between the frames and the object pairs’ occurrences in the Google Syntax Ngram corpus.
Baselines Baselines include making a RANDOM choice, picking between >, <, and '), picking the MAJORITY label, and a maximum entropy classifier based on the embedding representation (EMB-MAXENT).
5.1 Inferring Knowledge on Actions
Our first experiment is to predict knowledge implied by new frames. We experiment with two different seed knowledge sets: OUR MODEL (A) is based on more skimp seed knowledge, taking only 5% of the object pair data as seed. In contrast, OUR MODEL (B) is based on 20% of the object pairs as seed knowledge.
The full results for the baseline methods and our model are given in Table 4.7. Though the speed attribute has a skewed label distribution, giving the majority baseline high performance, our model outperforms the baselines on other attributes as well as overall.
Metaphorical Language: While our frame patterns are intended to capture action verbs, our templates also match senses of those verbs that can be used with abstract or metaphorical arguments, rather than directly physical ones. One example from the development set is “x contained y.” While x and y can be real objects, more abstract senses of “contained” could involve y as a “forest fire” or even a “revolution.” In these instances, x >size y is plausible as an abstract notion: if some entity can contain a revolution, we might think that entity as “larger” or “stronger” than the revolution.
5.2 Inferring Knowledge on Objects
Our second experiment is to predict the correct relations of new object pairs. Results for this task are also given in Table 4.7.
6 Related work
Several works straddle the gap between IE, knowledge base completion, and learning commonsense knowledge from text. Earlier works in these areas use large amounts of text to try to extract general statements like “A THING CAN BE READABLE” (Gordon et al., 2010) and frequencies of events (Gordon and Schubert, 2012). Our work focuses on specific domains of knowledge rather than general statements or occurrence statistics, and develops a frame- centric approach to circumvent reporting bias. Other work uses a knowledge base and scores unseen tuples based on similarity to existing ones (Angeli and Manning, 2013; Li et al., 2016). Relatedly, previous work uses natural language inference to infer new facts from a dataset of commonsense facts that can be ex-
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Development Test Algorithm size weight stren rigid speed overall size weight stren rigid speed overall RANDOM 0.33 0.33 0.33 0.33 0.33 0.33 0.33 0.33 0.33 0.33 0.33 0.33
MAJORITY 0.38 0.41 0.42 0.18 0.83 0.43 0.35 0.35 0.43 0.20 0.88 0.44 EMB-MAXENT 0.62 0.64 0.60 0.83 0.83 0.69 0.55 0.55 0.59 0.79 0.88 0.66 OUR MODEL (A) 0.71 0.63 0.61 0.82 0.83 0.71 0.55 0.55 0.55 0.79 0.89 0.65 OUR MODEL (B) 0.74 0.69 0.67 0.82 0.78 0.74 0.76 0.59 0.66 0.80 0.87 0.73
Development Test Algorithm size weight stren rigid speed overall size weight stren rigid speed overall RANDOM 0.33 0.33 0.33 0.33 0.33 0.33 0.33 0.33 0.33 0.33 0.33 0.33
MAJORITY 0.50 0.54 0.51 0.50 0.53 0.51 0.51 0.55 0.52 0.49 0.50 0.51 EMB-MAXENT 0.68 0.66 0.64 0.67 0.65 0.66 0.71 0.67 0.64 0.65 0.63 0.66 OUR MODEL (A) 0.74 0.69 0.67 0.68 0.66 0.69 0.68 0.70 0.66 0.66 0.60 0.66
Table 2: Accuracy of baselines and our model on both tasks. Top: frame prediction task, OUR MODEL (A) uses 5% object pairs as seed, OUR MODEL (B) uses 20% object pairs as seed. Bottom: object pair prediction task (5% object pairs as seed).
Ablated (or added) component Accuracy – Verb similarity 0.69 + Frame similarity 0.62 – Action-object compatibility 0.62 – Object similarity 0.70 + Attribute similarity 0.62 – Frame embeddings 0.63 – Frame seeds 0.62 – Object embeddings 0.62 – Object seeds 0.62 OUR MODEL (A) 0.71
Table 3: Ablation results on size attribute for the frame prediction task on the development dataset for OUR MODEL (A) (5% of the object pairs as seed data). Our final model (OUR MODEL (B), using 20% object pairs as seed data), employs only unary embeddings and selctional preference factors.
tracted from unstructured text (Angeli and Manning, 2014). While our work also focuses on commonsense knowledge, we attempt to directly learn a small number of specific types of knowledge from text without reasoning from an existing database or dataset of facts.
A handful of works have attempted to learn the types of knowledge we address in this work. One recent work tried to directly predict several binary attributes (such “is large” and “is yellow”) from on off-the-shelf word embeddings, noting that accuracy was very low (Rubinstein et al., 2015). Another line of work addressed grounding verbs in the context of robotic tasks. One paper in this line acquires verb meanings by observing state changes in the environment (She and Chai, 2016). While they have data on 165 verb frames, they only have enough to report statistical signifi-
cance on four of them. Another work in this line does a deep investigation of eleven verbs, modeling their physical effect via annotated images along eighteen attributes (Gao et al., 2016). These works are encouraging investigations into multimodal groundings of a small set of verbs. Our work instead grounds into a fixed set of attributes but leverages language on a broader scale to learn about more verbs in more diverse set of frames.
7 Conclusion
We presented a novel take on verb-centric frame semantics to learn implied physical knowledge latent in verbs. We showed that by modeling changes in physical attributes entailed by verbs together with objects that exhibit these properties, we are able to better solve both tasks. Experiments on our novel dataset confirm that a model which takes advantage of physical relations as they arise from verbs and between objects outperforms baselines lacking such information. Our dataset and code will be made publicly available upon publication.

Thank you for the author response. It addresses some my concerns, though much of it are promises ( we will... ) -- necessarily so, given space constraints, but then, this is precisely the problem: I would like to see the revision to the paper to be able to check that the drawbacks have been fixed. The changes needed are quite substantial, and the new experimental results that they promise to include will not have undergone review if the paper is accepted at this stage. I'm still not sure that we can simply leave it to the authors to make the necessary changes without a further reviewing round. I upgrade my score to a 3 to express this ambivalence (I do like the research in the paper, but it's extremely messy in its presentation).  --------------  - Strengths:  The topic of the paper is very creative and the purpose of the research really worthwhile: the paper aims at extracting common knowledge from text, overcoming the well-known problem of reporting bias (the fact that people will not state the obvious, such as the fact that a person is usually bigger than a ball), by doing joint inference on information that is possible to extract from text.  - Weaknesses:  1) Many aspects of the approach need to be clarified (see detailed comments below). What worries me the most is that I did not understand how the approach makes knowledge about objects interact with knowledge about verbs such that it allows us to overcome reporting bias. The paper gets very quickly into highly technical details, without clearly explaining the overall approach and why it is a good idea.  2) The experiments and the discussion need to be finished. In particular, there is no discussion of the results of one of the two tasks tackled (lower half of Table 2), and there is one obvious experiment missing: Variant B of the authors' model gives much better results on the first task than Variant A, but for the second task only Variant A is tested -- and indeed it doesn't improve over the baseline.   - General Discussion:  The paper needs quite a bit of work before it is ready for publication.   - Detailed comments:  026 five dimensions, not six  Figure 1, caption:  implies physical relations : how do you know which physical relations it implies?  Figure 1 and 113-114: what you are trying to do, it looks to me, is essentially to extract lexical entailments (as defined in formal semantics; see e.g. Dowty 1991) for verbs. Could you please explicit link to that literature?  Dowty, David.  Thematic proto-roles and argument selection.  Language (1991): 547-619.  135 around here you should explain the key insight of your approach: why and how does doing joint inference over these two pieces of information help overcome reporting bias?  141  values  ==>  value ?  143 please also consider work on multimodal distributional semantics, here and or in the related work section. The following two papers are particularly related to your goals:  Bruni, Elia, et al.  Distributional semantics in technicolor.  Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1. Association for Computational Linguistics, 2012.  Silberer, Carina, Vittorio Ferrari, and Mirella Lapata.  Models of Semantic Representation with Visual Attributes.  ACL (1). 2013.  146 please clarify that your contribution is the specific task and approach -- commonsense knowledge extraction from language is long-standing task.  152 it is not clear what  grounded  means at this point  Section 2.1: why these dimensions, and how did you choose them?  177 explain terms  pre-condition  and  post-condition , and how they are relevant here  197-198 an example of the full distribution for an item (obtained by the model, or crowd-sourced, or  ideal ) would help.  Figure 2. I don't really see the  x is slower than y  part: it seems to me like this is related to the distinction, in formal semantics, between stage-level vs. individual-level predicates: when a person throws a ball, the ball is faster than the person (stage-level) but it's not true in general that balls are faster than people (individual-level). I guess this is related to the pre-condition vs. post-condition issue. Please spell out the type of information that you want to extract.  248  Above definition : determiner missing  Section 3   Action verbs : Which 50 classes do you pick, and you do you choose them? Are the verbs that you pick all explicitly tagged as action verbs by Levin?   306ff What are  action frames ? How do you pick them?  326 How do you know whether the frame is under- or over-generating?  Table 1: are the partitions made by frame, by verb, or how? That is, do you reuse verbs or frames across partitions? Also, proportions are given for 2 cases (2 3 and 3 3 agreement), whereas counts are only given for one case; which?  336  with... PMI : something missing (threshold?)  371 did you do this partitions randomly?  376  rate *the* general relationship   378  knowledge dimension we choose : ? (how do you choose which dimensions you will annotate for each frame?)  Section 4  What is a factor graph? Please give enough background on factor graphs for a CL audience to be able to follow your approach. What are substrates, and what is the role of factors? How is the factor graph different from a standard graph? More generally, at the beginning of section 4 you should give a higher level description of how your model works and why it is a good idea.  420  both classes of knowledge : antecedent missing.  421  object first type   445 so far you have been only talking about object pairs and verbs, and suddenly selectional preference factors pop in. They seem to be a crucial part of your model -- introduce earlier? In any case, I didn't understand their role.  461  also ?  471 where do you get verb-level similarities from?  Figure 3: I find the figure totally unintelligible. Maybe if the text was clearer it would be interpretable, but maybe you can think whether you can find a way to convey your model a bit more intuitively. Also, make sure that it is readable in black-and-white, as per ACL submission instructions.  598 define term  message  and its role in the factor graph.  621 why do you need a  soft 1  instead of a hard 1?  647ff you need to provide more details about the EMB-MAXENT classifier (how did you train it, what was the input data, how was it encoded), and also explain why it is an appropriate baseline.  654  more skimp seed knowledge : ?  659 here and in 681, problem with table reference (should be Table 2).   664ff I like the thought but I'm not sure the example is the right one: in what sense is the entity larger than the revolution? Also,  larger  is not the same as  stronger .  681 as mentioned above, you should discuss the results for the task of inferring knowledge on objects, and also include results for model (B) (incidentally, it would be better if you used the same terminology for the model in Tables 1 and 2)  778  latent in verbs : why don't you mention objects here?  781  both tasks : antecedent missing  The references should be checked for format, e.g. Grice, Sorower et al for capitalization, the verbnet reference for bibliographic details.


Combining distributional and referential information for naming objects through cross-modal mapping and direct word prediction
We compare three recent models of referential word meaning that link visual object representations to lexical representations in a distributional vector space, either directly through cross-modal mapping or indirectly through visual predictors for individual words. We use these models to predict object names as they could be used in naturalistic referring expressions. We find that cross-modal mapping generally produces semantically appropriate and mutually highly similar object names in its topn list, but sometimes fails to make desired distinctions. Visual word predictors, on the other hand, can react to more subtle visual distinctions and select specific terms, but sometimes stray taxonomically very far from the correct one. Combination of the approaches improves over the individual predictions in a standard naming task. All approaches can be extended to the zero-shot naming case, where the correct name is one for which no instances were seen during training; again they show complementary strengths and weaknesses, depending on the setup and the lexical relation of the unattested object name to known ones.
1 Introduction
Expressions referring to objects in visual scenes typically include a word naming the type of the object: E.g., “house” in Figure 1 (a), or, as a very general type, “thingy” in Figure 1 (d). Determining such a name is is a crucial step for referring expression generation (REG) systems, as many other decisions, concerning e.g. the selection of attributes, follow from it (Dale and Reiter,
1995; Krahmer and Van Deemter, 2012). For a long time, however, research on REG mostly assumed the availability of symbolic representations of referent and scene, and sidestepped questions about how speakers actually choose these names, due to the lack of models capable of capturing what a word like house refers to in the real world.
Recent advances in image processing promise to fill this gap, with state-of-the-art computer vision systems being able to classify images into thousands of different categories (eg. Szegedy et al. (2015)). However, classification is not naming (Ordonez et al., 2016). Classification schemes are typically designed to be “flat”, with labels being on the same ontological level and, ideally, having disjunct extensions. In contrast, humans seem to be more flexible as to the chosen level of generality. Depending on the prototypicality of the object to name, and possibly other visual properties, a general name might be more or less appropriate. For instance, a robin can be named “bird”, but a penguin is better referred to as “penguin” (Rosch, 1978); along the same lines, the rather unusual building in Figure 1 that is not easy to otherwise
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
categorise was named “structure”. Other work at the intersection of image and language processing has investigated models that learn to directly associate visual objects with a representation of word meaning, for example through cross-modal transfer into distributional vector spaces. Under the assumption that such semantic spaces represent, in some form at least, taxonomic knowledge, this makes labels on different levels of specificity available for a given object. Moreover, if the mapping is sufficiently general, it should be able to map objects to an appropriate label, even if during training of the mapping this label has not been seen (zero-shot learning). While indeed performing with some promise on this task (Lazaridou et al., 2014), this approach does not generally outperform standard object classification with known categories (Frome et al., 2013; Norouzi et al., 2013).
This paper pursues the hypothesis that an accurate model of referential word meaning does not need to fully integrate visual and lexical knowledge (e.g. as expressed in a distributional vector space), but at the same time, has to go beyond treating words as independent labels. We extend upon work on learning models of referential word use from corpora of images paired with referring expressions (Schlangen et al., 2016; Anonymous, in press) that treats words as individual predictors capturing referential appropriateness. We explore different ways of linking these predictors to distributional knowledge, during application and during training. We find that these improve over direct cross-modal mapping and direct visual classification in a standard and a zero-shot setup of an object naming task, as they allow for a more flexible combination of lexical and visual information when modeling referential meaning.
2 Related Work
Grounding and Reference An early example for work in REG that goes beyond Dale and Reiter (1995)’s dominant symbolic paradigm is Deb Roy’s work from the early 2000s (Roy et al., 2002; Roy, 2002, 2005). More recently, research on REG, which has traditionally been done on small toy data sets, is being scaled up to realworld images (Kazemzadeh et al., 2014; Gkatzia et al., 2015; Zarrieß and Schlangen, 2016; Mao et al., 2015). In this paper, we focus on a particular problem posed by REG on real-world images,
namely generating the appropriate head noun for a given object. Similarly, Ordonez et al. (2016) have studied the problem of deriving appropriate object names, or so-called entry-level categories, from the output of an object recognizer. Their approach focusses links abstract object categories in ImageNet to actual words via various translation procedures. We are interested in learning referential appropriateness and extensional word meanings directly from actual human referring expressions (REs) paired with objects in images, using an existing object recognizer for feature extraction.
Multi-modal and cross-modal distributional semantics Distributional semantic models are a well-known method for capturing lexical word meaning in a variety of tasks (Turney and Pantel, 2010; Mikolov et al., 2013; Erk, 2016). Recent work on multi-modal distributional vector spaces (Feng and Lapata, 2010; Silberer and Lapata, 2014; Kiela and Bottou, 2014; Lazaridou et al., 2015b; Kottur et al., 2016) has aimed at capturing semantic similarity even more accurately by integrating distributional and perceptual features associated with words (mostly taken from images) into a single representation. More related to our work are cross-modal mapping models (Socher et al., 2013; Frome et al., 2013; Norouzi et al., 2013; Lazaridou et al., 2014), that learn to transfer a representation of an object or image in the visual space to a vector in a distributional space. When tested on standard object recognition tasks, transfer, however, comes at a price. Frome et al. (2013) and Norouzi et al. (2013) both find that it slightly degrades performance as compared to a plain object classification using standard accuracy metrics (called flat “hit @k metric” in their paper). Interestingly though, Frome et al. (2013) report better performance using “hierarchical precision”, which essentially means that transfer predicts words that are ontologically closer to the gold label and makes “semantically more reasonable errors”. To the best of our knowledge, this pattern has not been systematically investigated any further. Another known problem with cross-modal transfer is that it seems to generalize less well than expected, i.e. tends to reproduce word vectors observed during training (Lazaridou et al., 2015a). In this work, we present a model that exploits distributional knowledge for learning referential word meaning as well, but explore and compare different ways of combining visual and lexical aspects of referential word meaning.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
3 Task and Data
We define object naming as follows: Given an object x in an image, the task is to predict a word w that could be used as the head noun of a realistic referring expression. (Cf. discussion above: “bird” when naming a robin, but “penguin” when naming a penguin.) To get at this, we develop our approach using a corpus of referring expressions produced by human users under natural, interactive conditions (Kazemzadeh et al., 2014), and train and test on the corresponding head nouns in these REs. This is similar to picture naming setups used in psycholinguistic research (cf. Levelt et al. (1991)) and based on the simplifying assumption that the name used for referring to an object can be determined successfully without looking at other objects in the image.
We now summarise the details of our setup:
Corpus We train and test on the REFERIT corpus (Kazemzadeh et al., 2014), which is based on the SAIAPR image collection (Grubinger et al., 2006) (99.5k image regions;120K REs). We follow (Schlangen et al., 2016) and select words with a minimum frequency of 40 in these two data sets, which gives us a vocabulary of 793 words.
Names For most of our experiments, we only use a subset of this vocabulary, namely the set of object names. As the REs contain nouns that cannot be considered to be names (background, bottom, etc.), we extract from the semantically annotated portion of the REFERIT corpus a list of names which correspond to ‘entry-level’ nouns in terms of (Kazemzadeh et al., 2014). This gives us a list of 159 names. Thus, our experiments are on a smaller scale as compared to (Ordonez et al., 2016). Nevertheless, the data is challenging, as the corpus contains references to objects that fall outside of the object labeling scheme that available object recognition systems are typically optimized for, cf. Hu et al. (2015)’s discussion on “stuff” entities such “sky” or “grass” in the REFERIT data. For testing, we remove relational REs (containing a relational preposition such as ‘left of X’), because here we cannot be sure that the head noun of the target is fully informative; we also remove REs with more than one head noun from our list (i.e. these are mostly relational expressions as well such as ‘girl laughing at boy’). We pair each image region from the test set with its corresponding names from the remaining REs.
Image and Word Embeddings Following Schlangen et al. (2016), we derive representations of our visual inputs with a convolutional neural network, ‘GoogleNet’ (Szegedy et al., 2015), which was trained on the ImageNet corpus (Deng et al., 2009), and extract the final fully-connected layer before the classification layer, to give us a 1024 dimensional representation of the region. We add 7 features that encode information about the region relative to the image, thus representing each object as a vector of 1031 features. As distributional word vectors, we use the word2vec representations provided by Baroni et al. (2014) (trained with CBOW, 5-word context window, 10 negative samples, 400 dimensions).
4 Three Models of Interfacing Visual and
Distributional Information
4.1 Direct Cross-Modal Mapping
Following e.g. Lazaridou et al. (2014), referential meaning can be represented as a translation function that projects visual representations of objects to linguistic representations of words in a distributional vector space. Thus, in contrast to standard object recognition systems or the other models we will use here, cross-modal mapping does not treat words as individual labels or classifiers, but learns to directly predict continuous representations of words in a vector space, such as the space defined by the word2vec embeddings that we use in this work. This model will be called TRANSFER below.
During training, we pair each object with the distributional embedding of its name, and use standard Ridge regression for learning the transformation. Lazaridou et al. (2014) and Lazaridou et al. (2015a) test a range of technical tweaks and different algorithms for cross-modal mapping. For ease of comparison with other models, we stick with simple Ridge Regression in this work.
For decoding, we map an object into the distributional space, and retrieve the nearest neighbors of the predicted vector using cosine similarity. In theory, the model should generalize easily to words that it has not observed in a pair with an object during training as it can map an object anywhere in the distributional space.
4.2 Lexical Mapping Through Individual Word Classifiers
Another approach is to keep visual and distributional information separate, by training a separate visual classifier for each word w in the vocabu-
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
lary. Predictions can then be mapped into distributional space during application time via the vectors of the predicted words. Here, we use Schlangen et al. (2016)’s WAC model, building the training set for each word w as follows: all visual objects in a corpus that have been referred to as w are used as positive instances, the remaining objects as negative instances. Thus, the classifiers learn to predict referential appropriateness for individual words based on the visual features of the objects they refer to, in isolation of other words.
During decoding, we apply all word classifiers from the model’s vocabulary to the given object, and take the argmax over the individual word probabilities. The model can be used to predict names directly, without links into a distributional space.
In order to extend the model’s vocabulary for zero-shot learning, we follow Norouzi et al. (2013) and associate the top n words with their corresponding distributional vector and compute the convex combination of these vectors. Then, in parallel to cross-modal mapping, we retrieve the nearest neighbors of the combined embedding from the distributional space. Thus, with this model, we use two different modes of decoding: one that projects into distributional space, one that only applies the available word classifiers.
4.3 Word Prediction via Cross-Modal Similarity Mapping
Finally, we implement an approach that combines ideas from cross-modal mapping with the WAC model: we train individual predictors for each word in the vocabulary, but, during training, we exploit lexical similarity relations encoded in a distributional space. Instead of treating a word as a binary classifier, we annotate its training instances with a fine-grained similarity signal according to their object names. When building the training set for such a word predictor w, instead of simply dividing objects into w and ¬w instances, we label each object with a real-valued similarity obtained from cosine similarity between w and v in a distributional vector space, where v is the word that was used to refer to the object. Thus, we task the model with jointly learning similarities and referential appropriateness, by training it with Ridge regression on a continuous output space. Object instances where v = w (i.e., the positive instances in the binary setup) have maximal similarity; the
remaining instances have a lower value which is more or less close to maximal similarity. This is the SIM-WAP model, recently proposed in (Anonymous).
Importantly, and going beyond (Anonymous), this model allows for an innovative treatment of words that only exist in a distributional space (without being paired with visual referents in the image corpus): as the predictors are trained on a continuous output space, no genuine positive instances of a word’s referent are needed. When training a predictor for such a word w, we use all available objects from our corpus and annotate them with the expected lexical similarity between w and the actual object names v, which for all objects will be below the maximal value that marks genuine positive instances. During decoding, this model does not need to project its predictions into a distributional space, but it simply applies all available predictors to the object, and takes the argmax over the predicted referential appropriateness scores.
5 Experiment 1: Naming Objects
This Section reports on experiments in a standard setup of the object naming task where all object names are paired with visual instances of their referents during training. In a comparable task, i.e. object recognition with known object categories, cross-modal projection or transfer approaches have been reported to perform worse than standard object classification methods (Frome et al., 2013; Norouzi et al., 2013). This seems to suggest that lexical or at least distributional knowledge is detrimental when learning what a word refers to in the real world and that referential meaning should potentially be learned from visual object representation only.
5.1 Model comparison
Setup We use the train/test split of REFERIT data as in (Schlangen et al., 2016). We consider image regions with non-relational referring expressions that contain at least one of the 159 head nouns from the list of entry-level nouns (see section 3). This amounts to 6208 image regions for testing and 73K instances for training.
Results Table 1 shows accuracies in the object naming task for the TRANSFER, WAC and SIMWAP models according to their accuracies in the top n, including two variants of WAC where its top
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
5 and top 10 predictions are project into the distributional space. Overall, the differences in accuracy between the models are small, but the various models that link their predictions to word representations in the distributional space all perform slightly worse than the plain WAC model, i.e. individual word classifiers trained on visual features only. This suggests that referential meanings for a word are learned less accurately when mapping from visual to distributional space, which replicates results reported in the literature on standard object recognition benchmarks.
hit @k(%) @1 @2 @5
transfer 48.34 60.49 74.89 wac 49.34 61.86 75.35 wac, project top5 48.73 61.10 74.07 wac, project top10 48.68 61.23 74.31 sim-wap 48.13 60.60 75.40
Table 1: Accuracies in object naming
5.2 Model combination
In order to get more insight into why the TRANSFER and SIM-WAP models produce slightly worse results than individual visual word classifiers, we now test to what extent the different models are complementary and combine them by aggregating over their naming predictions. If the models are complementary, their combination should lead to more confident and accurate naming decisions.
Setup We combine TRANSFER, SIM-WAP and WAC by aggregating the scores they predict for different object names for a given object. During testing, we apply all models to an image region and consider words ranked among the top 10. We first normalize the referential appropriateness scores in each top-10 list and then compute their sum. This aggregation scheme will give more weight to words that appear in the top 10 list of different models, and less weight to words that only get top-ranked by a single model. We test on the same data as in Section 5.1.
hit @k(%) 1 5 10
sim-wap + transfer 49.10 61.78 75.81 sim-wap + wac 51.10 63.45 77.92 transfer + wac 51.13 63.76 77.84 wac + transfer + sim-wap 52.19 64.71 78.40
Table 2: Object naming acc., combined models
Results Table 2 shows that all model combinations improve over the results of their isolated models in Table 1, suggesting that WAC, TRANSFER and SIM-WAP indeed do capture complementary aspects of referential word meaning. On their own, the distributionally informed models are less tuned to specific word occurrences than the visual word classifiers in the WAC model, but they can add useful information which leads to a clear overall improvement. We take this as a promising finding, supporting our initial hypothesis that knowledge on lexical distributional meaning should and can be exploited when learning how to use words for reference.
Av. cosine distance among top k gold - top k
5 10 5 10
transfer 0.68 0.73 0.72 0.75 wac 0.82 0.80 0.82 0.84 sim-wap 0.68 0.74 0.72 0.75
Table 3: Cosine distances between word2vec embeddings of nouns generated in the top k
5.3 Analysis
Figure 2 illustrates objects from our test set where the combination of TRANSFER, SIM-WAP and WAC predicts an accurate name, whereas the models in isolation do not. These examples give some interesting insight into why the models capture different aspects of referential word use and meaning.
Word Similarities Many of the examples in Figure 2 suggest that the object names ranked among the top 3 by the TRANSFER and SIMWAP model are semantically similar to each other, whereas WAC generates object names on top that describe very different underlying object categories, such as seal / rock in Figure 2(a), animal / lamp in Figure 2(g) or chair / shirt in Figure 2(c). To quantify this general impression, Table 3 shows cosine distances among words in the top n generated by our models, using their word2vec embeddings. The average cosine distance between words in our vocabulary is 0.83. The transfer and sim-wap model rank words on top that are clearly more similar to each other than word pairs on average, whereas words ranked top by the wac model are more dissimilar. This parallels findings by Frome et al. (2013), discussed in Section 2. Additional evaluation metrics, such as success rates
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
in a human evaluation (cf. Zarrieß and Schlangen (2016)), would be an interesting direction for more detailed investigation here.
Word Use But even though the WAC classifiers lack knowledge on lexical similarities, they seem to able to detect relatively specific instances of word use such as hut in Figure 2(b), shirt in 2(c) or lamp in 2(h). Here, the combination with TRANSFER and SIM-WAP is helpful to give more weight to the object name that is taxonomically correct (sometimes pushing up words below the top-3 and hence not shown in Figure 2). In Figure 1(e), SIMWAP and TRANSFER give more weight to typical names for persons, whereas WAC top-ranks more unusual names, reflecting that the person is difficult to identify visually. Another observation is that the mapping models have difficulties dealing with object names in singular and plural. As these words have very similar representations in the distributional space, they are often predicted as likely variants among the top 10 by SIM-WAP and TRANSFER, whereas the WAC model seems to predict inappropriate plural words less often among the top 3. Such specific phenomena at the intersection of visual and semantic similarity have found very little attention in the literature. We will investigate them further in our Experiments on zeroshot naming in the following Section.
6 Zero-Shot Naming
Zero-shot learning is an attractive prospect for REG from images, as it promises to overcome dependence on pairings of visual instances and natural names being available for all names, if visual/referential data can be generalised from other types of information. Previous work has looked at the feasibility of zero-shot learning as a function of semantic similarity or ontological closeness between unknown and known categories, and confirmed the intuition that the task is harder the less close unknown categories are to known ones (Frome et al., 2013; Norouzi et al., 2013).
Our experiments on object naming in Section 5 suggest that lexical similarities encoded in a distributional space might not always fully carry over to referential meaning. This could constitute an additional challenge for zero-shot learning, as distributional similarities might be misleading when the model has to fully rely on them for learning referential word meanings. Therefore, the following experiments investigate the performance of
our models in zero-shot naming as a function of the lexical relation between unknown and known object names, i.e. namely hypernyms and singular/plurals. Both relations are typically captured by distributional models of word meaning in terms of closeness in the vector space, but their visual and referential relation is clearly different.
6.1 Vocabulary Splits and Testsets
Random As in previous work on zero-shot learning, we consider zero-shot naming for words of varying degrees of similarity in our vocabulary. We randomly split our 159 names from Experiment 1 into 10 subsets. We train the models on 90% of the nouns (and all their visual instances in the image corpus) and test on the set of image regions that are named with words which the model did not observe during training. Results reported in Table 4 on the random test set correspond to averaged scores from cross-validation over the 10 splits.
Hypernyms We manually split the model’s vocabulary into set of hypernyms (see Appendix A) and the remaining nouns. We train the models on those 84K image regions that where not named with a hypernym, and test on 8895 image regions that were named with a hypernym in the corpus. We checked that for each of these hypernyms, the vocabulary contains at least one or two names that can be considered as hyponyms, i.e. the model sees objects during training that are instances of vehicle for example, but never encounters actual uses of that name. This test set is particularly interesting from an REG perspective, as objects named with very general terms by human speakers are often difficult to describe with more common, but more specific terms, as is illustrated by the uses of structure and thingy in Figure 1.
Singulars/Plurals We pick 68 words from our vocabulary that can be grouped into 34 singularplural noun pairs (see Appendix A). From each pair, we randomly include the singular or plural noun in the set of zero-shot nouns. Thus, we make sure that the model encounters singular and plural names during training, but it never encounters both variants of a name. This results in a more even training/test split, i.e. we train on 23K image regions and evaluate on 13825 instances.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
(a) wac: seal, rock, water sim-wap: side, rock,rocks transfer: rocks, rock, water combination: rock
(c) wac: chair, shirt, guy sim-wap: woman, man, girl transfer: door, woman, window combination: shirt
(e) wac: chick, person, guy sim-wap: man, person, woman transfer: man, guy, girl combination: person
(g) wac: animal, lamp, table sim-wap: man, girl, person transfer: man, clouds, cloud combination: person
(b) wac: cactus, hut, mountain sim-wap: side, rock, mountain transfer: mountain, rocks, rock combination: hut
(d) wac: roof, house, building sim-wap: building, house, trees transfer: building, house, trees combination: house
(f) wac: bush, bushes, tree sim-wap: trees, tree, grass transfer: trees, tree, bushes combination: bushes
(h) wac: post, light, lamp sim-wap: tree, sky, pole transfer: tree, sky, trees combination: lamp
Figure 2: Examples from object naming experiment where model combination is accurate
Zero-shot Model full vocab disjoint vocab names @1 @2 @5 @10 @1 @2
Random
transfer 0.05 2.38 16.57 35.71 41.49 62.34 wac, project top10 0.00 4.42 21.16 39.17 38.03 58.07 wac, project top5 0.00 4.39 21.63 40.01 37.46 57.36 sim-wap 3.71 13.13 36.49 54.44 42.28 64.26
Hypernyms
transfer 0.07 1.25 7.75 29.93 59.88 73.88 wac, project top10 0.00 3.01 15.55 36.99 50.51 66.33 wac, project top5 0.00 2.78 16.75 38.13 47.73 64.38 sim-wap 3.16 10.33 31.14 49.62 57.55 70.15
Singulars/Plurals transfer 0.01 22.84 44.30 72.85 34.56 51.79 wac, project top10 0.00 22.21 43.43 68.95 31.46 48.76 wac, project top5 0.00 22.18 43.93 69.33 31.46 48.88 sim-wap 15.39 34.73 56.62 77.32 37.24 54.02
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
6.2 Evaluation
Some previous work on zero-shot image labeling assumes additional components that first identify whether an image should be labelled by a known or unknown word (Frome et al., 2013). We follow Lazaridou et al. (2014) and let the model decide whether to refer to an object by a known or unknown name. Related to that, distinct evaluation procedures have been used in the literature on zero-shot learning:
Testing on full vocabulary A realistic way to test zero-shot learning performance is to consider all words from a given vocabulary during testing, though the testset only contains instances of objects that have been named with a ‘zero-shot word’ (for which no visual instances were seen during training). Accuracies in this setup reflect how well the model is able to generalize, i.e. how often it decides to deviate from the words it was trained on, and (implicitly) predicts that the given object requires a “new” name. In case of the (i) hypernym and (ii) singular/plural test set, this accuracy also reflects to what extent the model is able to detect cases where (i) a more general or vague term is needed, where (ii) an unknown singular/plural counterpart of a known object type occurs.
Testing on disjoint vocabulary Alternatively, the model’s vocabulary can be restricted during testing to zero-shot words only, such that names encountered during training and testing are disjoint, see e.g. (Lampert et al., 2009, 2013). This setup factors out the generalization problem, and assesses to what extent a model is able to capture the referential meaning of a word that does not have instances in the training data.
6.3 Results
As compared to Experiment 1 where models achieved similar performance, differences are more pronounced in the zero-shot setup, as shown in Table 4. In particular, we find that the SIMWAP model which induces individual predictors for words that have not been observed in the training data is clearly more successful than TRANSFER or WAC that project predictions into the distributional space. When tested on the full vocabulary, we find that TRANSFER and WAC very rarely generate names whose referents were excluded from training, which is in line with observations made by Lazaridou et al. (2015a). The SIM-WAP
predictors generalize much better, in particular on the singular/plural testset.
An interesting exception is the good performance of the TRANSFER model on the hypernym test set, when evaluated with a disjoint vocabulary. This corroborates evidence from Experiment 1, namely that the transfer model captures taxonomic aspects of object names better than the other models. Projection via individual word classifiers, on the other hand, seems to generalize better than TRANSFER, at least when looking at accuracies @2 ... @10. Thus, combining several vectors predicted by a model of referential word meaning can provide additional information, as compared to mapping an object to a single vector in distributional space. More work is needed to establish how these approaches can be integrated more effectively.
7 Discussion and Conclusion
In this paper, we have investigated models of referential word meaning, using different ways of combining visual information about a word’s referent and distributional knowledge about its lexical similarities. Previous cross-modal mapping models essentially force semantically similar objects to be mapped into the same area in the semantic space regardless of their actual visual similarity. We found that cross-modal mapping produces semantically appropriate and mutually highly similar object names in its top-n list, but does not preserve differences in referential word use (e.g. appropriatness of person vs. woman) especially within the same semantic field. We have shown that it is beneficial for performance in standard and zeroshot object naming to treat words as individual predictors that capture referential appropriateness and are only indirectly linked to a distributional space, either through lexical mapping during application or through cross-modal similarity mapping during training. As we have tested these approaches on a rather small vocabulary, which may limit generality of conclusions, future work will be devoted to scaling up these findings to larger test sets, as e.g. recently collected through conversational agents (Das et al., 2016) that circumvent the need for human-human interaction data. Also from an REG perspective, various extensions of this approach are possible, such as the inclusion of contextual information during object naming and its combination with attribute selection.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

COMMENTS AFTER AUTHOR RESPONSE:  Thanks for your response, particularly for the clarification wrt the hypothesis. I agree with the comment wrt cross-modal mapping. What I don't share is the kind of equation  visual = referential  that you seem to assume. A referent can be visually presented, but visual information can be usefully added to a word's representation in aggregate form to encode perceptual aspects of the words' meaning, the same way that it is done for textual information; for instance, the fact that bananas are yellow will not frequently be mentioned in text, and adding visual information extracted from images will account for this aspect of the semantic representation of the word. This is kind of technical and specific to how we build distributional models, but it's also relevant if you think of human cognition (probably our representation for  banana  has some aggregate information about all the bananas we've seen --and touched, tasted, etc.).  It would be useful if you could discuss this issue explicitly, differentiating between multi-modal distributional semantics in general and the use of cross-modal mapping in particular.  Also, wrt the  all models perform similarly  comment: I really urge you, if the paper is accepted, to state it in this form, even if it doesn't completely align with your hypotheses goals (you have enough results that do). It is a better description of the results, and more useful for the community, than clinging to the n-th digit difference (and this is to a large extent independent of whether the difference is actually statistical significant or not: If one bridge has 49% chances of collapsing and another one 50%, the difference may be statistically significant, but that doesn't really make the first bridge a better bridge to walk on).  Btw, small quibble, could you find a kind of more compact and to the point title? (More geared towards either generally what you explore or to what you find?)  ----------  The paper tackles an extremely interesting issue, that the authors label  referential word meaning , namely, the connection between a word's meaning and the referents (objects in the external world) it is applied to. If I understood it correctly, they argue that this is different from a typical word meaning representation as obtained e.g. with distributional methods, because one thing is the abstract  lexical meaning  of a word and the other which label is appropriate for a given referent with specific properties (in a specific context, although context is something they explicitly leave aside in this paper). This hypothesis has been previously explored in work by Schlangen and colleagues (cited in the paper). The paper explores referential word meaning empirically on a specific version of the task of Referential Expression Generation (REG), namely, generating the appropriate noun for a given visually represented object.  - Strengths:  1) The problem they tackle I find extremely interesting; as they argue, REG is a problem that had previously been addressed mainly using symbolic methods, that did not easily allow for an exploration of how speakers choose the names of the objects. The scope of the research goes beyond REG as such, as it addresses the link between semantic representations and reference more broadly.  2) I also like how they use current techniques and datasets (cross-modal mapping and word classifiers, the ReferIt dataset containing large amounts of images with human-generated referring expressions) to address the problem at hand.   3) There are a substantial number of experiments as well as analysis into the results.   - Weaknesses:  1) The main weakness for me is the statement of the specific hypothesis, within the general research line, that the paper is probing: I found it very confusing.  As a result, it is also hard to make sense of the kind of feedback that the results give to the initial hypothesis, especially because there are a lot of them and they don't all point in the same direction.  The paper says:   This paper pursues the hypothesis that an accurate model of referential word meaning does not need to fully integrate visual and lexical knowledge (e.g. as expressed in a distributional vector space), but at the same time, has to go beyond treating words as independent labels.   The first part of the hypothesis I don't understand: What is it to fully integrate (or not to fully integrate) visual and lexical knowledge? Is the goal simply to show that using generic distributional representation yields worse results than using specific, word-adapted classifiers trained on the dataset? If so, then the authors should explicitly discuss the bounds of what they are showing: Specifically, word classifiers must be trained on the dataset itself and only word classifiers with a sufficient amount of items in the dataset can be obtained, whereas word vectors are available for many other words and are obtained from an independent source (even if the cross-modal mapping itself is trained on the dataset); moreover, they use the simplest Ridge Regression, instead of the best method from Lazaridou et al. 2014, so any conclusion as to which method is better should be taken with a grain of salt. However, I'm hoping that the research goal is both more constructive and broader. Please clarify.   2) The paper uses three previously developed methods on a previously available dataset. The problem itself has been defined before (in Schlangen et al.). In this sense, the originality of the paper is not high.   3) As the paper itself also points out, the authors select a very limited subset of the ReferIt dataset, with quite a small vocabulary (159 words). I'm not even sure why they limited it this way (see detailed comments below).  4) Some aspects could have been clearer (see detailed comments).  5) The paper contains many empirical results and analyses, and it makes a concerted effort to put them together; but I still found it difficult to get the whole picture: What is it exactly that the experiments in the paper tell us about the underlying research question in general, and the specific hypothesis tested in particular? How do the different pieces of the puzzle that they present fit together?  - General Discussion: [Added after author response]  Despite the weaknesses, I find the topic of the paper very relevant and also novel enough, with an interesting use of current techniques to address an  old  problem, REG and reference more generally, in a way that allows aspects to be explored that have not received enough attention. The experiments and analyses are a substantial contribution, even though, as mentioned above, I'd like the paper to present a more coherent overall picture of how the many experiments and analyses fit together and address the question pursued.  - Detailed comments:  Section 2 is missing the following work in computational semantic approaches to reference:  Abhijeet  Gupta,  Gemma  Boleda,  Marco  Baroni,  and Sebastian  Pado. 2015.   Distributional                                            vectors  encode  referential          attributes. Proceedings of EMNLP, 12-21  Aurelie Herbelot and Eva Maria Vecchi.                                            2015.  Building a shared world: mapping distributional to model-theoretic semantic spaces. Proceedings of EMNLP, 22 32.  142 how does Roy's work go beyond early REG work?  155 focusses links  184 flat  hit @k metric :  flat ?  Section 3: please put the numbers related to the dataset in a table, specifying the image regions, number of REs, overall number of words, and number of object names in the original ReferIt dataset and in the version you use. By the way, will you release your data? I put a  3  for data because in the reviewing form you marked  Yes  for data, but I can't find the information in the paper.  229  cannot be considered to be names  ==>  image object names   230 what is  the semantically annotated portion  of ReferIt?  247 why don't you just keep  girl  in this example, and more generally the head nouns of non-relational REs? More generally, could you motivate your choices a bit more so we understand why you ended up with such a restricted subset of ReferIt?  258 which 7 features? (list) How did you extract them?  383  suggest that lexical or at least distributional knowledge is detrimental when learning what a word refers to in the world : How does this follow from the results of Frome et al. 2013 and Norouzi et al. 2013? Why should cross-modal projection give better results? It's a very different type of task setup than object labeling.  394-395 these numbers belong in the data section  Table 1: Are the differences between the methods statistically significant? They are really numerically so small that any other conclusion to  the methods perform similarly  seems unwarranted to me. Especially the  This suggests...  part (407).   Table 1: Also, the sim-wap method has the highest accuracy for hit @5 (almost identical to wac); this is counter-intuitive given the @1 and @2 results. Any idea of what's going on?  Section 5.2: Why did you define your ensemble classifier by hand instead of learning it? Also, your method amounts to majority voting, right?   Table 2: the order of the models is not the same as in the other tables + text.  Table 3: you report cosine distances but discuss the results in terms of similarity. It would be clearer (and more in accordance with standard practice in CL imo) if you reported cosine similarities.  Table 3: you don't comment on the results reported in the right columns. I found it very curious that the gold-top k data similarities are higher for transfer+sim-wap, whereas the results on the task are the same. I think that you could squeeze more information wrt the phenomenon and the models out of these results.  496 format of  wac   Section 6 I like the idea of the task a lot, but I was very confused as to how you did and why: I don't understand lines 550-553. What is the task exactly? An example would help.   558  Testsets   574ff Why not mix in the train set examples with hypernyms and non-hypernyms?  697  more even : more wrt what?  774ff  Previous cross-modal mapping models ... force... : I don't understand this claim.  792  larger test sets : I think that you could even exploit ReferIt more (using more of its data) before moving on to other datasets.