Named entities are frequently used in a metonymic manner. They serve as references to related entities such as people and organisations. Accurate identification and interpretation of metonymy can be directly beneficial to various NLP applications, such as Named Entity Recognition and Geographical Parsing. Until now, metonymy resolution (MR) methods mainly relied on parsers, taggers, dictionaries, external word lists and other handcrafted lexical resources. We show how a minimalist neural approach combined with a novel predicate window method can achieve state-of-the-art results on the SemEval 2007 task on Metonymy Resolution. Additionally, we contribute with a new Wikipedia-based MR dataset called RelocaR, which is tailored towards locations as well as improving previous deficiencies in annotation guidelines.
1 Introduction
In everyday language, we come across many types of figurative speech. These irregular expressions are understood with little difficulty by humans but require special attention in NLP. One of these is metonymy, a type of common figurative language, which stands for the substitution of the concept, phrase or word being meant with a semantically related one. For example, in “Moscow traded gas and aluminium with Beijing.”, both location names were substituted in place of governments.
Named Entity Recognition (NER) taggers have no provision for handling metonymy, meaning that this frequent linguistic phenomenon goes largely undetected within current NLP. Classi-
fication decisions presently focus on the entity using features such as orthography to infer its word sense, largely ignoring the context, which provides the strongest clue about whether a word is used metonymically. A common classification approach is choosing the N words to the immediate left and right of the entity or the whole paragraph as input to the model. However, this “greedy” approach also processes input that should in practice be ignored.
Metonymy is problematic for applications such as Geographical Parsing (GP) (see a survey by Monteiro et al. (2016)) and other information extraction tasks in NLP. In order to accurately identify and ground location entities, for example, we must recognise that metonymic entities constitute false positives and should not be treated the same way as regular locations. For example, in “London voted for the change.”, London refers to the concept of “people” and should not be classified as a location. There are many types of metonymy (Shutova et al., 2013), however, in this paper, we primarily address metonymic location mentions with reference to GP and NER.
Contributions: 1. We investigate how to improve classification tasks by introducing a novel minimalist method called Predicate Window (PreWin), which is highly discriminating with its selection of input (achieved SOTA on SemEval 2007 MR task). PreWin outperforms other systems, which use many external features and tools. 2. We also improve the annotation guidelines in MR and contribute with a new Wikipedia-based MR dataset called ReLocaR to address the training data shortage. 3. We also make an annotated subset of the CoNLL 2003 (NER) Shared Task available for extra MR training data, alongside models, tools and other data for full replicability.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
2 Related Work
Some of the earliest work on MR that used an approach similar to our method (machine learning and dependency parsing) was by Nissim and Markert (2003a). The decision list classifier with backoff was evaluated using syntactic head-modifier relations, grammatical roles and a thesaurus to overcome data sparseness and generalisation problems. However, the method was still limited for classifying unseen data. Our method uses the same paradigm but adds more features, a different machine learning architecture and a better usage of the parse tree structure.
Much of the later work on MR comes from the SemEval 2007 Shared Task 8 (Markert and Nissim, 2007) and later (Markert and Nissim, 2009). The feature set from (Nissim and Markert, 2003a) was updated to include: grammatical role of the potentially metonymic word (PMW) (such as subj, obj), lemmatised head/modifier of PMW, determiner of PMW, grammatical number of PMW (singular, plural), number of words in PMW and number of grammatical roles of PMW in current context. The winning system by Farkas et al. (2007) used these features and a maximum entropy classifier to achieve 85.2% accuracy. This was also the “leanest” system but still made use of feature engineering and some external tools. Brun et al. (2007) achieved 85.1% accuracy using local syntactical and global distributional features generated with an adapted, proprietary Xerox deep parser. This was the only unsupervised approach, based on using syntactic context similarities calculated on large corpora such as the the British National Corpus (BNC) with 100M tokens.
Nastase and Strube (2009) used a Support Vector Machine (SVM) with handcrafted features (in addition to the features provided in Markert and Nissim (2007)) including grammatical collocations extracted from the BNC to learn selectional preferences, WordNet 3.0, Wikipedia’s category network, whether the entity “has-a-product” such as Suzuki and whether the entity “has-an-event” such as Vietnam (both obtained from Wikipedia). The bigger set of around 60 features and leveraging global (paragraph) context enabled them to achieve 86.1% accuracy. Once again, we draw attention to the extra training, external tools and additional feature generation.
Similar recent work by Nastase and Strube (2013), extending Nastase et al. (2012) involved transforming Wikipedia into a large-scale multilingual concept network called WikiNet. By building on Wikipedia’s existing network of categories and articles, their method automatically discovers new relations and their instances on a large scale. As one of their extrinsic evaluation tasks, metonymy resolution was tested. Global context (whole paragraph) was used to interpret the target word. Using an SVM and the powerful knowledge base built from Wikipedia, the highest performance to date (a 0.1% improvement from Nastase and Strube (2009)) was achieved at 86.2%, which has remained the SOTA until now.
The related work on MR so far has made limited use of dependency trees. Typical features came in the form of a head dependency of the target entity, its dependency label and its role (subj-of-win, dobj-of-visit, etc). However, other classification tasks made good use of dependency trees. Liu et al. (2015) used the shortest dependency path and dependency sub-trees successfully to improve relation classification (new SOTA on SemEval 2010 Shared Task). Bunescu and Mooney (2005) show that using dependency trees to generate the input sequence to a model performs well in relation extraction tasks. Dong et al. (2014) used dependency parsing for Twitter sentiment classification to find the words syntactically connected to the target of interest. Joshi and Penstein-Rosé (2009) used dependency parsing to explore how features based on syntactic dependency relations can be used to improve performance on opinion mining. In unsupervised lymphoma (type of cancer) classification, (Luo et al., 2014) constructed a sentence graph from the results of a two-phase dependency parse to mine pathology reports for the relationships between medical concepts. Our methods also exploit the versatility of dependency parsing to leverage information about the sentence structure.
2.1 SemEval 2007 Dataset
Our main standard for performance evaluation is the SemEval 2007 Shared Task 8 (Markert and Nissim, 2007) dataset first introduced in Nissim and Markert (2003b). Two types of entities were evaluated, organisations and locations, randomly retrieved from the British National Corpus (BNC).
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
We only use the locations dataset, which comprises a train (925 samples) and a test (908 samples) partition. For coarse evaluation, the classes are literal (geographical territories and political entities), metonymic (place-for-people, place-forproduct, place-for-event, capital-for-government or place-for-organisation) and mixed (metonymic and literal frames invoked simultaneously or unable to distinguish). The metonymic class further breaks down into two levels of subclasses allowing for medium and fine evaluation. The class distribution within SemEval is approx 80% literal, 18% metonymic and 2% mixed. This seems to be the approximate natural distribution of the classes for location metonymy, which we have also observed while sampling Wikipedia for our new dataset.
3 Our Approach
Our contribution broadly divides into two main parts, data and methodology. Section 3 introduces our new dataset, Section 4 introduces our new feature extraction method.
3.1 Design and Motivation
As part of our contribution, we created a new MR dataset called ReLocaR (Real Location Retrieval), partly due to the lack of quality annotated train/test data and partly because of the shortcomings with the SemEval 2007 dataset (see Section 3.2). Our corpus is designed to evaluate the capability of a classifier to distinguish literal, metonymic and mixed location mentions. In terms of dataset size, ReLocaR contains 1,000 training and 1,000 test instances. The data was sampled using Wikipedia’s Random Article API1. We kept the sentences, which contained at least one of the places from a manually compiled list2 of countries and capitals of the world. The natural distribution of literal versus metonymic examples is approximately 80/20 so we had to discard the excess literal examples during sampling to balance the classes.
3.2 ReLocaR - Improvements over SemEval
1. We do not break down the metonymic class further as the distinction between the subclasses is subtle and hard to agree on.
2. The distribution of the three classes in ReLocaR (literal, metonymic, mixed) is approximately
1https://www.mediawiki.org/wiki/API:Random 2Available on GitHub as locations.txt
(49%, 49%, 2%) eliminating the high bias (80%, 18%, 2%) of SemEval. We will show how such a high bias transpires in the test results (Section 5).
3. We have reviewed the annotation of the test partition and found that we disagreed with up to 11% of the annotations. Zhang and Gelernter (2015) disagreed with the annotation 8% of the time. Poibeau (2007) also challenged some annotation decisions. ReLocaR was annotated by 4 trained linguists (undergraduate and graduate) and 2 computational linguists (authors). Linguists were independently instructed (see section 3.3) to assign one of the two classes to each example with little guidance. We leveraged their linguistic training and expertise to make decisions rather than imposing some specific scheme. Unresolved sentences would receive the mixed class label.
4. The most prominent difference is a small change in the annotation scheme (after independent linguistic advice). The SemEval 2007 Task 8 annotation scheme, which can be found in (Markert and Nissim, 2007) considers the political entity interpretation a literal reading. It suggests that in “Britain’s current account deficit...”, Britain refers to a literal location, rather than a government (which is an organisation). This is despite acknowledging that “The locative and the political sense is often distinguished in dictionaries as well as in the ACE annotation scheme...”. In ReLocaR, we consider a political entity a metonymic reading.
3.3 Annotation Guidelines (Summary)
ReLocaR has three classes, literal, metonymic and mixed. Literal reading comprises territorial interpretations (the geographical territory, the land, soil and physical location) i.e. inanimate places that serve to point to a set of coordinates (where something might be located and/or happening) such as “The treaty was signed in Italy.”, “Peter comes from Russia.”, “Britain’s Andy Murray won the Grand Slam today.”, “US companies increased exports by 50%.”, “China’s artists are among the best in the world.” or “The reach of the transmission is as far as Brazil.”.
A metonymic reading is any location occurrence that expresses animacy (Coulson and Oakley, 2003) such as “Jamaica’s indifference
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
will not improve the negotiations.”, “Sweden’s budget deficit may rise next year.”. The following are other metonymic scenarios: a location name, which stands for any persons or organisations associated with it such as “We will give aid to Afghanistan.”, a location as a product such as “I really enjoyed that delicious Bordeaux.”, a location posing as a sports team “India beat Pakistan in the playoffs.”, a governmental or other legal entity posing as a location “Zambia passed a new justice law today.”, events acting as locations “Vietnam was a bad experience for me”.
The mixed reading is assigned in two cases: either both readings are invoked at the same time such as in “The Central European country of Slovakia recently joined the EU.” or there is not enough context to ascertain the reading i.e. both are plausible such as in “We marvelled at the art of ancient Mexico.”. In difficult cases such as these, the mixed class is assigned.
3.4 Inter-Annotator Agreement
We give the IAA for the test partition only. The whole dataset was annotated by the first author as the main annotator. Two pairs of annotators (4 linguists) then labelled 25% of the dataset each for a 3-way agreement. The agreement before adjudication was 91% and 93%, 97.2% and 99.2% after adjudication (for pair one and two respectively). The other 50% of sentences were then once again labelled by the main annotator with a 97% agreement with self. The remainder of the sentences (unable to agree on among annotators even after adjudication) were labelled as a mixed class (1.8% of all sentences).
3.5 CoNLL 2003 and MR
We have also annotated a small subset of the CoNLL 2003 NER Shared Task data for metonymy resolution (locations only). Following the Reuters RCV1 Corpus (Lewis et al., 2004) distribution permissions3, this data is only available by emailing the first author. There are 4,089 positive (literal) and 2,126 negative (metonymic) sentences to assist with algorithm experimentation and model prototyping. Due to the lack of annotated training data for MR, this is a valuable resource. The data was annotated by the first author, there are no IAA figures.
3http://trec.nist.gov/data/reuters/reuters.html
4 Methodology
4.1 Predicate Window (PreWin)
Through extensive experimentation and observation, we arrived at the intuition behind PreWin, our novel feature extraction method. The classification decision of the class of the target entity is mostly informed not by the whole sentence (or paragraph), rather it is a small and focused “predicate window” pointed to by the entity’s head dependency. In other words, most of the sentence is not only superfluous for the task, it actually lowers the accuracy of the model due to irrelevant input. This is particularly important in metonymy resolution as the entity’s surface form does not change for subsequent classifications.
In Figure 1, we show the process of extracting the Predicate Window from a sample sentence (more examples are available in the Appendix). We start by using the SpaCy dependency parser by Honnibal and Johnson (2015), which is the fastest in the world, open source and highly customisable. Each dependency tree provides the following features: dependency labels and entity head dependency. Rather than using most of the tree, we only use a single local head dependency relationship to point to the predicate. Leveraging a dependency parser helps PreWin with selecting the minimum relevant input to the model while discarding irrelevant input, which may cause the neural model to behave unpredictably. Finally, the entity itself is never used as input in any of our methods, we only rely on context.
PreWin then extracts up to 5 words and their dependency labels starting at the head of the entity, going in the away (from the entity) direction. The method always skips punctuation and the conjunct (“and”, “or”) relationships in order to find the predicate (see Figure 3 in the Appendix for a visual example of why this is important). The rest of the model’s input is set to zeroes (see Figure 2 in the Appendix for a detailed diagram and the final model). The reason for the choice of 5 words is the balance between too much input, feeding the model with less relevant context and just enough context to capture the necessary semantics.
4.2 Neural Network Architecture
The output of PreWin is processed using the following machine learning model. We decided
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Figure 1: The predicate window starts at the head of the target entity and ends up to 4 words further, going away from the entity. The “conj” relations and punctuation are always skipped. In the above example, the head of “UK” is “decided” so PreWin takes 5 words plus labels as the input to the model. In this case, the left hand side input to the model is set to zeroes (see the Appendix for full architecture).
to use the Long Short Term Memory (LSTM) architecture by Keras4 (Chollet, 2015). Four LSTMs are used in total, two for the left and right side (up to 5 words each) and two for the left and right dependency relation labels (up to 5 labels each). The full architecture is available in the Appendix. LSTMs are excellent at modelling language sequences (Hochreiter and Schmidhuber, 1997), (Sak et al., 2014), (Graves et al., 2013), which is why we use this type of model.
Both the Multilayer Perceptron and the Convolutional Neural Network were consistently inferior (typically 5% - 10% lower accuracy) in our earlier performance comparisons and experiments, which is also why we opted for LSTMs. For all experiments, we used a vocabulary of the first (most frequent) 100,000 word vectors in GloVe5 Pennington et al. (2014). Finally, unless explicitly stated otherwise, the standard dimension of word embeddings was 50, which we found to work best.
4.3 “Immediate” Baseline
A common approach in lexical classification tasks is choosing the 5 to 10 words to the immediate right and left of the entity as input to a model such as Mikolov et al. (2013), Mesnil et al. (2013), Baroni et al. (2014) and Collobert et al. (2011). We evaluate this method (its 5 and 10-word variant) alongside PreWin and Paragraph.
4.4 Paragraph Baseline
The paragraph baseline method extends the “immediate” one by taking 50 words from each side of the entity as the input to the classifier. In practice,
4https://keras.io/ 5http://nlp.stanford.edu/projects/glove/
this extends the feature window to include extrasentential evidence in the paragraph. This approach is also popular in machine learning (Melamud et al., 2016), (Zhang et al., 2016).
4.5 Ensemble of Models
In addition to a single best performing model, we have combined several models trained on different data and/or using different model configurations. The Ensemble method enabled us to reach SOTA results. For SemEval data, we combined two separate models (using PreWin) trained on the newly annotated 2003 CoNLL NER dataset and the training data for SemEval. For ReLocaR data, we let three models vote, all trained on the ReLocaR training data. The first model trained with 300-dimensional embeddings (PreWin method), the second used the paragraph baseline and the third used PreWin (standard 50-dimensional embeddings).
5 Results
We evaluate all methods using (any one or a combination of) three datasets for training (ReLocaR, SemEval, CoNLL) and two for testing (ReLocaR, SemEval).
5.1 Metrics and Significance
Following the SemEval 2007 convention, we use two metrics to evaluate performance, accuracy and f-score (for each class). We only evaluate at the coarse level, which means literal versus nonliteral (metonymic and mixed are merged into one class). In terms of statistical significance, both the SemEval dataset (1,000 samples) and our accuracy improvement (although new SOTA) are too small to be significant at the 95% confidence level.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
However, the accuracy improvements of PreWin over each baseline are highly statistically significant with 99.9%+ confidence.
5.2 Predicate Window
Tables 1 and 2 show PreWin performing consistently better than other baselines, in many instances, significantly better and with fewer words (smaller input). Compared with the 5 and 10 window “immediate” baseline, which is the common approach in classification, PreWin is more discriminating with its input. Due to the linguistic variety and the myriad of ways the target word sense can be triggered in a sentence, it is not always the case that the 5 or 10 nearest words inform us of the target entity’s meaning/type. We ought to ask what else is being expressed in the same 5 to 10-word window?
Conventional classification methods (Immediate, Paragraph) can also be seen as prioritising either feature precision or feature recall. Paragraph maximises the input sequence size, which maximises recall at the expense of including features that are either irrelevant or mislead the model, lowering performance. Immediate maximises precision by using features close to the target entity at the expense of missing important features positioned outside of its small window, once again lowering performance. PreWin can be understood as an integration of both approaches. It retains high precision by limiting the size of the feature window to 5 while maximising recall by searching anywhere in the sentence, frequently outside of a limited “immediate” window.
Perhaps we can also caution against a simple adherence to Firth (1957) “You shall know a word by the company it keeps”. This does not appear to be the case in our experiments as the PreWin regularly performs better than the “immediate” baseline. Further prototypical examples of the method can be viewed in the Appendix. Our intuition that most words in the sentence, indeed in the paragraph do not carry the semantic information required to classify the target entity is ultimately based on evidence. Aiming to approximate human decision making, the neural model uses only a small window (which may be far away from the entity), linked to the entity via a head dependency relationship for the final classification decision.
5.3 Common Errors
Most of the time (typically 85% for the two datasets), PreWin is sufficient for an accurate classification. However, it does not work well in some cases.
Discarding important context: Sometimes the 5 or 10 word “immediate” baseline method would actually have been preferred such as in the sentence “...REF in 2014 ranked Essex in the top 20 universities...”. PreWin discards the right-hand side input, which is required in this case for a correct classification. Since ”ranked” is the head of ”Essex”, the rest of the sentence gets ignored and the valuable context gets lost.
More complex semantic patterns: Many common mistakes were due to the lack of the model’s understanding of more complex predicates such as in the following sentences: “ ...of military presence of Germany.”, “Houston also served as a member and treasurer of the...” or ”...invitations were extended to Yugoslavia ...”. We think this is due to a lack of training data (approx 1,000 sentences per dataset). Additional examples such as “...days after the tour had exited Belgium.” expose some of the limitations of the neural model to recognise uncommon ways of expressing a reference to a literal place. Recall that no external resources or tools were used to supplement the training/features, the model had to learn to generalise from what it has seen during training, which was limited in our experiments.
Parsing mistakes were less common though still present. It is important to choose the right dependency parser for the task since different parsers will often generate slightly different parse trees. We have used SpaCy6 for all our experiments, which is a Python-based industrial strength NLP library. Sometimes, tokenisation errors for acronyms like “U.S.A.” and wrongly hyphenated words may also cause parsing errors, however, this was infrequent.
5.4 Flexibility of Neural Model
The top accuracy figures for ReLocaR are almost identical to SemEval. The highest single model accuracy for ReLocaR was 84.3% (85.7% with Ensemble), which was within 1% of the equivalent
6https://spacy.io/
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Method Training (Size) Accuracy
PreWin SemEval (925) 66.8 Immediate 5 SemEval (925) 59.5 Immediate 10 SemEval (925) 59.6 Paragraph SemEval (925) 60.4
PreWin CoNLL (6,215) 81.3 Immediate 5 CoNLL (6,215) 78.6 Immediate 10 CoNLL (6,215) 80.6 Paragraph CoNLL (6,215) 78.0
PreWin ReLocaR (1,000) 84.3 Immediate 5 ReLocaR (1,000) 81.9 Immediate 10 ReLocaR (1,000) 82.6 Paragraph ReLocaR (1,000) 82.3
Ensemble ReLocaR (1,000) 85.7
Table 1: Results for the ReLocaR dataset.
methods for SemEval. Both were achieved using the same methods (PreWin or Ensemble), neural architecture and size of corpora. When the model is trained on the CoNLL (NER) data, the accuracies are 79.7% and 81.3%. This shows a good degree of flexibility in our minimalist neural network. However, when the model trained on ReLocaR and tested on SemEval (and vice versa), accuracy drops to between 66.8% and 72.5% showing that what was learnt does not seem to transfer well to another dataset. We think the reason for this is the difference in annotation guidelines; the government is a metonymic reading, not a literal one. This causes the model to make more mistakes.
Method Training (Size) Accuracy
PreWin SemEval (925) 85.0 Immediate 5 SemEval (925) 81.5 Immediate 10 SemEval (925) 81.8 Paragraph SemEval (925) 81.3
PreWin CoNLL (6,215) 79.7 Immediate 5 CoNLL (6,215) 77.8 Immediate 10 CoNLL (6,215) 78.2 Paragraph CoNLL (6,215) 79.0
PreWin ReLocaR (1,000) 72.5 Immediate 5 ReLocaR (1,000) 64.8 Immediate 10 ReLocaR (1,000) 66.0 Paragraph ReLocaR (1,000) 66.3
Nastase et al. (2013) SemEval (1,000) 86.2
Ensemble PreWin SemEval & CoNLL 86.3
Table 2: Results for the SemEval dataset.
5.5 Ensemble Method
The highest accuracy and f-scores were achieved with the ensemble method for both datasets. We combined two models (previously described in section 4.5) for SemEval to achieve 86.3% accuracy (previous SOTA 86.2%) and three models for ReLocaR to achieve 85.7% for the new dataset. Training separate models with different parameters and/or on different datasets does increase classification capability as various models learn distinct aspects of the task. Letting them vote for the final label enabled the 1.3 - 1.4% improvement.
5.6 Dimensionality of Word Embeddings
We found that increasing dimension size (to 100 and 300) did not improve performance, in fact, accuracy decreased by an average of 3%. The neural network tended to overfit, even with fewer epochs, the results were inferior to our default 50-dimensional embeddings. We posit that fewer dimensions of the distributed word representations force the abstraction level higher as the meaning of words must be expressed more succinctly. We think this helps the model generalise better, particularly for smaller datasets. Lastly, learning word embeddings from scratch on datasets this small (1,000 samples) is possible but impractical, the performance typically decreases 5% if word embeddings are not initialised first.
Dataset / Method Literal Non-Literal
SemEval / PreWin 91.9 60.6 SemEval / SOTA 91.6 59.1 ReLocaR / PreWin 86.1 85.6
Table 3: Per class f-scores - the old versus new SOTA figures (all using the Ensemble method). Note the model class bias for SemEval.
5.7 F-Scores and Class Imbalance
Table 3 shows the new SOTA f-scores using the Ensemble method, the previous SOTA on SemEval and the best f-scores for ReLocaR. The class imbalance inside SemEval (80% literal, 18% metonymic, 2% mixed) is reflected as a high bias in the final model. This is not the case with ReLocaR and its 49% literal, 49% metonymic and 2% mixed ratio of 3 classes. The model was equally capable of distinguishing between literal and nonliteral cases.
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
5.8 Another baseline
There was another baseline we tested, however, it was not covered anywhere so far because of its low performance. It was a type of extreme parse tree pruning, during which most of the sentence gets discarded and we only retain 3 to 4 content words. The method uses non-local (long range) dependencies to construct a short input sequence. However, the method was a case of ignoring too many relevant words and accuracy was fluctuating in the mid-60% range, which is why we did not report the results. However, it serves to further justify the choice of 5 words as the predicate window as fewer words caused the model to underperform.
6 Discussion
6.1 NER, GP and Metonymy
We think the next frontier is a NER tagger, which actively handles metonymy. The task of labelling entities should be mainly driven by context rather than the word’s surface form. If the target entity looks like “London”, this should not mean the entity is automatically a location. Metonymy is a frequent linguistic phenomenon and could be handled by NER taggers to enable many innovative downstream NLP applications.
Geographical Parsing is a pertinent use case. In order to monitor/mine text documents for geographical information only, the current NER technology does not have a solution. We think it is incorrect for any NER tagger to label “Vancouver” as a location in “Vancouver welcomes you.”. A better output might be something like the following: Vancouver = location AND metonymy = True. This means Vancouver is usually a location but is used metonymically in this case. How this information is used will be up to the developer. Organisations behaving as persons, share prices or products are but a few other examples of metonymy.
6.2 Simplicity and Minimalism
Previous work in MR such as most of the SemEval 2007 participants (Farkas et al. (2007), Nicolae et al. (2007), Leveling (2007), (Brun et al., 2007), (Poibeau, 2007)) and the more recent contributions used a selection of many of the following features/tools for classification: handmade trigger word lists, WordNet, VerbNet, FrameNet, extra features generated/learnt from parsing Wikipedia (approx 3B words) and BNC (approx 100M
words), custom databases, handcrafted features, multiple (sometimes proprietary) parsers, Levin’s verb classes, 3,000 extra training instances from a corpus called MAScARA 7 by Markert and Nissim (2002) and other extra resources including the SemEval Task 8 features. We managed to achieve higher performance with a small neural network, minimal training data, a basic dependency parser and the new PreWin method by being highly discriminating in choosing signal over noise.
7 Conclusions and Future Work
We showed how a minimalist neural approach can replace substantial external resources, handcrafted features and how the PreWin method can even ignore most of the paragraph where the entity is positioned and still achieve state of the art performance in metonymy resolution. The pressing new question is: “How much better the performance could have been if our method availed itself of the extra training data and resources used by previous works?” Indeed this is the next research chapter for PreWin.
We discussed how tasks such as Geographical Parsing can benefit from “metonymy-enhanced” NER tagging. We have also presented a case for better annotation guidelines for MR (after consulting with a number of linguists), which now means that a government is not of a literal class, rather it is a metonymic one. We fully agreed with the rest of the previous annotation guidelines. We also introduced ReLocaR, a new corpus for (location) metonymy resolution and encourage researchers to make effective use of it (including the additional CoNLL 2003 data subset we annotated for metonymy).
Further future work will be to test the PreWin method on an NER task to see if and how it can generalise to a different classification task and how the results compare to the SOTA and similar methods such as Collobert et al. (2011) using the CoNLL 2003 NER datasets. Word Sense Disambiguation (Yarowsky, 2010), (Pilehvar and Navigli, 2014) with neural networks (Yuan et al., 2016) is another related classification task suitable for testing PreWin. If it does perform better, this will be of considerable interest to classification research (and beyond) in NLP.
7http://homepages.inf.ed.ac.uk/mnissim/mascara/
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths: Great paper: Very well-written, interesting results, creative method, good and enlightening comparisons with earlier approaches. In addition, the corpus, which is very carefully annotated, will prove to be a valuable resource for other researchers. I appreciated the qualitative discussion in section 5. Too many ML papers just give present a results table without much further ado, but the discussion in this paper really provides insights for the reader.   - Weaknesses: In section 4.1, the sentence  The rest of the model  input is set to zeroes...  is quite enigmatic until you look at Figure 2. Some extra sentence here explaining what is going on would be helpful. Furthermore, in Figure 2, in the input layers to the LSTMs it says  5*Embeddings(50D)  also for the networks taking dependency labels as input. Surely this is wrong? (Or if it is correct, please explain what you mean).   - General Discussion: Concerning the comment in 4.2  LSTMs are excellent at modelling language sequences ... which is why we use this type of model. . This comment seems strange to me. This is not a sequential problem in that sense. For each datapoint, you feed the network all 5 words in an example in one go, and the next example has nothing to do with the preceding one. The LSTM architecture could still be superior, of course, but not for the reason you state. Or have I misunderstood something? I'd be interested to hear the authors' comments on this point.


A Geometric Contextual Model for Identifying Unseen Metaphors
We present a method for metaphor identification based on an explicitly geometric approach to modelling salient lexical associations, and test it on a task of distinguishing metaphoric from literal uses of adjective-noun phrases. By contextually projecting candidate word pairs into interpretably geometric spaces, we show that it obtains state-of-the-art performance, while providing a method which is effectively zero-shot – able to be applied to new unseen phrases. Our dynamically context-sensitive model is inspired by theoretical insight into the situational nature of language and cognition.
1 Introduction
The standard approach to the computational modelling of metaphor has been grounded in theoretical work suggesting that metaphors are mappings between conceptual domains, and has accordingly dealt with the construction of representational structures that allow symbolic mappings between semantic categories. The result has been a rich literature of both symbolic and statistical approaches to metaphor identification, interpretation, and even generation. Recent approaches in the statistical vein have tended to centre on the construction of linear algebraic representations of words – words as vectors and matrices – which can be compositionally interpreted in terms of their mathematical properties. One such example for identification of metaphor is (Gutiérrez et al., 2016), who reported strong results on a new dataset using a compositional technique involving the supervised and semi-supervised learning of adjective tensors for the classification of candidate adjective-noun metaphors.
Here, we also take a distributional semantic approach, but predicated on a slightly different theoretical premise: we hold that metaphors are linguistic phenomena that emerge dynamically in the course of language use, as a semantic phenomenon at the end of a fluid spectrum, related to the ad hoc, situated nature of concept formation. Our work, based on a method described by McGregor et al. (2015) for concept formation, contextually generates a new subspace for every candidate metaphor, projecting word pairs into unique geometric relationships based on an analysis of their independent word-vector representations. In addition to its contextual flexibility, the strengths of our model are its simplicity and its generality: it requires only vectors for individual words, rather than representations of the potentially metaphoric phrases themselves, and is therefore suitable for the “zero-shot” setting of identifying a novel, unseen metaphor. Analysing the subspaces produced by our model by means of a logistic regression, we report F-scores that are comparable with the supervised method described by Gutiérrez et al. and superior to their semi-supervised method.
2 Background
As Shutova (2010) points out, the conceptual work that has had the greatest impact across various computational models of metaphor emerged from a spate of theoretical productivity in the late 1970s and early 1980s (see Ortony, 1993, for a valuable compendium). Per Shutova’s analysis, two particularly influential theories of metaphor are the selectional preference violation model due to Wilks (1978) and the conceptual metaphor model of Lakoff and Johnson (1980). Wilks takes an approach that correlates with the pragmatic maxims of Grice (1975), suggesting that metaphor constitutes some sort of breach of conceptual ex-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
pectation. Lakoff and Johnson, on the other hand, in work that has been much discussed by cognitive linguists and cognitive scientists, describe metaphor in terms of isomorphic mappings between broad conceptual domains, with the anatomy of these often culturally specific domains coercing the nuances of particular metaphors.
NLP researchers have generally interpreted this theoretical position to indicate that concepts are stable categories essentially denoted by linguistic symbols, and that communication is more or less about the efficacious application of these symbols in trafficking propositions about things in the world. A result of this stance has been that the computational modelling of metaphor tends to focus on the construction of static representational structures where metaphoricity is assessed in terms of conceptually transgressive compositions of these representations: the metaphor is taken to transport meaning from one domain to another in a semantically effective way (for a comprehensive overview, see Shutova, 2015). In terms of symbolic approaches, which often incorporate some sort of either hand-made or automatically encoded knowledge base, this has resulted in models involving the chaining of inferences across sequences of symbols to discover novel conceptual relationships (Lee and Barnden, 2001) and the scaling of taxonomies such as WordNet (Veale and Hao, 2008) or custom databases (Veale, 2016).
Statistical approaches have often focused on the distributional semantic paradigm, using observations of co-occurrences across large-scale corpora to build word representations which can subsequently be manipulated through linear algebraic operations (see Clark, 2015, for an overview). So, for instance, Utsumi (2011) models metaphor comprehension as a process of clustering terms that are proximate to both elements of two-word metaphors in a latent semantic analysis model (Deerwester et al., 1990). In the specific area of metaphor identification, Turney et al. (2011) present a model which uses distributional statistics to derive sets of abstract and concrete nouns, the assumption being that metaphor can be detected as a mapping between likewise abstract and concrete domains, and Hovy et al. (2013) use a combination of word-vector representations and annotated trees to identify syntactic consistencies that indicate the presence of metaphor.
Metaphoricity detection via mappings In recent work, Gutiérrez et al. (2016) provide a good example of this general approach, together with a dataset and task involving the classification of a fairly evenly balanced set of literal and metaphoric adjective-noun pairs, specifically designed to include adjectives which tend to occur in both metaphoric and literal compositions. We describe the dataset in more detail in Section 4. Their approach takes the Lakoff and Johnson (1980) view of metaphor as mapping, implemented via supervised learning of distinct adjective tensors for metaphoric and literal senses (given a view of adjectives as tensors and nouns as word vectors, a technique previously employed by Baroni and Zamparelli (2010) and more generally in line with current developments in compositional distributional semantics (Mitchell and Lapata, 2010; Coecke et al., 2011)).
They test two versions of this approach. Given an adjective-noun pair {a, n}, the first version learns a tensor Aa for each adjective a by building vectors −→n ,−→an from observed co-occurrence statistics of the noun n and the combined phrase an, and then minimising the error between−→an and the product Aa−→n . For each adjective a, they learn two separate tensors AM(a), AL(a) for literal and metaphoric senses, learned from labelled an pairings. An phrase an is then considered metaphoric if −→an is closer to the product AM(a)−→n than to AL(a)
−→n , or vice versa. This version gives good performance (details below), but has the disadvantage of requiring many labelled examples of phrases an. They therefore present a second version, in which a general “conceptual metaphor” mappingCM for adjectives is learned, to minimise the error between −→an and the product CMAL(a)−→n for metaphoric examples. This reduces performance, but does not require labelled examples for every adjective a. However, it does still require a large number of observations of every phrase an in context, to build the vector−→an; and is therefore not suitable for analysing novel phrases as yet unseen.
Divergent views Our own approach, while it is situated within the distributional semantic paradigm, is inspired by theoretical considerations which diverge somewhat from the doctrinaire stance on conceptual metaphors and selectional violation. In their deflationary account of metaphors, Wilson and Sperber (2012) argue that metaphor is properly understood as a particularly
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
salient instance of the pragmatically resolved underspecification inherent in all language use. Similarly, Carston (2012) suggests that metaphor – and indeed all language use – involves the construction of ad hoc conceptual schemes in which the implications of an utterance are resolved contextually based on expectations of maximal relevance. Davidson (1978) has even gone so far as to suggest that metaphor should always primarily be taken literally in the first instance, a controversial view which has received more recent support from Reimer (2001), who engages with the Davidsonian rejection of cognitive content in order to tease out a distinction between the lexical meaning of words, which exist in an encyclopedic sense, versus the pragmatic use of language.
These theoretical considerations, coupled with psycholinguistic research indicating a neurological ambiguity between the processing of ostensibly literal versus metaphoric language (McElree and Nordlie, 1999; Wolff and Gentner, 2011), motivates us to propose a model for metaphor identification which uses information about input words to project an ad hoc geometry, in which words and their associations are mapped to features of a contextually specific space. A distributional semantic model of this kind has been proposed by McGregor et al. (2015) for concept discovery, and Agres et al. (2016) have suggested its appropriateness for modelling metaphoricity; we take this as our starting point (more details below).
3 A Method for Contextual Geometry
We seek to redress the evident schism between pragmatic and computational accounts of metaphor by taking into account the significance of context in the online construction of situated conceptualisations. With this in mind, we characterise our model in terms of three desiderata:
1. Spaces of word representations for identifying metaphoricity should be in some sense selected contextually;
2. The generation of these spaces should be independent of a priori ratings of objective metaphoricity or literalness;
3. The contextualisation should generate a space which is geometrically interpretable.
Following McGregor et al. (2015), the model is based on a standard distributional vector space,
using pointwise mutual information (PMI) cooccurrence statistics calculated over a large-scale corpus, in this case the December 2014 dump of English language Wikipedia. The top 200,000 most frequent words are taken as the model’s vocabulary, and every single word type which appears within a context window of n words on either side of one of these target words is a cooccurrence basis dimension. We use a version of positive PMI as shown in (1), where nw,c is the frequency at which target word w occurs in the context of context word c; nw is the total frequency of w; nc is the total frequency at which c occurs as a context word; W is the total count of vocabulary word tokens in the corpus, and a is a smoothing constant (set in this case at 10,000):
PMIw,c = log2
( nw,c ×W
nw × (nc + a) + 1
) (1)
Note that rather than discarding negative PMI values as in standard approaches, they are weighted by the +1 term, ensuring that all values will be positive, with a value of 0 indicating that w and c are never observed to co-occur. With a total of c.7.5 million word types in the corpus, this results in a very large, very sparse matrix of 200,000 rows representing vocabulary word-vectors.
This matrix can now serve as a base space which can be contextualised by selecting relevant lower-dimensional subspaces based on an analysis of input words. McGregor et al. (2015) use this to discover ad-hoc concepts; here, potentially metaphoric adjective-noun word pairs will be used as input, and dimensions selected based on the independent and correlated properties of the wordvectors for each word in the pair. As these dimensions relate to PMI between target and context words, they contain information about a word’s most salient associations. We expect the combination of these properties to be informative about the conceptual relationship denoted by the phrase in question; and therefore predict that the geometric relationships of the words within this subspace will be indicative of the likelihood of an assessment of metaphoric versus literal usage.
With our base space established, the primary parameter of our experiment will therefore be the criteria for selecting a subspace based on any given word pair. Following and expanding upon Agres et al. (2016), we experiment with five different techniques for selecting k-dimensional subspaces
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
based on an adjective-noun word pair{a, n} and corresponding word vectors −→a and −→n :
Joint Select all dimensions with non-zero PMI values for both −→a and −→n ; then select the top k dimensions with the highest mean value across −→a and −→n from this subset. Independent Select the top k/2 dimensions for −→a and −→n independently. Zipped Select all mutually non-zero dimensions; then select the top dimensions for −→a and −→n independently from the subset.
Adjective Only Select the top k dimensions for −→a , regardless of their values for n. Noun Only Select the top k dimensions for −→n .
In each of these subspaces, we consider the situation of −→a and −→n in terms of a set of triangulations between these vectors, the origin of the space, and a line extending from the origin through the centre of the space, as illustrated in Figure 1. The points A and B correspond to the word-vectors for the adjective and the noun in a potentially metaphoric adjective-noun phrase. The points of the triangle CDE sit on the surface of a hypersphere emanating from the origin of the subspace, with C and D corresponding to the vectors of A and B normalised to length r respectively. The point E corresponds to the likewise normalised value of a vector composed of the mean PMI values of each all points on each dimension delineating the subspace—so, a point along a scaled centre-line, effectively.
We predict that ratings of metaphoricity will correlate with features of the geometry of these triangulations in subspaces. In particular, we predict that, in the case of spaces generated from adjective-noun pairs that would be considered typically metaphoric, sides AB and CD will tend to increase in length, with the points A and B tending to diverge on a dimension-by-dimension basis as the contextual subspace selected by the corresponding input terms and word-vectors becomes less consistent. We likewise expect points A and B to move closer to the origin in the cases of jointly selected subspaces, or, in the cases of the singly selected subspaces, the point corresponding to the word-vector not involved in the selection will recede. In other words, we predict that the geometry of the space will reflect a dithering of correspondence in what might be considered the way of talking about each of the things denoted by the input
O
A
BC
D E
Figure 1: A Metaphor In Space: Word-vectors for adjective-noun pairs are projected into contextualised subspaces where metaphoricity is found to correlate with geometrical relationships between the word-vectors and the centre of their subspace.
terms as the conceptualisation suggested by their composition moves towards what is perceived as a more metaphoric interpretation of the language.
4 Experiment and Results
In order to test the contextual, geometric model described in Section 3, we apply the method outlined above to the dataset of Gutiérrez et al. (2016). The data consists of 8,592 adjective-noun dyads, each using one of 23 adjectives chosen for their propensity to be used in both literal and metaphoric contexts, and each occurring at least ten times in a large-scale composite corpus. Each dyad (adjective-noun pair) is tagged with a binary annotation indicating either metaphoricity or literalness; 53.5% were deemed metaphoric, with each adjective type occurring at an average rate of 0.664 as either metaphoric or literal.
For each dyad, we use each of the techniques described in Section 3 above to project the wordvectors for the adjective and the noun into a contextualised subspace for that adjective-noun pair (i.e. its basis dimensions are the words with contextually salient associations). We then take the lengths of the sides of the triangles illustrated in Figure 1 – triangles ABO and CDE – as features in a logistic regression model, trained to predict the metaphoric/literal label. We use scikit-learn’s LogisticRegression class, applying mean-zero standard-deviation-1 normalisation to the features, and evaluate using standard shuffled 10-fold cross-validation (again following
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Gutiérrez et al. (2016)). Our evaluation metrics are the standard (micro-)average precision, recall, f-score and accuracy across the 10 folds. Note that while we train the regression model using labelled instances, our input features are only information about the individual words - there is no requirement for, or learning of, phrase representations (thus allowing application to unseen phrases).
We test various versions of our model, experimenting with the five different subspace selection techniques described in Section 3. For each technique, we experiment with varying the number of dimensions used to build the subspaces (we test the top 20, 50, 200, and 500 dimensions); and with the size of the context window used to build the base co-occurrence model (we test 2-word and 5- word windows, i.e. taking context words up to 2 words and 5 words on either side of a target word). F-scores returned for these various parameters are listed in Table 1.
Table 1: F-scores for subspaces of dimensionalities 20, 50, 200, and 500 built from 2+2 and 5+5 word context window models.
Joint Indy Zip Adj Noun
2+ 2
20 0.761 0.776 0.778 0.727 0.750 50 0.776 0.784 0.790 0.727 0.750 200 0.795 0.793 0.795 0.755 0.761 500 0.790 0.790 0.784 0.757 0.765
5+ 5
20 0.772 0.782 0.785 0.751 0.757 50 0.786 0.798 0.798 0.756 0.764 200 0.806 0.802 0.813 0.783 0.771 500 0.811 0.806 0.812 0.774 0.777
We compare to the method of Gutiérrez et al. (2016), which learns tensors for each adjective; as described above, they give two versions, one fully supervised version which learns separate tensors for literal and metaphoric senses from tagged data, and one which uses only the literal-sense tensors for each adjective and combines these with a tensor corresponding to the mapping from literal to metaphoric senses in general. They take the second method as more suitable for “unseen” use cases, although it still requires enough observations of the adjective-noun pair to build its own phrase vector. As our method uses no information about the whole phrase occurrence, and is thus suited for truly unseen cases, a more relevant comparison is the second version.
However, while they evaluate their first version (“V1” in our tables below) on the full dataset, obtaining an f-score of 0.817, they only evaluate the second version (“V2”) on a reduced subset of the data due to their data and labelling requirements. For this, they obtain an f-score of 0.793, but it seems likely that performance on the full dataset would be lower, as this is the case for V1 (full set f-score 0.817, reduced set 0.838). We give results for comparison in Table 2, but take the V2 results as the more relevant comparison, although they may over-estimate full dataset performance.1 We also show a majority-class baseline score, and a score for a logistic regression based on cosine similarity between adjectives and nouns as returned by word2vec (Mikolov et al., 2013).
5 Analysis
First of all, our model gives f-scores close to Gutiérrez et al.’s V1 model, and outperforms the more relevant V2 model (both significant at p<0.05 using χ2). The mechanisms for achieving these results, however, are evidently somewhat different. It is noteworthy – and indeed impressive – that the compositional V1 model gives particularly strong precision scores, given that, with a dataset slightly biased towards the metaphoric, a strategy of simply calling everything a metaphor would result in perfect recall, a relatively high fscore, but low precision (see the baseline results).
One of the strengths of our model is that it does not require pre-built phrase vectors based on observations across a corpus; we could, in principle, evaluate the metaphoricity of any given com-
1Evaluation on the reduced set is not currently possible, as the identity of the subset used is not available. Note that Table 2 gives the V1 and V2 results as reported in the original paper, although the V2 combination of recall, precision and f-score does not quite add up; precision should perhaps be higher, or f-score lower.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
position of words within our vocabulary by projecting a contextual subspace based on an analysis of all the independent words in the phrase. Gutiérrez et al. acknowledge the desirability of such a model, although both their V1 and V2 methods require the construction of and comparison with phrase vectors. Their V2 method does not learn individual metaphoric/literal tensors for adjectives, so is more general: but returns slightly weaker results (f = 0.793 on a reduced dataset) than the original, phrase-dependent methodology (f = 0.838), with the balance tipped more towards recall than precision, as with our model. As we’re not aware of the phrases which were included in the reduced dataset, however, a direct comparison isn’t possible at this point.
Returning to the parametric comparison of results presented in Table 1, the progression of fscores for each technique for subspace progression bears further analysis. The first thing to note is that the models based on larger co-occurrence windows do better, a finding which lines up with Sahlgren’s (2008) description of the tendency towards the semantic (syntagmatic) and away from the syntactic (paradigmatic) as this co-occurrence window widens. This finding could motivate an exploration of spaces employing broader cooccurrence windows, not to mention spaces spaces populated by word-vectors consisting of more sequentially and syntactically nuanced features.
Next, clearly, and not surprisingly, the spaces that take information from both terms involved in the candidate phrase perform better than those that consider only one of the two words, and it must be noted that the best-performing joint and zipped subspaces will converge as the dimensionality parameter increases, with the subspace projected eventually including all the dimensions with non-zero values for both input words in all three cases, while the independent subspaces will eventually contain all dimensions with non-zero values for either input word. It’s likewise noteworthy that the zipped technique, which returns subspaces consisting of dimensions with relatively independent saliency for each input term, does best at dimensionalities of up to 200, whereas both the joint and independent spaces increase in performance towards 500 dimensions. We interpret this as indicating that the dimensions of a subspace most predictive of metaphor contain some combination of information about the terms involved in the can-
AO 0.669 0.616 0.660 0.632 0.721
BO 0.739 0.747 0.736 0.753 0.734
AB 0.644 0.657 0.638 0.617 0.700
CE 0.652 0.691 0.658 0.657 0.648
DE 0.651 0.702 0.668 0.629 0.690
CD 0.651 0.689 0.679 0.628 0.646
didate phrase that is neither maximally salient to both (or all) terms nor particularly skewed towards one term. Again, a more nuanced approach to subspace projection is probably available and warrants further research.
In order to better understand the geometry selected by our analysis, we now perform logistic regressions on the two triangles delineated in our subspaces as well as each of the features of the geometry – each side of both triangles – independently, and likewise return regression results for the sides of each of the two (ABO and CDE in Figure 1) separately. The results are reported in Table 3. Notably, the length of side BO, corresponding to the norm of the noun word-vector, is by far the feature most predictive of metaphor. That the norm of the adjective word-vector (side AO) is not nearly as predictive is to be expected given the nature of the dataset: as the data was constructed specifically to test adjectives that tend to occur in both metaphoric and literal adjective-noun compositions, each adjective contributes to the construction of both metaphoric and literal subspaces. With this in mind, it also makes sense that AO is relatively predicative in the noun-only type subspace, where the adjectival word-vector isn’t contributing at all to the selection of dimensions.
On the other hand, it’s also notable that AO is relatively predictive in other subspaces (relatively is itself relative here—aside from the noun norm BO, very few of the features independently outperform the majority class baseline of f = 0.697). We interpret this to mean that, notwithstanding the legitimate efforts on the part of the dataset designers to choose versatile adjectives, there is
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
a general tendency for some adjectives to be involved in more metaphoric compositions than others. As mentioned in Section 4, the typical balance of adjectives in this dataset is an almost 2:1 preference towards selecting either metaphoric or literal compositions. The strategy of simply determining, based on observations across the annotated data, that a given adjective was always either metaphoric or literal would therefore result in both an accuracy rate and an f-score of 0.664.
Finally, the relatively poor predictiveness of triangle ADE – the lengths between the normalised input word-vectors and a normlised centre line – bears further analysis, particularly in light of the fairly even distribution of performances between geometric features other than BO. In order to understand how each the shape of these related triangles shifts as we move from the metaphoric to the literal end of the spectrum of adjective-noun compositions, we examine the geometry of specific cases of metaphors in space, as visualised in Figure 2. Treating our logistic regression as a generative model, we extract the word-pair deemed to be most metaphoric in 200-dimensional zippedtype spaces, and similarly the word-pair considered to be least metaphoric and a word-pair that falls close to the centre of the model’s predictions. As can be seen, there is an lengthening of the distance between adjective and noun moving from the metaphoric to the neutral example, and then an extension of the disance of both points from the origin moving from the neutral to the literal instance.
We were admittedly initially surprised by this evident bulging at the middle of the metaphoricliteral spectrum; we had predicted a steady widening and corresponding move towards the periphery of subspaces as word pairs move towards the more metaphoric. In order to check if this trend is general rather than just a peculiarity of the metaphors examined in Figure 2, we calculate the average ratio of the length of AB to the distance of its midpoint from the origin for the 500 most metaphoric, most neutral, and most literal adjective-noun pairs in the dataset based on our regression. Results are reported in Table 4, considering not only the zipped type subspace anecdotally depicted in Figure 2 but also the two other subspace varieties selected involving information from both input word-vectors. (The subspaces selected based only on an analysis of either noun or adjective vectors tend to be characterised by significant difference
between norms of each projected vector, so this analysis is less meaningful there.)
Significantly, the trend is evident across all three multi-word input subspaces, with especially pronounced differences in the joint type subspaces. It would seem that the move from metaphoric to literal is in fact characterised by a two part expansion, beginning with a widening of the space between the input terms as they become more neutrally (or perhaps ambiguously) interpreted and then an extension away from the origin as they become more literally interpreted. What seems to be happening here is actually an increase in the dimensions available to the model as the terms move towards the neutral, but at the same time a diminishing of the correspondence between the input words and so a widening of the space between them on a dimension-by-dimension basis. The end result is actually a kind of horseshoe phenomenon, with both literal and metaphoric compositions tending towards one another in the subspaces they select, but with literal compositions moving away from the origin as there tends to be more correspondence between the set of salient (ie, high valued) co-occurrence dimensions for each word. Meanwhile, what we now refer to as ambiguous rather than neutral compositions tend to select for subspaces where a wider range of dimensions have some information about both terms, but there tends to be less correspondence between each word across these dimensions.
Returning to some of the theoretical suppositions outlined in Section 2, we recall that we postulated that metaphor, and indeed language in general, is always ultimately determined contextually. We outline a small set of examples sampled from the middle and either end of the spectrum outlined by our regression to illustrate this point:
metaphoric: murky franc, clear result, brilliant exception, smooth solicitor, icy formality
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
murky
franc
(a) most metaphoric: “murky franc”
brilliant
glass
(b) neutral: “brilliant glass”
sour
sauce
(c) least metaphoric: sour sauce
Figure 2: Phrases Across the Spectrum of Metaphoricity: Three phrases, the most metaphoric, least metaphoric, and one sampled from the middle of the spectrum, based on an analysis of subspaces projected by our model. This projection preserves all lengths within the triangles, as well as the position of the smaller triangle tangentialed to a hypersphere eminating from the origin.
ambiguous: brilliant glass, bitter spirit, dense information, shallow center, heavy betting
literal: sour sauce, sweet garlic, sour cabbage, sour lemon, sweet sauce
We note in particular that phrases such as “bitter spirit” and “shallow center” are open to interpretation depending on the context that they are used. Indeed, we maintain that all these phrases are, to some extent, open to interpretation when taken out of context: a “smooth solicitor” could conceivably be a soft-skinned lawyer, while “sweet sauce” might describe a sauce which is generally desirable rather than particularly sugary.
So here at last we return to the issue of metaphor in context, and make two final comments by way of reinforcing observations which have been made frequently in the theoretical literature. The first is that metaphor is probably properly construed as being an extent of a spectrum, and a spectrum that is not clearly defined in absolute terms but rather is probably specific to a contextual expectation. The second point follows on from this: metaphor is contextual, and and given adjectivenoun phrase could probably, at a stretch, be forced into either a metaphoric or literal interpretation under the correct conditions. So, while our model provides the basic equipment for an unsupervised analysis on any given word pair, we feel it also lays the groundwork for generating more contextually nuanced spaces in which conceptual relationships map to the geometric situations of word-vectors.
6 Conclusion
The conceptual metaphor model — with metaphors as mappings between conceptual
schemas — has been rightly influential in computational linguistics; it is not our intention to be iconoclastic. However, our results suggest that it might be enhanced by combining it with the insight that lexical semantics is always resolved contextually. Conceptual schemas can be recast as geometries within ad-hoc, contextualised subspaces, delineated by dimensions that are tractable, interpretable features of language use which relate to salient co-occurrence associations; this view, even simply implemented as here, gives accuracy comparable to state-of-the-art approaches while retaining the zero-shot ability to apply to truly unseen examples.
The primary features of our method are its contextual dynamism and its geometry. Through the online selection of subspaces based on contextspecific input, we’re able to build ad hoc subspaces in which novel conceptual relationships emerge as geometric features. A strength of this model is its simplicity, and likewise its availability as an interpretable framework that can be easily analysed using standard classification and regression techniques. In principle, any words in our vocabulary could be projected into a subspace, and we might expect the geometry of such subspaces to do more work in the long run, issuing features involving other semantic relationships and compositionality. Ultimately, we hold that language itself essentially affords conceptualisation: while words are not ideas and thought are not words, language provides a structure with which to, as Barsalou (1993) puts it, haphazardly compose novel concepts in an unpredictable world. Here we have attempted to consider how a computational approach to such a flexible model might work.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

This paper proposes an approach for classifying literal and metaphoric adjective-noun pairs. The authors create a word-context matrix for adjectives and nouns where each element of the matrix is the PMI score. They then use different methods for selecting dimensions of this matrix to represent each noun adjective as a vector. The geometric properties of average, nouns, and adjective vectors and their normalized versions are used as features in training a regression model for classifying the pairs to literal or metaphor expressions. Their approach performs similarly to previous work that learns a vector representation for each adjective.  Supervision and zero-shot learning. The authors argue that their approach requires less supervision (compared to previous work)  and can do zero-shot learning. I don t think this is quite right and given that it seems to be one of the main points of the paper, I think it is worth clarifying. The approach proposed in the paper is a supervised classification task: The authors form vector representations from co-occurrence statistics, and then use the properties of these representations and the gold-standard labels of each pair to train a classifier. The model (similarly to any other supervised classifier) can be tested on words that did not occur in the training data; but, the model does not learn from such examples. Moreover, those words are not really  unseen d because the model needs to have a vector representation of those words.  Interpretation of the results. The authors provide a good overview of the previous related work on metaphors. However, I am not sure what the intuition about their approach is (that is, using the geometric properties such as vector length in identifying metaphors). For example, why are the normalized vectors considered? It seems that they don t contribute to a better performance. Moreover, the most predictive feature is the noun vector; the authors explain that this is a side effect of the data which is collected such that each adjective occurs in both metaphoric and literal expressions. (As a result, the adjective vector is less predictive.) It seems that the proposed approach might be only suitable for the given data. This shortcoming is two-fold: (a) From the theoretical perspective (and especially since the paper is submitted to the cognitive track), it is not clear what we learn about theories of metaphor processing. (b) From the NLP applications standpoint, I am not sure how generalizable this approach is compared to the compositional models.  Novelty. The proposed approach for representing noun adjective vectors is very similar to that of Agres et al. It seems that the main contribution of the paper is that they use the geometric properties to classify the vectors.