Detecting Lexical Entailment in Context
Detecting entailment between words is a key task for several NLP applications. Previous work has largely focused on entailment between words out of context. We propose, instead, to address lexical entailment in context, providing exemplar sentences to ground the meaning of words considered in the entailment relation. We show that contextualized word representations constructed from existing word embeddings, and word-context similarity features lead to significant improvements over context-agnostic models on two novel entailment test sets, and also improve the state-of-the-art on the related task of detecting semantic relations in context (Shwartz and Dagan, 2015).
1 Introduction
Many NLP applications require detecting relations between word meanings beyond synonymy and paraphrasing. For instance, given “Carlsen plays chess.” and the question “Which game does Carlsen play?”, successfully answering the question requires knowing that chess is a kind of game, or more generally, that chess entails game.
While prior work has defined lexical entailment as a relation between word types (Turney and Mohammad, 2013), we argue entailment relations are better defined when illustrating word meaning with an example in context. Ignoring context is problematic since entailment might hold between some senses of the words, but not others. Consider the word game in two distinct contexts:
1. The championship game was played in NYC.
2. The hunters were interested in the big game.
Given the sentence, Carlsen is the world chess champion, chess =⇒ game as used in the first context, while chess 6=⇒ game in the second context.
In this paper, we investigate how to represent and compare the meaning of words in context for lexical entailment. Since distributional representations for word types have proved useful to detect lexical entailment out of context in supervised settings (Baroni et al., 2012; Roller et al., 2014; Turney and Mohammad, 2013), we propose to transform context-agnostic word type representations into contextualized representations that highlight salient properties of the context (Section 3), and use these contextualized representations with a range of semantic similarity features (Section 4) to successfully detect entailment.
As we will see, these context representations significantly improve performance over contextagnostic baselines not only in English, but also between English and French words (Section 7) on two novel datasets (Section 5). We also show that our features are sensitive to word sense changes indicated by context, and adequately capture the direction of entailment relation (Section 8). Moreover, we establish a new state-of-the-art on an existing dataset that captures a broader range of semantic relations in context (Shwartz and Dagan, 2015), and show that the proposed features, induced solely from large amounts of raw text, yield systems that perform as well, or better than existing systems that require additional human annotation.
2 Defining Lexical Entailment in Context
We frame the task of lexical entailment in context as a binary classification task on examples consisting of a 4-tuple (wl, wr, cl, cr), where wl and wr are two words, and cl and cr are sentences which
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
Words (wl, wr) Exemplars (cl,cr) Does wl =⇒ wr ?
staff , stick cl = He walked with the help of a wooden staff . Yes cr = The kid had a candied apple on a stick. staff , body cl = The hospital has an excellent nursing staff . Yes cr = The whole body filed out of the auditorium. staff , stick cl = The hospital has an excellent nursing staff . No cr = The kid had a candied apple on a stick.
Table 1: Examples of the context-aware lexical entailment task
illustrate each word usage. The example is treated as positive if wl =⇒ wr, given the meaning of each word exemplified by the contexts, and negative otherwise. Table 1 provides examples.
We say that entailment holds if the meaning of wl in the context of cl is more specific than the meaning of wr in the context of cr. The nature of entailment relations captured out-of-context can be broader depending on the test beds considered1. Zhitomirsky-Geffet and Dagan (2009) formalize lexical entailment as a substitutional relationship, which encompasses synonymy, hypernymy, some meronymy relations, and also cause-effect relations. However, we limit entailment to the specificity relation in this work to better understand the impact of context.
Note that lexical entailment in context is not textual entailment. Recognizing textual entailment (Dagan et al., 2013) and natural language inference (Bowman et al., 2015) involves detecting entailment relations between sentences, while lexical entailment is a relation between words.
3 Representing Words and their Contexts for Entailment
How can we construct vector representations of the meaning of target words wl and wr in their respective exemplar contexts cl and cr? We start from existing representations for word types which have proven useful for detecting lexical entailment and other semantic relations out of context (Baroni et al., 2012; Kruszewski and Baroni, 2015; Vylomova et al., 2016; Turney and Mohammad, 2013).
Given an example (wl, wr, cl, cr), let ~wl and ~wr refer to the context-agnostic representations of wl and wr, and let Cl and Cr represent the matrices obtained by row-wise stacking of the context-
1We refer the reader to Turney and Mohammad (2013) and Shwartz et al (2017) for comprehensive surveys of supervised and unsupervised methods for the out-of-context task.
agnostic representations of words in cl and cr respectively.
3.1 Contextualized Word Representations
Following Thater et al. (2011); Erk and Padó (2008), our first approach is to apply a filter to word type representations to highlight the salient dimensions of the exemplar context, emphasizing relevant dimensions of and downplaying unimportant ones. However, while prior work represents context by averaging word vectors, we propose richer representations that better capture the salient geometrical properties of the exemplar context that might get lost by averaging (Figure 1).
First, we construct fixed length representations for the contexts cl and cr by running convolutional filters over Cl and Cr. Specifically, we calculate the column-wise maximum, minimum and the mean over the matrices Cl and Cr, as done by (Tang et al., 2014) for supervised sentiment classification. This yields three ddimensional vectors for cl (~cl,max, ~cl,min, ~cl,mean), and three d-dimensional vectors for cr (~cr,max, ~cr,min, ~cr,mean). Computing the maximum and minimum across all vector dimensions captures the exterior surface of the “instance manifold”
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
(the volume in embedding space within which all words in the instance reside), while the mean summarizes the density per-dimension within the manifold (Hovy, 2015).
Second, we transform initial context-agnostic representations for target word types by taking an element-wise product of the word type vectors ( ~w∗) with vectors representing salient dimensions of the exemplar context (~c∗,max, ~c∗,min, ~c∗,mean) where ∗ ∈ {l, r} . This yields three d-dimensional vectors for wl (~wl,max, ~wl,min, ~wl,mean), and three for wr (~wr,max, ~wr,min, ~wr,mean). We refer to our final word-in-context representations for wl and wr as ~wl,mask and ~wr,mask respectively, where ~wl,mask is the concatenation of ~wl,max, ~wl,min, ~wl,mean, and ~wr,mask is also similarly constructed.
3.2 A Shared Vector Space for Words and Contexts: Context2vec
An alternative approach to contextualizing word representations is to directly compare the representations of words with representations of contexts. In a single language, this can be done using Context2Vec (Melamud et al., 2016), a neural model that, given a target word and its sentential context, embeds both the word and the context in the same low-dimensional space, with the objective of having the context predict the target word via a log linear model. This model approaches the state-of-the-art on lexical substitution, sentence completion, and supervised word sense disambiguation.
4 Comparing Words and Contexts for Entailment
Given words and context representations described above, how can we predict entailment?
4.1 Supervised Logistic Regression model
Prior work on lexical entailment out of context suggests that the entailment relationship between words is a learnable function of the concatenation of their individual representations (Baroni et al., 2012; Turney and Mohammad, 2013). After concatenation, they are used as features for a logistic regression (Roller et al., 2014) or an SVM classifier (Baroni et al., 2012; Turney and Mohammad, 2013). We follow the same practice with our context-aware word representations. Intuitively, the classifier learns to weight the importance of each dimension of a word representation to detect
entailment. By learning different weights for the same features describing wl or wr, the classifier can detect asymmetric behavior too, as we will see in Section 8.
4.2 Similarity Features
We hypothesize that entailment relations hold between related words and introduce similarity features to capture this non-directional relation between words and contexts.
Given a pair of vectors (~l, ~r), we use three similarity measures: the cosine similarity cosine(~l, ~r), the dot product ~l · ~r, and the euclidean distance ‖~l − ~r‖. The cosine similarity captures the difference in the two vectors in terms of the angle between then, the euclidean distance measures the difference in magnitude, and the dot product captures both magnitude and angle.
We apply these measures to three types of representations. Our first set of similarities is intended to directly capture the similarities between contextualized word representations (Section 3.1). We calculate pairwise similarities between (~wl,max, ~wr,max), (~wl,min, ~wr,min), (~wl,mean, ~wr,mean), as well as between the context-agnostic representations (~wl, ~wr). Second, we add similarities based on Context2Vec representations, i.e. between each pair of ~wl,c2v, ~wr,c2v, ~cl,c2v, and ~cr,c2v. Finally, following Shwartz and Dagan (2015), we capture the similarity of the most relevant word to wl in cr, that to wr in cl, as well as between cl and cr. We therefore add the following three similarities, as well as their pairwise products:
max w∈cr ~wl · ~w, max w∈cl ~wr · ~w, max w∈cl,w′∈cr ~w · ~w′
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Word : Study Example : he knocked on the door of the study
Word : Study Example : he made several studies before the final painting
Word : Room Example : the rooms were small but comfortable
Word : Drawing Example : he did complicated pen-and-ink drawings
Figure 2: Sample dataset creation process based on two synsets of the word study. The green/solid lines indicate positive examples, while the red/dashed lines indicate negative examples
5 Evaluation Framework
We evaluate our models of lexical entailment in context on three complementary datasets.
5.1 CONTEXT-PPDB
The first dataset, CONTEXT-PPDB, is a finegrained lexical inference dataset created using 375 word pairs from a subset of the English Paraphrase Database (Ganitkevitch et al., 2013; Pavlick et al., 2015b). These word pairs are semiautomatically labeled with semantic relations outof-context. Shwartz and Dagan (2015) augmented them with examples of word usage in context, and re-annotating word pairs given the extra contextual information. The final dataset consists of 3750 words/contexts tuples with a corresponding label from Table 2, one of which is entailment.
5.2 CONTEXT-WN
In addition to CONTEXT-PPDB, we would like a test set for controlled analysis of lexical entailment in context. We would like to directly assess the sensitivity of our models to contexts that signal different word senses, as well as quantify the extent to which our models detect asymmetric entailment relations rather than semantic similarity.
Based on the above criteria, we introduce a large-scale dataset automatically extracted from WordNet (Fellbaum, 1998a). We call this dataset CONTEXT-WN. WordNet groups synonyms into synsets. Most synsets are further accompanied by one or more short sentences illustrating the use of the members of the synset. The idea behind CONTEXT-WN is to use these example sentences as context for the words, and hypernymy relations to draw examples of lexical entailment relations. The entire process starts from a seed list of words W and proceeds as follows (see Figure 2 for an illustration) :
1. For each word type w ∈ W with multiple synsets, obtain all synsets Sw.
2. For each synset i ∈ Sw, pick a hypernym synset sih, with a corresponding word form wih. Also obtain c
i and cih which are example sentences corresponding to wi and wih respectively - (wi, wih, c
i, cih) serves as a positive example.
3. Permute the positive examples to get negative examples. From (wi, wih, c
i, cih) and (wj , wjh, c
j , cjh), generate negative examples (wi, wjh, c i, cjh) and (w j , wih, c j , cih).
4. Flip the positive examples to generate more negative examples. From (wi, wih, c
i, cih) generate the negative example (wih, w i, cih, c i).
We run this process using the 9000 most frequent words from Wikipedia as W (after filtering the top 1000 as stopwords). This yields a total of 5239 positive examples. We sample an equal number of negative examples from Step 3, with another 5239 examples being generated in Step 4. The final dataset consists of 10478 negative examples.
CONTEXT-WN satisfies our desiderata. The dataset has a very specific focus since we only pick hypernym-hyponym pairs. The negative examples generated in Steps 3 and 4 require discriminating between different word senses and entailment directions. Finally, with over 15000 examples distributed over 6000 word pairs, the dataset is almost five times as large as CONTEXT-PPDB.
We use a 70/5/25 train/dev/test split as in CONTEXT-PPDB. We also ensure that each set contains different word pairs, to avoid memorization and overfitting (Levy et al., 2015).
5.3 Crosslingual dataset
Our final dataset takes a cross-lingual view of lexical entailment in context. To create this dataset, we follow the same methodology that was used to create CONTEXT-PPDB, replacing word pairs from PPDB with a set of 100 English-French
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
word pairs labeled as positive out-of-context in the cross-lingual lexical entailment dataset of Vyas and Carpuat (2016). Each word pair (wl, wr) consists of an English word and a French word. To add context, we extract 10 sentence pairs (cl, cr) per word pair from the Wikipedia in the corresponding language. These thousand examples are then given to annotators on a crowdsourcing platform 2. For each example (wl, wr, cl, cr), the annotators were asked to judge whether wl =⇒ wr given the contexts cl and cr. Each example was annotated by 5 workers3.
For each example, we assign the majority judgment as the correct label. 4 or more annotators agree on the label on 540 examples, and we retain only these examples as our test set. This yields a dataset of 374 positive examples and 166 negative examples.
6 Experimental Set-up
6.1 Monolingual
All experiments on CONTEXT-PPDB and CONTEXT-WN are with the default train/dev/test splits. We use 50 dimensional off-the-shelf GloVe embeddings (Pennington et al., 2014) to create ~wl, ~wr, Cl, and Cr. To obtain the Context2Vec representations ~wl,c2v, ~wl,c2v, ~cl,c2v, and ~cr,c2v, we use an existing model trained on the ukWaC corpus (Ferraresi et al., 2006) .
We use Logistic Regression as our classifier for both tasks, to allow for direct comparisons with previous work. We do not tune parameters of the classifier, except for adding class weights in the CONTEXT-WN experiments to account for the unbalanced data. As evaluation metric, we use weighted F1 score to compare against previous results, as well as to account for label imbalance in CONTEXT-WN.
6.2 Crosslingual
We evaluate our models on cross-lingual test set in a transfer setting by training on the training set of CONTEXT-PPDB. To represent ~wl, ~wr, we experiment with two different pre-trained, 200- dimensional, bilingual embeddings - BiVec (Luong et al., 2015) and BiCVM (Hermann and Blunsom, 2014), both of which have been shown to be
2www.crowdflower.com 3We will release further details of our annotation task including guidelines and the specific task after the review period.
suitable for cross-lingual semantic tasks such as dictionary induction and document classification (Upadhyay et al., 2016). The classifier is again a Logistic Regression classifier with class weights.
7 Results
7.1 CONTEXT-PPDB
Representation-based features only A context-agnostic baseline (Baroni et al., 2012) based on the concatenation of the word type features for (wl, wr) reaches F1 scores of 53.5 using Glove and 54 with Context2Vec (Table 4). The contextualized word representations and similarity features yield the best result and improves performance by 14 points. The similarity features are also effective when used in combination with the Glove or Context2Vec representations, yielding improvements of 12 to 13 points on the baseline. We also note that using ~cl,c2v and ~cr,c2v to add contextual information performs very poorly (F = ~60), regardless of the word type feature, indicating the superiority of our masked representations.
Adding PPDB-specific features CONTEXTPPDB comes with rich information about word pairs drawn from PPDB (Pavlick et al., 2015a,b). These features include scores for likelihood of context-agnostic entailment labels, distributional similarities, and probabilities of the word pair being paraphrases, among other scores. Shwartz and Dagan (2015) establish a strong baseline of 67 F1 using these features and the most salient word/context similarities described in Section 4.2. Augmenting this system with our proposed context-aware similarities, we observe an improvement of ~2 F1 points. The contextualized representations further gives a ~3 point boost. Overall, our new context aware features help in improving upon the previous state-of-the-art by 4.8 F1 points.
7.2 CONTEXT-WN
We repeat all experiments on CONTEXT-WN (Table 4), except for those that require PPDB-specific features. The trends on CONTEXT-WN are similar to those on CONTEXT-PPDB. Using similarity features with the context-agnostic baseline, improves performance by ~4 F1 points. Adding the context-aware representations, further gives a tiny boost, enabling us to score 5 F1 points more than
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Word Type Features Context-aware Features Scores Similarities Representations P R F
7 All [~wl,mask ; ~wr,mask] 0.674 0.694 0.677 [~wl ; ~wr] 7 7 0.531 0.539 0.535 [~wl ; ~wr] 7 [~wl,mask ; ~wr,mask] 0.512 0.521 0.516 [~wl ; ~wr] All 7 0.642 0.642 0.642 [~wl ; ~wr] All [~wl,mask ; ~wr,mask] 0.638 0.637 0.637 [~wl ; ~wr] All [~cl,c2v ; ~cr,c2v] 0.605 0.604 0.603
[~wl,c2v ; ~wr,c2v] 7 7 0.533 0.569 0.540 [~wl,c2v ; ~wr,c2v] 7 [~wl,mask ; ~wr,mask] 0.518 0.545 0.526 [~wl,c2v ; ~wr,c2v] All 7 0.655 0.674 0.659 [~wl,c2v ; ~wr,c2v] All [~wl,mask ; ~wr,mask] 0.652 0.670 0.656 [~wl,c2v ; ~wr,c2v] All [~cl,c2v ; ~cr,c2v] 0.601 0.601 0.600 PPDB(wl, wr) Most salient only 7 0.677 0.685 0.670 PPDB(wl, wr) All 7 0.695 0.701 0.692 PPDB(wl, wr) All [~wl,mask ; ~wr,mask] 0.721 0.720 0.718
Table 3: Experimental results on CONTEXT-PPDB. The evaluation metric is weighted F score.
the baseline. Again, we observe that using ~cl,c2v and ~cr,c2v does worse than the alternatives.
The absolute value of gains on CONTEXT-WN is smaller than on CONTEXT-PPDB, as can be expected, given that CONTEXT-WN was designed to be challenging (Section 5.2).
7.3 Cross-lingual dataset
Results on the cross-lingual dataset are similar to monolingual results (Table 5). Context-aware features outperform context-agnostic baselines - similarities boost the score by 3 and 6 points in the BiVec and BiCVM settings respectively. The contextualized word representations, when used instead of the context-agnostic representations, give a boost of almost 3 points, allowing us to score 6 points and 10 points above the context-agnostic baseline in the BiVec and BiCVM cases, respectively. The strength of these results attest to the generalizable nature of our features, which helps capture correspondences beyond a single language.
8 Analysis
The experiments in Section 7 attest to the overall strength of our features. In this section, we aim to further test the assumptions underlying our proposed context representations.
8.1 Sensitivity to context
How sensitive are our models to changes in contexts? To answer this, we focus on the subset of
CONTEXT-WN that comprises of positive examples, and the equivalent number of negative examples from Step 3 of the dataset creation. These examples are created by permuting the contexts of the positive examples, and thus, can directly help in answering our question. We analyze the predictions made on this subset by our model using a metric we call ‘Macro-F1’, defined as the weighted F1 calculated over each (wl,wr) word pair, and then averaged over all word pairs.
Models using context-aware features do consistently better on Macro-F1 than those without (Table 4). Interestingly though, the model that uses [~cl,c2v ; ~cr,c2v] does the best on this metric, indicating that while directly using the Context2Vec representations might not be ideal, these representation do capture some useful contextual knowledge.
8.2 Sensitivity to Entailment Direction
To quantitatively evaluate our claim of our features learning to discriminate directionality 4, we report results on the subset of CONTEXT-WN that consists of all positive examples, and the equivalent number of flipped negative examples generated in Step 4. We measure directionality by looking at the number of example pairs ((wl, wr, cl, cr), (wr, wl, cr, cl)) where both examples are correctly labeled, i.e. the former is labeled as =⇒ and the latter as 6=⇒ .
Context agnostic baselines detect direction significantly above chance (Table 4). Adding our
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Word Type Feat.s Context-aware Features Scores Similarities Representations P R F Macro F1 Pairwise Acc.
Random 0.5 0.5 0.5 0.5 0.25 7 All [~wl,mask ; ~wr,mask] 0.700 0.661 0.670 0.677 0.623
[~wl ; ~wr] 7 7 0.653 0.615 0.625 0.594 0.582 [~wl ; ~wr] 7 [~wl,mask ; ~wr,mask] 0.656 0.618 0.628 0.608 0.592 [~wl ; ~wr] All 7 0.700 0.663 0.672 0.651 0.597 [~wl ; ~wr] All [~wl,mask ; ~wr,mask] 0.704 0.666 0.675 0.673 0.617 [~wl ; ~wr] All [~cl,c2v ; ~cr,c2v] 0.692 0.658 0.667 0.682 0.592
[~wl,c2v ; ~wr,c2v] 7 7 0.695 0.655 0.664 0.663 0.667 [~wl,c2v ; ~wr,c2v] 7 [~wl,mask ; ~wr,mask] 0.694 0.655 0.665 0.668 0.660 [~wl,c2v ; ~wr,c2v] All 7 0.728 0.689 0.697 0.704 0.700 [~wl,c2v ; ~wr,c2v] All [~wl,mask ; ~wr,mask] 0.724 0.685 0.694 0.704 0.691 [~wl,c2v ; ~wr,c2v] All [~cl,c2v ; ~cr,c2v] 0.713 0.678 0.687 0.708 0.638
Table 4: Results on CONTEXT-WN. Macro-F1 and Pairwise accuracy, are intended to capture contextawareness and directionality-discrimination abilities of our features, repsectively.
Word Type Features Context-aware Features Scores Similarities Representations P R F Random 0.570 0.481 0.501
[~wl,bivec ; ~wr,bivec] 7 7 0.618 0.537 0.560 [~wl,bivec ; ~wr,bivec] 7 [~wl,mask ; ~wr,mask] 0.618 0.537 0.560 [~wl,bivec ; ~wr,bivec] All 7 0.632 0.576 0.595 [~wl,bivec ; ~wr,bivec] All [~wl,mask ; ~wr,mask] 0.620 0.554 0.575
7 All [~wl,mask ; ~wr,mask] 0.650 0.607 0.622 [~wl,bicvm ; ~wr,bicvm] 7 7 0.651 0.517 0.530 [~wl,bicvm ; ~wr,bicvm] 7 [~wl,mask ; ~wr,mask] 0.655 0.528 0.543 [~wl,bicvm ; ~wr,bicvm] All 7 0.681 0.574 0.590 [~wl,bicvm ; ~wr,bicvm] All [~wl,mask ; ~wr,mask] 0.682 0.566 0.581
7 All [~wl,mask ; ~wr,mask] 0.676 0.614 0.629
Table 5: Results on cross-lingual dataset with training on CONTEXT-PPDB using bilingual embeddings.
context features improves by ~3, with the overall best performing model again scoring highest on this metric. It is reassuring to observe that improved detection of context with our features does not come at the cost of detecting entailment direction. Unlike in the previous section, the model that uses [~cl,c2v ; ~cr,c2v] is consistently poor, scoring lower than the corresponding context-agnostic baseline.
8.3 Contextualized Masks
We also hypothesized that masked contextualized representations based on the full volume of the context using min and max operations (Section 3.1) better capture salient context dimensions than the more usual vector averaging approach. We test this hypothesis empirically by replacing
masked word-in-context representations ~wl,mask and ~wr,mask by two other ways to capture context. In the first method, we use the mean of the contexts (~cl,mean,~cr,mean). In the second method, we use (~wl,mean, ~wr,mean), i.e. the masked representations calculated by using only the mean. On both CONTEXT-PPDB and CONTEXT-WN, our preferred method outperforms the two alternatives by 2-3 F1 points. In fact, on CONTEXT-WN we can see that our method also captures directionality best.
9 Related Work
WordNet and lexical entailment The is-a hierarchy of WordNet (Fellbaum, 1998b) is a prominent source of information for unsupervised detection of hypernymy and entailment (Harabagiu
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Dataset Context aware features Scores Similarities Representations P R F Macro F1 Pairwise Acc.
CONTEXT-PPDB All [~wl,mean ; ~wr,mean] 0.646 0.672 0.643
NA NAAll [~cl,mean ; ~cr,mean] 0.655 0.678 0.652 All [~wl,mask ; ~wr,mask] 0.674 0.694 0.677
CONTEXT-WN All [~wl,mean ; ~wr,mean] 0.674 0.635 0.645 0.641 0.577 All [~cl,mean ; ~cr,mean] 0.688 0.648 0.657 0.665 0.608 All [~wl,mask ; ~wr,mask] 0.700 0.661 0.670 0.677 0.623
Table 6: Impact of masks on CONTEXT-PPDB and CONTEXT-WN.
and Moldovan, 1998; Shwartz et al., 2015), as well as a source of various datasets (Baroni and Lenci, 2011; Baroni et al., 2012). The dataset we introduce in this work is inspired by the latter line of work, but instead of just extracting word pairs we also obtain exemplar contexts from WordNet.
Modeling word meaning in context Several approaches have been proposed to model the meaning of a word in a given context to capture semantic equivalence in tasks such as lexical substitution, word sense disambiguation or paraphrase ranking (but not entailment).
One line of work (Dinu and Lapata, 2010; Reisinger and Mooney, 2010) treats each word as a set of latent word senses. These models methods start with token representations for individual occurrences of a word and then choose a set of token vectors based on the current context. An alternate set of models (Erk and Padó, 2008; Thater et al., 2011; Dinu et al., 2012) avoids defining a fixed set of word senses, and instead contextualizes word type vectors as we do in this paper. These models share the idea of using an element-wise multiplication to apply a context mask to word type representations. The nature of the context representation varies: Erk and Pado (2008) use inverse selectional preferences; Thater et al (2010) combine a first order representation for the context with a second order representation for the target, Thater et al. (2011) rely on syntactic dependencies to define context. Apidianaki (2016) shows that bag-ofword context representation within a small context window works as well as syntactic definitions of context for ranking paraphrases in context.
The use of convolution is motivated by success of similar models on sentence classification tasks. Tang et al (2014) uses convolution over embedding matrices for unigrams, bigrams, and trigrams, while Hovy (2015) uses just unigrams. However, all these works only use convolved representations
to predict properties of the sentence (e.g., sentiment). We use them, instead, to contextualize our target word representations.
In-context lexical semantic tasks Besides entailment, other lexical semantic tasks studied in the presence of context include lexical substitution (McCarthy and Navigli, 2007), cross-lingual lexical substitution (Mihalcea et al., 2010) and paraphrase ranking (Apidianaki, 2016). The last work is also notable because of their successful use of models of word-meaning in context from Thater et al (2011), which is closely related to our methods.
10 Conclusion
We proposed to address lexical entailment in context, providing exemplar sentences to ground the meaning of words being considered for entailment. We show that contextualized word representations constructed by transforming contextagnostic representations, combined with wordcontext similarity features, lead to large improvements over context-agnostic model, not only in English, but also between English and French words on two novel datasets. We also improve the state-of-the-art on a related task of detecting semantic relations in context. Our features are sensitive to changes in entailment based on context, and also capture the directionality of entailment.
In future work, we aim to further improve performance on both monolingual and cross-lingual datasets. We anticipate that richer features, capturing second-order comparisons used to detect lexical contrast (Mohammad et al., 2013) and entailment out of context (Turney and Mohammad, 2013), might be useful for this purpose, perhaps in combination with non-linear classifiers. Given the usefulness of similarities, we also plan to investigate asymmetric scoring functions(Kotlerman et al., 2010; Shwartz et al., 2017).
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

This paper proposes a method for recognizing lexical entailment (specifically, hypernymy) in context. The proposed method represents each context by averaging, min-pooling, and max-pooling its word embeddings. These representations are combined with the target word's embedding via element-wise multiplication. The in-context representation of the left-hand-side argument is concatenated to that of the right-hand-side argument's, creating a single vectorial representation of the input. This input is then fed into a logistic regression classifier.  In my view, the paper has two major weaknesses. First, the classification model used in this paper (concat + linear classifier) was shown to be inherently unable to learn relations in  Do Supervised Distributional Methods Really Learn Lexical Inference Relations?  (Levy et al., 2015). Second, the paper makes superiority claims in the text that are simply not substantiated in the quantitative results. In addition, there are several clarity and experiment setup issues that give an overall feeling that the paper is still half-baked.  = Classification Model =  Concatenating two word vectors as input for a linear classifier was mathematically proven to be incapable of learning a relation between words (Levy et al., 2015). What is the motivation behind using this model in the contextual setting?  While this handicap might be somewhat mitigated by adding similarity features, all these features are symmetric (including the Euclidean distance, since |L-R| = |R-L|). Why do we expect these features to detect entailment?  I am not convinced that this is a reasonable classification model for the task.  = Superiority Claims =  The authors claim that their contextual representation is superior to context2vec. This is not evident from the paper, because:  1) The best result (F1) in both table 3 and table 4 (excluding PPDB features) is the 7th row. To my understanding, this variant does not use the proposed contextual representation; in fact, it uses the context2vec representation for the word type.  2) This experiment uses ready-made embeddings (GloVe) and parameters (context2vec) that were tuned on completely different datasets with very different sizes. Comparing the two is empirically flawed, and probably biased towards the method using GloVe (which was a trained on a much larger corpus).  In addition, it seems that the biggest boost in performance comes from adding similarity features and not from the proposed context representation. This is not discussed.  = Miscellaneous Comments =  - I liked the WordNet dataset - using the example sentences is a nice trick.  - I don t quite understand why the task of cross-lingual lexical entailment is interesting or even reasonable.  - Some basic baselines are really missing. Instead of the  random  baseline, how well does the  all true  baseline perform? What about the context-agnostic symmetric cosine similarity of the two target words?  - In general, the tables are very difficult to read. The caption should make the tables self-explanatory. Also, it is unclear what each variant means; perhaps a more precise description (in text) of each variant could help the reader understand?  - What are the PPDB-specific features? This is really unclear.  - I could not understand 8.1.  - Table 4 is overfull.  - In table 4, the F1 of  random  should be 0.25.  - Typo in line 462: should be  Table 3   = Author Response =  Thank you for addressing my comments. Unfortunately, there are still some standing issues that prevent me from accepting this paper:  - The problem I see with the base model is not that it is learning prototypical hypernyms, but that it's mathematically not able to learn a relation.  - It appears that we have a different reading of tables 3 and 4. Maybe this is a clarity issue, but it prevents me from understanding how the claim that contextual representations substantially improve performance is supported. Furthermore, it seems like other factors (e.g. similarity features) have a greater effect.


TextFlow: A Text Similarity Measure based on Continuous Sequences
Text similarity measures are used in multiple tasks such as plagiarism detection, information ranking and recognition of paraphrases and textual entailment. While recent advances in deep learning highlighted further the relevance of sequential models in natural language generation, existing similarity measures do not fully exploit the sequential nature of language. Examples of such similarity measures include ngrams and skip-grams overlap which rely on distinct slices of the input texts. In this paper we present a novel text similarity measure inspired from a common representation in DNA sequence alignment algorithms. The new measure, called TextFlow, represents input text pairs as continuous curves and uses both the actual position of the words and sequence matching to compute the similarity value. Our experiments on eight different datasets show very encouraging results in paraphrase detection, textual entailment recognition and ranking relevance.
1 Background
The number of pages required to print the content of the World Wide Web was estimated to 305 billion in a 2015 article1. While a big part of this content consists of visual information such as pictures and videos, texts also continue growing at a very high pace. A recent study shows that the average webpage weights 1200KB with plain text accounting for up to 16% of that size2.
While efficient distribution of textual data and computations are the key to deal with the increas-
1http://goo.gl/p9lt7V 2http://goo.gl/c41wpa
ing scale of textual search, similarity measures still play an important role in refining search results to more specific needs such as the recognition of paraphrases and textual entailment, plagiarism detection and fine-grained ranking of information. These tasks also often target dedicated document collections for domain oriented research where text similarity measures can be directly applied.
Finding relevant approaches to compute text similarity motivated a lot of research in the last decades (Sahami and Heilman, 2006; Hatzivassiloglou et al., 1999), and more recently with deep learning methods (Socher et al., 2011; Yih et al., 2011; Severyn and Moschitti, 2015). However, most of the recent advances focused on designing high performance classification methods, trained and tested for specific tasks and did not offer a standalone similarity measure that could be applied (i) regardless of the application domain and (ii) without requiring training corpora.
Yih and Meek (2007) presented an approach to improve text similarity measures for web search queries, with a length ranging from one word to short sequences of words. The proposed method is tailored to this specific task, and the training models are not expected to perform well on different kinds of data such as sentences, questions or paragraphs. Achananuparp et al. (2008) compared several text similarity measures for paraphrase recognition, textual entailment, and the TREC 9 question variants task. In their experiments the best performance was obtained with a linear combination of semantic and lexical similarities, including a word order similarity proposed in (Li et al., 2006) which constructs two vectors of the common words between two sentences and uses their respective positions in the sentences as term weights. The word order similarity measure is then computed by subtracting the two vec-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
tors and taking the absolute value. While such representation takes into account the actual positions of the words, it does not allow detecting subsequence matches and takes into account missing words only by omission.
More generally, existing standalone (or traditional) text similarity measures rely on intersections between token sets, text sizes, and frequency, including measures such as the Cosine similarity, Euclidean distance, Levenshtein, Jaccard and Jaro (Jaro, 1989). The sequential nature of natural language is taken into account mostly through word n-grams and skip-grams which capture distinct slices of the analysed texts but do not preserve the order in which they appear.
In this paper, we use intuitions from a common representation in DNA sequence alignment to design of a new standalone similarity measure called TextFlow (XF). The proposed measure uses at the same time the full sequence of input texts in a natural sub-sequence matching approach together with individual token matches and mismatches. Our contributions can be detailed further as follows:
• A novel standalone similarity measure which:
– exploits the full sequence of words in the compared texts.
– is asymmetric in a way that allows it to provide the best performance on different tasks (e.g., paraphrase detection, textual entailment and ranking).
– when required, it can be trained with a small set of parameters controlling the impact of sub-sequence matching, position gaps and unmatched words.
– provides consistent high performance across tasks and datasets compared to traditional similarity measures.
• A neural network architecture to train TextFlow parameters for specific tasks.
• An empirical study on both performance consistency and standard evaluation measures, performed with eight datasets from three different tasks.
• A new evaluation measure, called CORE, used to better show the consistency of a system at high performance using both its rank
2 The TextFlow Similarity
XF is inspired from a dot matrix representation commonly used in pairwise DNA sequence alignment (cf. figure 2). We use a similar dot matrix representation for text pairs and draw a curve oscillating around the diagonal (cf. figure 2). The area under the curve is considered to be the distance between the two text pairs which is then normalized with the matrix surface. For practical computation, we transform this first intuitive representation using the delta of positions as in figure 3. In this setting, the Y axis is the delta of positions of a word occurring in the two texts being compared. If the word does not occur in the target text, the delta is considered to be a maximum reference value (l in figure 2).
The semantics are: the bigger the area under curve is, the lower the similarity between the compared texts. XF values are real numbers in the [0,1] interval, with 1 indicating a perfect match, and 0 indicating that the compared texts do not have any common token. With this representation, we are able to take into account all matched words and sub-sequences at the same time. The exact value for the XF similarity between two texts X = {x1, x2, .., xn} and Y = {y1, y2, .., ym} is therefore computed as:
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Figure 2: Illustration of TextFlow Intuition
Figure 3: Practical TextFlow Computation
XF (X,Y ) = 1− 1 nm n∑ i=2 1 Si Ti,i−1(X,Y )
− 1 nm n∑ i=2 1 Si Ri,i−1(X,Y ) (1)
With Ti,i−1(X,Y ) corresponding to the triangular area in the [i − 1, i] step (cf. figure 3) and Ri,i−1(X,Y ) corresponding to the rectangular component. They are expressed as:
Ti,i−1(X,Y ) = |∆P (xi, X, Y )−∆P (xi−1, X, Y )|
2 (2)
and:
Ri,i−1(X,Y ) = Min(∆P (xi, X, Y ),∆P (xi−1, X, Y )) (3)
With:
• ∆P (xi, X, Y ) the minimum difference between xi positions in X and Y . If xi /∈ X ∩ Y , ∆P (xi, X, Y ) is set to the same reference value equal to m, (i.e., the cost of a
missing word is set by default to the length of the target text), and:
• Si is the length of the longest matching sequence between X and Y including the word xi, if xi ∈ X ∩ Y , or 1 otherwise.
XF computation is performed in O(nm) in the worst case where we have to check all tokens in the target text Y for all tokens in the input text X . XF is an asymmetric similarity measure. Its asymmetric aspect has interesting semantic applications as we show in the example below (cf. figure 2). The minimum value of XF provided the best differentiation between positive and negative text pairs when looking for semantic equivalence (i.e., paraphrases), the maximum value was among the the top three for the textual entailment example. We conduct this comparison at a larger scale in the evaluation section.
We add 3 parameters to XF in order to represent the importance that should be given to position deltas (Position factor α), missing words (sensitivity factor β), and sub-sequence matching (sequence factor γ), such as:
XFα,β,γ(X,Y ) = 1− 1
βnm n∑ i=2 α Sγi T βi,i−1(X,Y )
− 1 βnm n∑ i=2 α Sγi Rβi,i−1(X,Y )
(4)
With:
T βi,i−1(X,Y ) = |∆βP (xi, X, Y )−∆βP (xi−1, X, Y )|
2 (5)
Rβi,i−1(X,Y ) = Min(∆βP (xi, X, Y ),∆βP (xi−1, X, Y )) (6) and:
• ∆βP (xi, X, Y ) = βm, if xi /∈ X ∩ Y
• α < β: forces missing words to always cost more than matched words.
• Sγi = {
1ifSi = 1orxi /∈ X ∩ Y γ SiforSi > 1
The γ factor increases or decreases the impact of sub-sequence matching, α applies to individual token matches whether inside or outside a sequence, and β increases or decreases the impact of
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Positive Entailment E1 Under a blue sky with white clouds, a child reaches up to touch the propeller of a plane
standing parked on a field of grass. E2 A child is reaching to touch the propeller of a plane.
Negative Entailment E3 Two men on bicycles competing in a race. E4 Men are riding bicycles on the street. Positive Paraphrase P1 The most serious breach of royal security in recent years occurred in 1982 when 30-
year-old Michael Fagan broke into the queen’s bedroom at Buckingham Palace. P2 It was the most serious breach of royal security since 1982 when an intruder, Michael
Fagan, found his way into the Queen’s bedroom at Buckingham Palace.
Negative Paraphrase P3 “Americans don’t cut and run, we have to see this misadventure through,” she said. P4 She also pledged to bring peace to Iraq: “Americans don’t cut and run, we have to see
this misadventure through.”
Task Entailment Recognition Paraphrase Detection Sentence Pair (E1, E2) (E3, E4) (E1, E2) - (E3, E4) (P1, P2) (P3, P4) (P1, P2) - (P3, P4) Example class (Pos/Neg) (Pos) (Neg) (Gap) (Pos) (Neg) (Gap) Jaro-Winkler 0.629 0.712* -0.083** 0.884 0.738 0.146 Levenshtein 0.351 0.259 0.092 0.708 0.577 0.130 Jaccard 0.250* 0.143 0.107 0.571* 0.583 -0.012 Cosine 0.462 0.250 0.212 0.730 0.746** -0.016 Word Overlap 0.800 0.250 0.550 0.800 0.875* -0.075 MIN(XF (x,y), XF(y,x)) 0.260** 0.563** -0.303* 0.693** 0.497 0.196 MAX(XF(x,y), XF(y,x)) 0.707 0.563** 0.144 0.832 0.739 0.093
Figure 4: Example sentences and similarity values. The best value per column is highlighted. The second best is underlined. Worst and second worst values are followed by one and two stars. Entailment examples are taken from SNLI (Bowman et al., 2015). Paraphrase examples are taken from MSRP 4.
missing tokens as well as the normalization quantity βnm in equation 4 to keep the similarity values in the [0,1] range.
2.1 Parameter Training
By default XF has canonical parameters set to 1. However, when needed, α, β, and γ can be trained on learning data for a specific task. We designed a neural network to perform this task, with a hidden layer dedicated to compute the exact XF value. To do so we compute, for each input text pair, the coefficients vector that would lead exactly to the XF value when multiplied by the vector< αβ , α βγ , β >. Figure 5) presents the training neural network considering several types of sequences (or translations) of the input text pairs (e.g., lemmas, words, synsets).
We use identity as activation function in the dedicated XF layer in order to have a correct comparison with the other similarity measures, including canonical XF where the similarity value is provided in the input layer (cf. figure 6).
3 Evaluation
Datasets. This evaluation was performed on 8 datasets from 3 different classification tasks: Tex-
tual Entailment Recognition, Paraphrase Detection, and ranking relevancy. The datasets are as follows:
• RTE 1, 2, and 3: the first three datasets from the Recognizing Textual Entailment (RTE) challenge (Dagan et al., 2006). Each dataset consists of sentence pairs which are annotated with 2 labels: entailment, and nonentailment. They contain respectively (200, 800), (800, 800), and (800, 800) (train, test) pairs.
• Guardian: an RTE dataset collected from 78,696 Guardian articles5 published from January 2004 onwards and consisting of 32K pairs which we split randomly in 90%/10% training/test sets. Positive examples were collected from the titles and first sentences. Negative examples were collected from the same source by selecting consecutive sentences and random sentences.
• SNLI: a recent RTE dataset consisting of 560K training sentence pairs annotated with
5https://github.com/daoudclarke/ rte-experiment
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Figure 5: NN architecture A1 for XF Parameter Training
3 labels: entailment, neutral and contradiction (Bowman et al., 2015). We discarded the contradiction pairs as they do not necessarily represent dissimilar sentences and are therefore a random noise w.r.t. our similarity measure evaluation.
• MSRP: the Microsoft Research Paraphrase corpus, consisting of 5,800 sentence pairs annotated with a binary label indicating whether the two sentences are paraphrases or not.
• Semeval-16-3B: a dataset of questionquestion similarity collected from StackOverflow (Nakov et al., 2016). The dataset was contains 3,169 training pairs and 700 test pairs. Three labels are considered: ”Perfect Match”, ”Relevant” or ”Irrelevant”. We combined the first two into the same positive category for our evaluation.
• Semeval-14-1: a corpus of Sentences Involving Compositional Knowledge (Marelli et al., 2014) consisting of 10,000 english sentence pairs annotated with both similarity scores and relevance labels.
Features. After a preprocessing step where we removed stopwords, we computed the similarity values using 7 different types of sequences constructed, respectively, with the following value from each token:
• Word (plain text value)
• Lemma
• Part-Of-Speech (POS) tag
• WordNet Synset6 OR Lemma
• WordNet Synset OR Lemma for Nouns
• WordNet Synset OR Lemma for Verbs
• WordNet Synset OR Lemma for Nouns and Verbs.
In the 4 last types of sequences the lemma is used when there is no corresponding WordNet synset. In a first experiment we compare different aggregation functions on top of XF (i.e., minimum, maximum and average) in table 3. We used the LibLinear7 SVM classifier for this task.
In the second part of the evaluation, we use neural networks to compare the efficiency of XFc, XFt and other similarity measures with in the same setting. We use the neural net described in figure 5 for the trained versionXFt and the equivalent architecture presented in figure 6 for all other similarity measures. For canonical XFc we use by default the best aggregation for the task as observed in table 3.
6https://wordnet.princeton.edu/ 7https://www.csie.ntu.edu.tw/˜cjlin/
liblinear/
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Task Entailment Recognition Paraphrase Detection Ranking Relevance Datasets RTE 1 RTE 2 RTE 3 Guardian SNLI MSRP Semeval16-t3B Semeval12-t17 XF MIN 55.3 53.8 60.0 77.3 58.0 72.1 77.4 77.8 XF AVG 51.4 1 57.2 62.5 84.9 62.0 72.0 77.6 79.5 XF MAX 53.9 61.3 64.7 86.7 64.3 71.4 76.7 77.7
Table 1: Accuracy evaluation with different aggregations of XF using an SVM classifier.
Figure 6: NN Architecture A2 for the equivalent evaluation of other similarity measures.
Similarity Measures. We considered nine traditional similarity measures included in the Simmetrics distribution8: Cosine, Euclidean distance, Word Overlap, Dice coefficient (Dice, 1945), Jaccard(Jain and Dubes, 1988), Damerau, Jaro-Winkler (JW) (Porter et al., 1997), Levenshtein (LEV) (Sankoff and Kruskal, 1983), and Longest Common Subsequence (LCS) (Friedman and Sideli, 1992). Implementation. XF was implemented in Java as an extension of the Simmetrics package, made available at this address9. The neural neworks were implemented in Python with TensorFlow10. We also share the training sets used for both parameter training and evaluation. The evaluation was performed on a 4-cores laptop with 32GB of RAM. The initial parameters forXFt were chosen with a random function. Evaluation Measures. We use the standard accuracy values and F1, precision and recall for the
8https://github.com/Simmetrics/ simmetrics
9Code link omitted for blind review. 10https://www.tensorflow.org/
positive class (i.e., entailment, paraphrase, and ranking relevance). We also study the relative rank in performance of each similarity measure across all datasets using the average rank, the rank variance and a new evaluation measure called cOnsistent peRformancE (CORE), computed as follows for a system m, a set of datasets D, a set of systems S, and an evaluation measure E ∈ {F1, P recision,Recall, Accuracy}:
CORE D,S,E (m) =
MIN p∈S ( AVG d∈D (RS(Ed(p)) + Vd∈D(RS(Ed(p))) ) AVG d∈D ( RS(Ed(m)) ) + Vd∈D ( RS(Ed(m))
) (7) With RS(Ed(m)) the rank of m according to the evaluation measure E on dataset d w.r.t. competing systems S. Vd∈D(RS(Ed(m))) is the rank variance of m over datasets. The intuition can be read on the results tables 2, 3, and 4. Basically, CORE tells us how consistent a system/method is in having high performance, relatively to the set of competing systems S. The maximum value of CORE is 1 for the best performing system according to its rank. It also allows quantifying how consistently a system achieves high performance for the remaining systems.
Tensorflow outperformed the results obtained with a combination of word order similarity and semantic similarities tested in (Achananuparp et al., 2008), with gaps of +1.0 in F1 and +6.1 accuracy on MSRP and +4.2 F1 and +2.7% accuracy on RTE 3.
4 Discussion
Canonical Text Flow. TFc had the best average and micro-average accuracy on the 8 classification datasets, with a gap of +0.4 to +6.3 when compared to the state-of-the-art measures. It also reached the best precision average with a gap of +1.8 to +6.3. On the F1 score level XFc achieved the second best performance with a gap of -1.7, mainly caused by its underperformance in recall, where it had the third best performance with a gap of -6.3. The recall table is omitted due to a lack
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Cosine Euc Overlap Dice Jaccard Damerau JW LEV LCS XFC XFT RTE 1 .561 .564 .550 .504 .511 .557 .532 .561 .568 .550 .575 RTE 2 .575 .555 .598 .566 .572 .548 .541 .551 .548 .597 .612 RTE 3 .652 .562 .636 .637 .630 .567 .538 .567 .562 .627 .647 Guardian .748 .750 .820 .778 .780 .847 .726 .847 .848 .867 .876 SNLI .621 .599 .665 .612 .608 .631 .556 .630 .619 .641 .656 MSRP .719 .689 .720 .729 .731 .687 .699 .685 .717 .724 .732 Semeval-16-3B .756 .734 .769 .781 .780 .759 .751 .759 .737 .777 .782 Semeval-14-1 .790 .756 .779 .783 .786 .749 .719 .749 .757 .783 .798 AVG .678 .651 .692 .674 .675 .668 .633 .669 .670 .696 .710 Micro Avg .699 .675 .725 .700 .700 .701 .646 .701 .701 .726 .739 RANK Avg 5.1 8.2 4.5 5.6 5.5 6.9 10.1 6.7 6.7 4.1 1.2 RANK Var. 9.0 5.9 4.3 10.0 8.6 5.3 1.6 6.2 8.2 2.7 0.2 CORE 0.104 0.103 0.167 0.094 0.104 0.121 0.125 0.113 0.098 0.215 1.000
Cosine Euc Overlap Dice Jaccard Damerau JW LEV LCS XFC XFT RTE 1 .612 .564 .636 .512 .523 .578 .513 .583 .494 .565 .599 RTE 2 .579 .590 .662 .565 .558 .549 .516 .551 .555 .616 .646 RTE 3 .705 .598 .682 .695 .682 .608 .556 .607 .603 .665 .690 Guardian .742 .749 .816 .774 .776 .849 .713 .849 .850 .862 .873 SNLI .582 .576 .641 .562 .564 .627 .479 .627 .611 .594 .585 MSRP .808 .797 .812 .814 .813 .784 .802 .783 .804 .804 .810 Semeval-16-3B .632 .462 .625 .648 .644 .544 .545 .547 .508 .633 .662 Semeval-14-1 .764 .707 .748 .753 .746 .706 .680 .706 .714 .744 .673 AVG .678 .630 .702 .665 .663 .655 .600 .656 .642 .685 .692 Micro Avg .684 .656 .716 .679 .677 .691 .608 .692 .688 .702 .687 RANK Avg 4.5 8.12 3.12 5.12 5.5 6.89 9.88 6.62 7.12 4.62 3.88 RANK Var. 9.7 4.7 4.4 14.7 6.6 8.7 1.8 9.1 8.1 2.3 11.0 CORE 0.485 0.538 0.915 0.348 0.571 0.443 0.588 0.438 0.452 1.000 0.464
Table 3: F1 scores. The best result is highlighted, the second best is underlined.
Cosine Euc Overlap Dice Jaccard Damerau JW LEV LCS XFC XFT RTE 1 .548 .564 .534 .503 .510 .552 .535 .555 .596 .546 .566 RTE 2 .574 .547 .571 .567 .578 .547 .546 .551 .546 .588 .594 RTE 3 .624 .565 .618 .611 .610 .568 .547 .568 .564 .616 .627 Guardian .759 .753 .836 .789 .789 .839 .749 .840 .839 .891 .894 SNLI .644 .608 .690 .642 .632 .631 .577 .630 .621 .679 .735 MSRP .740 .705 .732 .749 .755 .723 .713 .722 .743 .760 .765 Semeval-16-3B .634 .708 .678 .698 .698 .732 .698 .729 .674 .700 .686 Semeval-14-1 .745 .738 .738 .743 .769 .716 .672 .716 .727 .762 .740 AVG .659 .649 .675 .663 .668 .664 .630 .664 .664 .693 .701 Micro Avg .693 .674 .721 .699 .704 .694 .645 .693 .693 .737 .752 RANK Avg. 5.6 7.5 5.9 5.9 5.1 6.1 9.6 6.1 7.1 3.2 2.5 RANK Var. 9.4 10.0 6.4 5.3 7.8 7.0 4.6 7.6 11.6 3.1 6.9 CORE 0.420 0.361 0.515 0.567 0.488 0.482 0.446 0.462 0.338 1.000 0.676
Table 4: Precision values. The best result is highlighted, the second best is underlined.
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
of space. On a rank level, XFc had the best consistent rank for accuracy F1 and precision, and the second best for recall.
Trained Text Flow. When compared to soa measures and to canonical XF, the trained version, XFt, obtained the best accuracy with a gap ranging from +1.4 to +7.8. XFt also obtained the second best F1 average with a -1.0 gap, but with clear inconsistencies according to the dataset. XFt obtained the best precision with a gap ranging from +0.8 to +7.1. XFt did not perform well on recall with 64.5% micro-average compared to WordOverlap with 72%. Both its recall and F1 performance can be explained by the fact that the measure was trained to optimize accuracy, and not the F1 score for the positive class; which also suggests that the approach could be adapted to F1 optimization if needed. Canonical XF was more consistent than trained XF on all dimensions except accuracy, for which XFt was optimized. We argue that this consistency was made possible through the asymmetry of XF which allowed it to adapt to different kinds of similarities (i.e., equivalence/paraphrase, inference/entailment, and mutual distance/ranking). These results also show that the actual positions difference is a relevant factor for text similarity. We explain it mainly by the natural flow of language where the important entities and relations are often expressed first, in contrast with a purely logical-driven approach which has to consider, for instance, that active forms and passive forms are equivalent in meaning. The difference in positions is also not read literally, both because of the higher impact associated to missed words and to the α parameter which allows leveraging their impact in the trained version.
In additional experiments, we compared TFc and TFt with the other similarity measures when applied to bi-grams and tri-grams instead of individual tokens. The results were significantly lower on all datasets (between 3 and 10 points loss in accuracy) for both the soa measures and TextFlow variants. This result could be explained by the fact that n-grams are too rigid when a sub-sequence varies even slightly, e.g., the insertion of a new word inside a 3-words sequence leads to a tri-gram mismatch and reduces bi-gram overlap from 100% to 50% for the considered sub-sequence. This issue is not encountered with TextFlow as it relies on the token level, and such an insertion will
not cancel or reduce significantly the gains from the correct ordering of the words. We also conducted ranking correlation experiments on three test datasets provided at the semantic text similarity task at Semeval 2012, with gold score values for their text pairs. The datasets have 750 sentence pairs each, and are extracted from the Microsoft Research video descriptions corpus, MSRP and the SMTeuroparl11. When compared to the traditional similarity measures, TextFlow had the best correlation on the first two datasets with, for instance, 0.54 and 0.46 pearson correlation on the lemmas sequences and the second best on the MSRP extract where the Cosine similarity had the best performance with 0.71 vs 0.68, noting that the Cosine similarity uses word frequencies when the evaluated version of TextFlow did not use wordlevel weights.
Including word weights is one of the promising perspectives in line with this work as it could be done simply by making the deltas vary according to the weight/importance of the (un)matched word. Also, in such setting, the impact of a sequence of N words will naturally increase or decrease according to the word weights (cf. figure 3). We conducted a preliminary test using the inverse document frequency of the words as extracted from Wikipedia with Gensim12, which led to an improvement of up to 2% for most datasets, with performance decreasing slightly on two of them. Other kinds of weights could also be included just as easily, such as contextual word relatedness using embeddings or other semantic relatedness factors such as WordNet distances (Pedersen et al., 2004).
5 Conclusion
We presented a novel standalone similarity measure that takes into account continuous word sequences. Evaluation on eight datasets show promising results for textual entailment recognition, paraphrase detection and ranking. Among the potential extensions of this work are the inclusion of different kinds of weights such as TF-IDF, embedding relatedness and semantic relatedness. We also intend to test other variants around the same concept, including considering the matched words and sequences to have a negative weight to balance further the weight of missing words.
11goo.gl/NVnybD 12https://radimrehurek.com/gensim/
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
Acknowledgements

- Strengths:  originality of the CORE evaluation measure, good accuracy of proposed similarity measure and large number and diversity of datasets for evaluation.  - Weaknesses:    # some typos    - line 116-117, 'to design of a new' to 'to design a new'    - line 176-177, figure 2 to figure 1    - line 265, 'among the the top' to 'among the top'    - line 320, 'figure 4' should be introduced within the article body.    - line 434, 'the dataset was contains' to 'the dataset contains'    - line 486-487, table 3 to table 1    - a 'Tensorflow' should be replaced by 'TextFlow'   # imprecisions    - features computation accuracy of lemma, pos or wordnet synset should be detailed in the paper and it should be discussed if it impacts the general similarity accuracy evaluation or not   - the neural networks are said to be implemented in Python but the code is not said to be available - to be able to repeat the experiment   - the training and evaluation sets are said to be shared, but it is not said how (on demand?, under license?) - to be able to repeat the experiment  - General Discussion: