Enhanced LSTM for Natural Language Inference
Reasoning and inference are central to human and artificial intelligence. Modeling inference in human language is very challenging. With the availability of large annotated data (Bowman et al., 2015), it has recently become feasible to train neural network based inference models, which have shown to be very effective. In this paper, we present a new state-of-the-art result, achieving the accuracy of 88.6% on the Stanford natural language inference dataset. Unlike the previous top models that use very complicated network architectures, we first demonstrate that carefully designing sequential inference models based on chain LSTMs can outperform all previous models. Based on this, we further show that by explicitly considering recursive architectures in both local inference modeling and inference composition, we achieve additional improvement. Particularly, incorporating syntactic parsing information contributes to our best result—it further improves the performance even when added to the already very strong model.
1 000
010
011
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033
034
035
036
037
038
039
040
041
042
043
044
045
046
047
048
049
060
061
062
063
064
065
066
067
068
069
070
071
072
073
074
075
076
077
078
079
080
081
082
083
084
085
086
087
088
089
090
091
092
093
094
095
096
097
098
099
1 Introduction
Reasoning and inference are central to both human and artificial intelligence. Modeling inference in human language is notoriously challenging but is a basic problem towards true natural language understanding, as pointed out by MacCartney and Manning (2008), “a necessary (if not sufficient) condition for true natural language understanding is a mastery of open-domain natural language inference.” The efforts have also included a large bulk of work on recognizing textual entailment.
Specifically, natural language inference (NLI) is concerned with determining whether a naturallanguage hypothesis h can be inferred from a
premise p, as depicted in the following example from MacCartney (2009), where the hypothesis is regarded to be entailed from the premise.
p: Several airlines polled saw costs grow more than expected, even after adjusting for inflation.
h: Some of the companies in the poll reported cost increases.
The most recent years have seen advances in modeling natural language inference. An important contribution is the creation of a much larger annotated dataset, the Stanford Natural Language Inference (SNLI) dataset (Bowman et al., 2015). The corpus has 570,000 human-written English sentence pairs manually labeled by multiple human subjects. This makes it feasible to train more complex inference models. Neural network models, which often need relatively large annotated data to estimate their parameters, have shown to achieve the state of the art on SNLI (Bowman et al., 2015, 2016; Munkhdalai and Yu, 2016b; Parikh et al., 2016; Sha et al., 2016; Paria et al., 2016).
While some previous top-performing models use rather complicated network architectures to achieve the state-of-the-art results (Munkhdalai and Yu, 2016b), we demonstrate in this paper that enhancing sequential inference models based on chain models can outperform all previous results, suggesting that the potential of such sequential inference approaches have not been fully exploited yet. Our model may serve as a new baseline or starting point for deploying more complicated models for NLI. More specifically, we show that our sequential inference model achieves an accuracy of 88.0% on the SNLI benchmark.
Exploring syntax for NLI is very attractive to us. In many problems, syntax and semantics interact closely, as generally phrased in the slogan “the syntax and the semantics work together in tandem” (Barker and Jacobson, 2007), among others. Complicated tasks such as natural language
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
inference could well involve both, which has been discussed in the context of recognizing textual entailment (RTE) (Mehdad et al., 2010; Ferrone and Zanzotto, 2014). In this paper, we are interested in exploring this within the neural network frameworks, with the presence of relatively large training data. We show that by explicitly encoding parsing information with recursive networks in both local inference modeling and inference composition, and incorporating it into our framework, we achieve additional improvement, increasing the performance to a new state of the art with a 88.6% accuracy.
2 Related Work
Early work on natural language inference has been performed on rather small datasets with more conventional methods (refer to MacCartney (2009) for a good literature survey), which includes a large bulk of early work on recognizing textual entailment, such as (Dagan et al., 2005; Iftene and Balahur-Dobrescu, 2007), among others. More recently, Bowman et al. (2015) made available the SNLI dataset with the 570,000 human annotated sentence pairs. They also experimented with simple classification models as well as simple neural networks that encode the premise and hypothesis independently. Rocktäschel et al. (2015) proposed neural attention-based models for NLI, which captured the attention information. In general, attention based models have been shown to be effective in a wide range of tasks, including machine translation (Bahdanau et al., 2014), speech recognition (Chorowski et al., 2015; Chan et al., 2016), image caption (Xu et al., 2015), and text summarization (Rush et al., 2015; Chen et al., 2016), among others. For NLI, the idea allows neural models to pay attention to specific areas of the sentences.
A variety of more advanced networks have been developed since then (Bowman et al., 2016; Vendrov et al., 2015; Mou et al., 2016; Liu et al., 2016; Munkhdalai and Yu, 2016a), and inter-sentence attention-based models (Rocktäschel et al., 2015; Wang and Jiang, 2016; Cheng et al., 2016; Parikh et al., 2016; Munkhdalai and Yu, 2016b; Sha et al., 2016; Paria et al., 2016). Sha et al. (2016) proposes a special LSTM variant which considers the attention vector of another sentence as an inner state of LSTM. Paria et al. (2016) use a neural architecture with a complete binary tree-LSTM encoders while without syntactic information. Among them, more relevant to our work are the approaches proposed
by Parikh et al. (2016) and Munkhdalai and Yu (2016b), which are among the best models.
Parikh et al. (2016) propose a relatively simple but very effective decomposable model. The model decomposes the NLI problem into subproblems that can be solved separately. On the other hand, Munkhdalai and Yu (2016b) propose much more complicated networks that consider sequential LSTM-based encoding, recursive networks, and complicated combinations of attention models, which provide about 0.5% gain over the results reported by Parikh et al. (2016).
It is, however, not very clear if the potential of the sequential inference networks has been well exploited for NLI. In this paper, we first revisit this problem and show that enhancing sequential inference models based on chain networks can actually outperform all previous results. Our model may serve as a new baseline or starting point for deploying more complicated models for NLI. We further show that explicitly considering recursive architectures to encode syntactic parsing information for NLI could further improve the performance.
3 Hybrid Neural Inference Models
We present here our natural language inference networks which are composed of the following major components: input encoding, local inference modeling, and inference composition. Figure 1 shows a high-level view of the architecture. Vertically, the figure depicts the three major components, and horizontally, the left side of the figure represents our sequential NLI model named ESIM, and the right side represents networks that incorporate syntactic parsing information in tree LSTMs.
In our notation, we have two sentences a = (a1, . . . ,a`a) and b = (b1, . . . ,b`b), where a is a premise and b a hypothesis. The ai or bj ∈ Rl is an embedding of l-dimensional vector, which can be initialized with some pre-trained word embeddings and organized with parse trees. The goal is to predict a label y that indicates the logic relationship between a and b.
3.1 Input Encoding
We employ bidirectional LSTM (BiLSTM) as one of our basic building blocks for NLI. We first use it to encode the input premise and hypothesis (Equation (1) and (2)). Here BiLSTM learns to represent a word (e.g., ai) and its context. Later we will also use BiLSTM to perform inference composition to
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Figure 1: A high-level view of our hybrid neural inference networks.
construct the final prediction, where BiLSTM encodes local inference information and its interaction. To bookkeep the notations for later use, we write as āi the hidden (output) state generated by the BiLSTM at time i over the input sequence a. The same is applied to b̄j :
āi = BiLSTM(a, i),∀i ∈ [1, . . . , `a], (1) b̄j = BiLSTM(b, j),∀j ∈ [1, . . . , `b]. (2)
Due to the space limit, we will skip the description of the basic chain LSTM and readers can refer to Hochreiter and Schmidhuber (1997) for details. Briefly, when modeling a sequence, an LSTM employs a set of soft gates together with a memory cell to control message flows, resulting in an effective modeling of tracking long-distance information/dependencies in a sequence.
A bidirectional LSTM runs a forward and backward LSTM on a sequence starting from the left and the right end, respectively. The hidden states generated by these two LSTMs at each time step are concatenated to represent that time step and its context. Note that we used LSTM memory blocks in our models. We examined other recurrent memory blocks such as GRUs (Gated Recurrent Units) (Cho et al., 2014) and they are inferior to LSTMs on the heldout set for our NLI task.
As discussed above, it is intriguing to explore the effectiveness of syntax for natural language inference; for example, whether it is useful even when incorporated into the best-performing models. To this end, we will also encode syntactic parse trees of a premise and hypothesis through treeLSTM (Zhu et al., 2015; Tai et al., 2015; Le and Zuidema, 2015), which extends the chain LSTM to a recursive network (Socher et al., 2011).
Specifically, given the parse of a premise or hypothesis, a tree node is deployed with a tree-LSTM memory block depicted as in Figure 2 and computed with Equations (3–10). In short, at each node, an input vector xt and the hidden vectors of its two children (the left child hLt−1 and the right h R t−1) are taken in as the input to calculate the current node’s hidden vector ht.
We describe the updating of a node at a high level with Equation (3) to facilitate references later in the paper, and the detailed computation is described in (4–10). Specifically, the input of a node is used to configure four gates: the input gate it, output gate ot, and the two forget gates fLt and f R t . The memory cell ct considers each child’s cell vector, cLt−1 and c R t−1, which are gated by the left forget gate fLt and right forget gate f R t , respectively.
ht = TrLSTM(xt,hLt−1,h R t−1), (3)
ht = ot tanh(ct), (4) ot = σ(Woxt + U L o h L t−1 + U R o h R t−1), (5) ct = f L t cLt−1 + fRt cRt−1 + it ut, (6) fLt = σ(Wfxt + U LL f h L t−1 + U LR f h R t−1), (7) fRt = σ(Wfxt + U RL f h L t−1 + U RR f h R t−1), (8)
it = σ(Wixt + U L i h L t−1 + U R i h R t−1), (9)
ut = tanh(Wcxt + U L c h L t−1 + U R c h R t−1), (10)
where σ is the sigmoid function, is the elementwise multiplication of two vectors, and all W ∈ Rd×l, U ∈ Rd×d are weight matrices to be learned.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
In the current input encoding layer, xt is used to encode a word embedding for a leaf node. Since a non-leaf node does not correspond to a specific word, we use a special vector x′t as its input, which is like an unknown word. However, in the inference composition layer that we discuss later, the goal of using tree-LSTM is very different and the input xt will be very different as well—it will encode local inference information and will have values at all tree nodes.
3.2 Local Inference Modeling
Carefully modeling local subsentential inference between a premise and hypothesis is critical to help determine the overall inference between these two statements.
To closely examine local inference, we explore both the sequential and syntactic tree models that have been discussed above. The former helps collect local inference for words and their context, and the tree LSTM helps collect local information between (linguistic) phrases and clauses.
Locality of inference Modeling local inference needs to employ some forms of hard or soft alignment to associate the relevant subcomponents between a premise and a hypothesis. This includes early methods motivated from the alignment in conventional automatic machine translation (MacCartney, 2009). In neural network models, this is often achieved with soft attention.
Parikh et al. (2016) decomposed this process: the word sequence of the premise (or hypothesis) is regarded as a bag-of-word embedding vector and inter-sentence “alignment" (or attention) is computed individually to softly align each word to the content of hypothesis (or premise, respectively). While their basic framework is very effective, achieving one of the previous best results, using a pre-trained word embedding by itself does not automatically consider the context around a word in NLI. Parikh et al. (2016) did take into account the word order and context information through an optional distance-sensitive intra-sentence attention.
In this paper, we argue for leveraging attention over the bidirectional sequential encoding of the input, as discussed above. We will show that this plays an important role in achieving our best results, and the intra-sentence attention used by (Parikh et al., 2016) actually does not further improve over our model, while the overall framework they proposed is very effective.
Our soft alignment layer computes the attention weights as the similarity of a hidden state tuple <āi, b̄j> between a premise and a hypothesis with Equation (11). We did study more complicated relationships between āi and b̄j with multilayer perceptrons, but observed no further improvement on the heldout data.
eij = ā T i b̄j (11)
āi and b̄j are computed above in Equations (1) and (2), or with Equation (3) when tree-LSTM is used. Again, as discussed above, we will use bidirectional LSTM and tree-LSTM to encode the premise and hypothesis, respectively. In our sequential inference model, unlike in Parikh et al. (2016) which proposed to use a function F (āi), i.e., a feed-forward neural network, to map the original word representation for calculating eij , we instead advocate to use BiLSTM, which encodes the information in premise and hypothesis very well and achieves better performance shown in the experiment section; we tried to apply the F (.) function on our hidden states before computing eij and it did not further help our models. Local inference collected over sequences Local inference is determined by the attention weight eij computed above, which is used to obtain the local relevance between a premise and hypothesis. For the hidden state of a word in a premise, i.e., āi (already encoding the word itself and its context), the relevant semantics in the hypothesis is identified and composed using eij , more specifically with Equation (12).
ãi = `b∑ j=1 exp(eij)∑`b k=1 exp(eik) b̄j ,∀i ∈ [1, . . . , `a], (12)
b̃j = `a∑ i=1 exp(eij)∑`a k=1 exp(ekj) āi,∀j ∈ [1, . . . , `b], (13)
where ãi is a weighted summation of {b̄j}`bj=1. Intuitively, the content in {b̄j}`bj=1 that is relevant to āi will be selected and represented as ãi. The same is performed for each word in the hypothesis with Equation (13). Local inference collected over parse trees We use tree models to help collect local inference information over linguistic phrases and clauses in this layer. The tree structures of the premise and hypothesis are produced by a constituency parser.
Once the hidden states of a tree are all computed with Equation (3), we treat all tree nodes equally
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
as we do not have further heuristics to discriminate them, but leave the attention weights to figure out their relationship. So, we use Equation (11) to compute the attention weights for all node pairs between a premise and hypothesis. This connects all words, constituent phrases, and clauses between the premise and hypothesis. We then collect the information between all the pairs with Equations (12) and (13) and feed them into the next layer. Enhancement of local inference information In our models, we further enhance the local inference information collected. We compute the difference and the element-wise product for the tuple <ā, ã> as well as for <b̄, b̃>. We expect that such operations could help sharpen local inference information between elements in the tuples and capture inference relationships such as contradiction. The difference and element-wise product are then concatenated with the original vectors, ā and ã, or b̄ and b̃, respectively. The enhancement is performed for both the sequential and the tree models.
ma = [ā; ã; ā− ã; ā ã], (14) mb = [b̄; b̃; b̄− b̃; b̄ b̃]. (15)
This process could be regarded as a special case of modeling some high-order interaction between the tuple elements. Along this direction, we have also further modeled the interaction by feeding the tuples into feed-forward networks and added the top layer hidden states to the above concatenation. We found that it does not further help the inference accuracy on the heldout dataset.
3.3 Inference Composition
To determine the overall inference relationship between a premise and hypothesis, we explore a composition layer to compose the enhanced local inference information ma and mb. We perform the composition sequentially or in its parse context using BiLSTM and tree-LSTM, respectively. The composition layer In our sequential inference model, we keep using BiLSTM to compose local inference information sequentially. The formulas for BiLSTM are similar to those in Equations (1) and (2) in their forms so we skip the details, but the aim is very different here—they are used to capture local inference information ma and mb and their context here for inference composition.
In the tree composition, the high-level formulas of how a tree node is updated to compose local inference is as follows:
va,t = TrLSTM(F (ma,t),hLt−1,h R t−1), (16) vb,t = TrLSTM(F (mb,t),hLt−1,h R t−1). (17)
We propose to control model complexity in this layer, since the concatenation we described above to compute ma and mb can significantly increase the overall parameter size to potentially overfit the models. We propose to use a mapping F as in Equation (16) and (17). More specifically, we use a 1-layer feed-forward neural network with the ReLU activation. This function is also applied to BiLSTM in our sequential inference composition.
Pooling Our inference model converts the resulting vectors obtained above to a fixed-length vector with pooling and feeds it to the final classifier to determine the overall inference relationship.
We consider that summation (Parikh et al., 2016) could be sensitive to the sequence length and hence less robust. We instead suggest the following strategy: compute both average and max pooling, and concatenate all these vectors to form the final fixed length vector v. Our experiments show that this leads to significantly better results than summation. The final fixed length vector v is calculated as follows:
va,ave = `a∑ i=1 va,i `a , va,max = `a max i=1 va,i, (18)
vb,ave = `b∑ j=1 vb,j `b , vb,max = `b max j=1 vb,j , (19)
v = [va,ave;va,max;vb,ave;vb,max]. (20)
Note that for tree composition, Equation (20) is slightly different from that in sequential composition. Our tree composition will concatenate also the hidden states computed for the roots with Equations (16) and (17), which are not shown here.
We then put v into a final multilayer perceptron (MLP) classifier. The MLP has a hidden layer with tanh activation and softmax output layer in our experiments. The entire model (all three components described above) is trained end-to-end. For training, we use multi-class cross-entropy loss.
Overall inference models Our model can be based only on the sequential networks by removing all tree components and we call it Enhanced Sequential Inference Model (ESIM) (see the left part of Figure 1). We will show that ESIM outperforms all previous results. We will also encode parse information with tree LSTMs in multiple layers as described (see the right side of Figure 1). We train
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
this model and incorporate it into ESIM by averaging the predicted probabilities to get the final label for a premise-hypothesis pair. We will show that parsing information complements very well with ESIM and further improves the performance, and we call the final model Hybrid Inference Model (HIM).
4 Experimental Setup
Data The Stanford Natural Language Inference (SNLI) corpus (Bowman et al., 2015) focuses on three basic relationships between a premise and a potential hypothesis: the premise entails the hypothesis (entailment), they contradict each other (contradiction), or they are not related (neutral). The original SNLI corpus contains also “the other” category, which includes the sentence pairs lacking consensus among multiple human annotators. As in the related work, we remove this category. We used the same split as in Bowman et al. (2015) and other previous work.
The parse trees used in this paper are produced by the Stanford PCFG Parser 3.5.3 (Klein and D. Manning, 2003) and they are delivered as part of the SNLI corpus. We use classification accuracy as the evaluation metric, as in related work.
Training We use the development set to select models for testing. To help replicate our results, we publish all our code at [xxx]. Below, we list our training details. We use the Adam method (Kingma and Ba, 2014) for optimization. The first momentum is set to be 0.9 and the second 0.999. The initial learning rate is 0.0004 and the batch size is 32. All hidden states of LSTMs, tree-LSTMs, and word embeddings have 300 dimensions.
We use dropout with a rate of 0.5, which is applied to all feed-forward connections. We use pre-trained 300-D Glove 840B vectors (Pennington et al., 2014) to initialize our word embeddings. Out-of-vocabulary (OOV) words are initialized randomly with Gaussian samples. All vectors including word embedding are updated during training.
5 Results
Overall performance Table 1 shows the results of different models. The first row is a baseline classifier presented by Bowman et al. (2015) that considers handcrafted features such as BLEU score of the hypothesis with respect to the premise, the overlapped words, and the length difference between them, etc.
The next group of models (2)-(7) are based on sentence encoding. The model of Bowman et al. (2016) encodes the premise and hypothesis with two different LSTMs. The model in Vendrov et al. (2015) uses unsupervised ’skip-thoughts’ pre-training in GRU encoders. The approach proposed by Mou et al. (2016) considers tree-based CNN to capture sentence-level semantics, while the model of Bowman et al. (2016) introduces a stack-augmented parser-interpreter neural network (SPINN) which combines parsing and interpretation within a single tree-sequence hybrid model. The work by Liu et al. (2016) uses BiLSTM to generate sentence representations, and then replaces average pooling with intra-attention. The approach proposed by Munkhdalai and Yu (2016a) presents a memory augmented neural network, neural semantic encoders (NSE), to encode sentences.
The next group of methods in the table, models (8)-(15), are inter-sentence attention-based model. The model marked with Rocktäschel et al. (2015) is LSTMs enforcing the so called word-by-word attention. The model of Wang and Jiang (2016) extends this idea to explicitly enforce word-by-word matching between the hypothesis and the premise. Long short-term memory-networks (LSTMN) with deep attention fusion (Cheng et al., 2016) link the current word to previous words stored in memory. Parikh et al. (2016) proposed a decomposable attention model without relying on any word-order information. In general, adding intra-sentence attention yields further improvement, which is not very surprising as it could help align the relevant text spans between premise and hypothesis. The model of Munkhdalai and Yu (2016b) extends the framework of Wang and Jiang (2016) to a full n-ary tree model and achieves further improvement. Sha et al. (2016) proposes a special LSTM variant which considers the attention vector of another sentence as an inner state of LSTM. Paria et al. (2016) use a neural architecture with a complete binary tree-LSTM encoders without syntactic information.
The table shows that our ESIM model achieves an accuracy of 88.0%, which has already outperformed all the previous models, including those using much more complicated network architectures (Munkhdalai and Yu, 2016b).
We ensemble our ESIM model with syntactic tree-LSTMs (Zhu et al., 2015) based on syntactic parse trees and achieve significant improvement over our best sequential encoding model ESIM, at-
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Model #Para. Train Test
(1) Handcrafted features (Bowman et al., 2015) - 99.7 78.2
(2) 300D LSTM encoders (Bowman et al., 2016) 3.0M 83.9 80.6 (3) 1024D pretrained GRU encoders (Vendrov et al., 2015) 15M 98.8 81.4 (4) 300D tree-based CNN encoders (Mou et al., 2016) 3.5M 83.3 82.1 (5) 300D SPINN-PI encoders (Bowman et al., 2016) 3.7M 89.2 83.2 (6) 600D BiLSTM intra-attention encoders (Liu et al., 2016) 2.8M 84.5 84.2 (7) 300D NSE encoders (Munkhdalai and Yu, 2016a) 3.0M 86.2 84.6
(8) 100D LSTM with attention (Rocktäschel et al., 2015) 250K 85.3 83.5 (9) 300D mLSTM (Wang and Jiang, 2016) 1.9M 92.0 86.1 (10) 450D LSTMN with deep attention fusion (Cheng et al., 2016) 3.4M 88.5 86.3 (11) 200D decomposable attention model (Parikh et al., 2016) 380K 89.5 86.3 (12) Intra-sentence attention + (11) (Parikh et al., 2016) 580K 90.5 86.8 (13) 300D NTI-SLSTM-LSTM (Munkhdalai and Yu, 2016b) 3.2M 88.5 87.3 (14) 300D re-read LSTM (Sha et al., 2016) 2.0M 90.7 87.5 (15) 300D btree-LSTM encoders (Paria et al., 2016) 2.0M 88.6 87.6
(16) 600D ESIM 4.3M 92.6 88.0 (17) HIM (600D ESIM + 300D Syntactic tree-LSTM) 7.7M 93.5 88.6
Table 1: Accuracies of the models on SNLI. Our final model achieves the accuracy of 88.6%, the best result observed on SNLI, while our enhanced sequential encoding model attains an accuracy of 88.0%, which also outperform the previous models.
taining an accuracy of 88.6%. This shows that syntactic tree-LSTMs complement well with ESIM.
Model #Para. Train Test
(17) HIM (ESIM+syn.tree) 7.7M 93.5 88.6 (18) ESIM+tree 7.7M 91.9 88.2 (16) ESIM 4.3M 92.6 88.0 (19) (16)-ave./max 4.0M 92.9 87.1 (20) (19)-diff./prod. 3.6M 91.6 86.8
Table 2: Ablation performance of our best models. The difference between each pair of the results is statistically significant (t-test, p < 0.01).
Ablation analysis We further analyze the major components that are of importance to help us achieve good performance. From the best model, we first replace the syntactic tree-LSTM with the full tree-LSTM without encoding syntactic parse information. More specifically, two adjacent words in a sentence are merged to form a parent node, and this process continues and results in a full binary tree, where padding nodes are inserted when there are no enough leaves to form a full tree. Each tree node is implemented with a tree-LSTM block (Zhu et al., 2015) same as in model (17). Table 2 shows that with this replacement, the performance drops to 88.2%.
Furthermore, we note the importance of the layer
performing the enhancement for local inference information in Section 3.2 and the pooling layer in inference composition in Section 3.3. Table 2 suggests that the NLI task seems very sensitive to the layers. If we remove the pooling layer in inference composition and replace it with summation as in Parikh et al. (2016), the accuracy drops to 87.1%. We further remove the difference and element-wise product from the local inference enhancement layer, the accuracy drops to 86.8%.
The difference between any pairs of the models in Table 2 is statistically significant (t-test, p < 0.01). Note that in Table 1 above we do not have the output from other systems to perform a significance test, but the 0.2% difference is statistically significant among our systems. To provide some detailed comparison with Parikh et al. (2016), replacing bidirectional LSTMs in inference composition and also input encoding with MLP reduces the accuracy to 86.1% and 84.0% respectively.
Further analysis We showed that encoding syntactic parsing information helps recognize natural language inference—it additionally improves the strong system. Figure 3 shows an example where tree-LSTM makes a different and correct decision. In subfigure (c), the larger values at the input gates on nodes 9 and 10 indicate that those nodes are important in making the final decision. We observe
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
(a) Binarized constituency tree of premise
(b) Binarized constituency tree of hypothesis
(c) Input gate of tree-LSTM in inference composition (l2-norm)
(d) Input gate of BiLSTM in inference composition (l2-norm)
(e) Normalized attention weights of tree-LSTM
(f) Normalized attention weights of BiLSTM
Figure 3: An example for analysis. Subfigures (a) and (b) are the constituency parse trees of the premise and hypothesis, respectively. ’-’ means a non-leaf or a null node. Subfigures (c) and (d) are input gates’ l2-norm of tree-LSTM and BiLSTM in inference composition, respectively. Subfigures (e) and (f) are attention visualization of the tree model and ESIM, respectively. The darker the color, the greater the value. The premise is on the x-axis and the hypothesis is on y-axis.
that in subfigure (e), nodes 9 and 10 are aligned to node 29 in the premise. Such information helps the system decide that this pair is a contradiction. Accordingly, in the sequential BiLSTM, the words sitting and down do not play an important role for making the final decision. Subfigure (f) shows that sitting is equally aligned with reading and standing and the alignment for word down is not that useful.
6 Conclusion and Future Work
We propose neural network models for natural language inference, which achieve the best results reported on the SNLI benchmark. The results are first achieved through our enhanced sequential inference model, which outperformed the previous models, including those employing more complicated network architectures, suggesting that the potential of sequential inference models have not been fully exploited yet. Our model may serve as a new baseline or starting point for deploying
more complicated architectures for NLI. Based on this, we further show that by explicitly considering recursive architectures in both local inference modeling and inference composition, we achieve additional improvement. Particularly, incorporating syntactic parsing information contributes to our best result—it further improves the performance even when added to the already very strong model.
Future work interesting to us includes exploring the usefulness of knowledge resources to help alleviate data sparseness issues. We are also interested in studying more the fragments of sentences or parses highlighted by the attention mechanism in order to provide human-readable explanations of the decisions.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

This paper presents a purpose-built neural network architecture for textual entailment NLI based on a three step process of encoding, attention-based matching, and aggregation. The model has two variants, one based on TreeRNNs and the other based on sequential BiLSTMs. The sequential model outperforms all published results, and an ensemble with the tree model does better still.  The paper is clear, the model is well motivated, and the results are impressive. Everything in the paper is solidly incremental, but I nonetheless recommend acceptance.   Major issues that I'd like discussed in the response:   You suggest several times that your system can serve as a new baseline for future work on NLI. This isn't an especially helpful or meaningful claim it could be said of just about any model for any task. You could argue that your model is unusually simple or elegant, but I don't think that's really a major selling point of the model.   Your model architecture is symmetric in some ways that seem like overkill you compute attention across sentences in both directions, and run a separate inference composition (aggregation) network for each direction. This presumably nearly doubles the run time of your model. Is this really necessary for the very asymmetric task of NLI? Have you done ablation studies on this?**   You present results for the full sequential model (ESIM) and the ensemble of that model and the tree-based model (HIM). Why don't you present results for the tree-based model on its own?**  Minor issues:   I don't think the Barker and Jacobson quote means quite what you want it to mean. In context, it's making a specific and not-settled point about *direct* compositionality in formal grammar. You'd probably be better off with a more general claim about the widely accepted principle of compositionality.   The vector difference feature that you use (which has also appeared in prior work) is a bit odd, since it gives the model redundant parameters. Any model that takes vectors a, b, and (a - b) as input to some matrix multiplication is exactly equivalent to some other model that takes in just a and b and has a different matrix parameter. There may be learning-related reasons why using this feature still makes sense, but it's worth commenting on.   How do you implement the tree-structured components of your model? Are there major issues with speed or scalability there?  u001f  Typo: (Klein and D. Manning, 2003)    Figure 3: Standard tree-drawing packages like (tikz-)qtree produce much more readable parse trees without crossing lines. I'd suggest using them.  ---  Thanks for the response! I still solidly support publication. This work is not groundbreaking, but it's novel in places, and the results are surprising enough to bring some value to the conference.


Connecting the dots: Summarizing and Structuring Large Document Collections Using Concept Maps
Concept maps can be used to concisely represent important information and bring structure into large document collections. Therefore, we study a variant of multidocument summarization that produces summaries in the form of concept maps. However, suitable evaluation datasets for this task are currently missing. To close this gap, we present a newly created corpus of concept maps that summarize heterogeneous collections of web documents on educational topics. It was created using a novel crowdsourcing approach that allows us to efficiently determine important elements in large document collections. We release the corpus along with a baseline system and proposed evaluation protocol to enable further research on this variant of summarization.
1 Introduction
Multi-document summarization (MDS), the transformation of a set of documents into a short text containing their most important aspects, is a longstudied problem in NLP. Generated summaries have been shown to support humans dealing with large document collections in information seeking tasks (McKeown et al., 2005; Maña-López et al., 2004; Roussinov and Chen, 2001). However, when exploring a set of documents manually, humans rarely write a fully-formulated summary for themselves. Instead, user studies (Chin et al., 2009; Kang et al., 2011) show that they note down important keywords and phrases, try to identify relationships between them and organize them accordingly. Therefore, we believe that the study of summarization with similarly structured outputs is an important extension of the traditional task.
ACL 2017
long papers
substantial research invites submissions of must describe
Concept Relation Proposition
Figure 1: Elements of a concept map.
A representation more in line with observed user behavior is a concept map (Novak and Gowin, 1984). Concept maps are labeled graphs showing concepts as nodes and relationships between them as edges (Figure 1). Labels can be freely defined. A concept can be an entity, abstract idea, event or activity, designated by a unique label. Good maps should be propositionally coherent, meaning that every relation together with the two connected concepts form a meaningful proposition.
Introduced in 1972 as a teaching tool (Novak and Cañas, 2007), applications in education (Edwards and Fraser, 1983; Roy, 2008), for writing assistance (Villalon, 2012) or to structure information repositories (Briggs et al., 2004; Richardson and Fox, 2005) have been reported. For summarization, concept maps allow to represent a summary concisely and clearly reveal relations. Moreover, we see a second interesting use case beyond the capabilities of textual summaries: When concepts and relations are linked to corresponding locations in the documents, the graph can be used to navigate in a document collection, similar to how a table-of-contents is used in a single document.
The corresponding task that we propose is concept-map-based MDS: Given a set of related documents, create a concept map that represents its most important content, satisfies a specified size limit and is connected. Further, to ease the use of automatic evaluations, we focus on the extractive variant of the task in this work, requiring that all labels have to be taken from the documents.
The proposed task is complex, consisting of several interdependent subtasks. One has to ex-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
students
student loans
cost of their college education
a credit-worthy cosigner sufficient credit history a steady income
private lending institutions
federal student loans
credit check U.S. Department
of Schooling
are offered to
do not have
require a borrower to have
regularly offer
pay for
help students cover
will need to apply with
not requiring
provide a significant financial aid to
offered by
Figure 2: Excerpt from a summary concept map on the topic “students loans without credit history”.
tract appropriate labels for concepts and relations and recognize different expressions that refer to the same concept across multiple documents. Further, one has to select the most important concepts and relations for the summary and finally organize them in a graph satisfying the connectedness and size constraints. In order to develop and evaluate methods for the task, gold-standard corpora are necessary, but no suitable corpus is available. The manual creation of such a dataset is very timeconsuming, as annotators need to perform all subtasks described above. In particular, an annotator would need to manually identify all potential concepts in the documents, while only a few of them will eventually end up in the summary.
To overcome these issues, we developed a corpus creation method that effectively combines automatic preprocessing, scalable crowdsourcing and high-quality expert annotations. Using it, we can avoid the high effort for single annotators, allowing us to scale to document clusters that are 15 times larger than in traditional summarization corpora. In addition, the method includes a novel crowdsourcing scheme, low-context importance annotation, that resolves quality issues observed in previous work (Lloret et al., 2013).
For concept-map-based MDS, we created a new corpus of 30 topics, each with around 40 source documents and a summarizing concept map that is the consensus of many crowdworkers. The document clusters were built from a web crawl, comprising a variety of different genres, text styles and document lengths. Figure 2 shows one example.
To summarize, we make the following contributions: (1) We propose a novel summarization task, concept-map-based MDS, (2) present a new crowdsourcing scheme to create reference summaries, (3) publish a new dataset for the proposed task and (4) provide an evaluation protocol and baseline. These resources are publicly available
under a permissive license. 1
2 Related Work
Some attempts have been made to automatically construct concept maps from text, working with either single documents (Zubrinic et al., 2015; Villalon, 2012; Valerio and Leake, 2006; Kowata et al., 2010) or document clusters (Qasim et al., 2013; Zouaq and Nkambou, 2009; Rajaraman and Tan, 2002). These approaches extract concept and relation labels from syntactic structures and connect them to build a concept map. However, common task definitions and comparable evaluations are missing. In addition, only a few of them, namely Villalon (2012) and Valerio and Leake (2006), define summarization as their goal and try to compress the input to a substantially smaller size. Our newly proposed task and the created large-cluster dataset fill these gaps as they emphasize the summarization aspect of the task.
For the subtask of selecting summary-worthy concepts and relations, techniques developed for traditional summarization (Nenkova and McKeown, 2011) and keyphrase extraction (Hasan and Ng, 2014) are related and applicable. Approaches that build graphs of propositions to create a summary (Fang et al., 2016; Li et al., 2016; Liu et al., 2015; Li, 2015) seem to be particularly related, however, there is one important difference: While they use graphs as an intermediate representation from which a textual summary is then generated, the goal of the proposed task is to create a graph that is directly interpretable and useful for a user.
For traditional summarization, the most wellknown datasets emerged out of the DUC and TAC competitions.2 They provide clusters of news articles with gold-standard summaries that serve as a benchmark to compare different approaches.
1URL hidden, anon. resources are attached for review. 2duc.nist.gov, tac.nist.gov
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Imagine you want to learn something about students loans without credit history. How useful would the following statements be for you?
(P1) students with bad credit history - apply for - federal loans with the FAFSA 2 Extremely Important 2 Very Important 2 Moderately Important 2 Slightly Important 2 Not at all Important
(P2) students - encounter - unforeseen financial emergencies 2 Extremely Important 2 Very Important 2 Moderately Important 2 Slightly Important 2 Not at all Important
Figure 3: Likert-scale crowdsourcing task with topic description and two example propositions.
Extending these efforts, several more specialized corpora have been created: With regard to size, Nakano et al. (2010) present a corpus of summaries for large-scale collections of web pages. Recently, corpora with more heterogeneous documents have been suggested, e.g. (Zopf et al., 2016) and (Benikova et al., 2016). The corpus we present combines these aspects, as it has large clusters of heterogeneous documents, and provides a necessary benchmark to evaluate the proposed task. For concept map generation, one corpus with humancreated summary concept maps for student essays has been created (Villalon et al., 2010). In contrast to our corpus, it only deals with single documents, requires a two orders of magnitude smaller amount of compression and is not publicly available .
Other types of information representation that also model concepts and their relationships are knowledge bases, such as Freebase (Bollacker et al., 2009), and ontologies. However, they both differ in important aspects: Whereas concept maps follow an open label paradigm and are meant to be interpretable by humans, knowledge bases and ontologies are usually more strictly typed and made to be machine-readable. Moreover, approaches to automatically construct them from text typically try to extract as much information as possible, while we want to summarize a document. As a result, approaches from these fields cannot be directly applied. Vice versa, Zouaq et al. (2011) showed that concept maps can be a useful first step to create a domain ontology.
3 Low-Context Importance Annotation
Lloret et al. (2013) describe several experiments to crowdsource reference summaries. Workers are asked to read 10 documents and then select 10 summary sentences from them for a reward of $0.05. They discovered several challenges, including poor work quality and the subjectiveness of the annotation task, indicating that crowdsourcing is not useful for this purpose. To overcome these is-
sues, we introduce a new task design, low-context importance annotation, to determine summaryworthy parts of documents. Compared to Lloret et al.’s approach, it is more in line with crowdsourcing best practices, as the tasks are simple, intuitive and small (Sabou et al., 2014) and workers receive reasonable payment (Fort et al., 2011).
3.1 Task Design
We break the importance annotation down to single propositions. The goal of our crowdsourcing scheme is to obtain a score for each proposition indicating its importance in a document cluster, such that a ranking according to the score would reveal what is most important and should be included in a summary. In contrast to other work, we do not show the documents to the workers at all, but provide only a description of the document cluster’s topic. This ensures that tasks are small, simple and can be done quickly (see Figure 3). In preliminary tests, we found that this design, despite the minimal context, works reasonably well as long as the topic is something the worker can relate to. For instance, consider Figure 3: One can easily say that P1 is more important than P2 without reading the documents. We distinguish two task variants:
Likert-scale Tasks Instead of enforcing binary importance decisions, we use a 5-point Likertscale to allow more fine-grained annotations. The obtained labels are translated into scores (5..1) and the average of all scores for a proposition is used as an estimate for its importance. This follows the idea that while single workers might find the task subjective, the consensus of multiple workers, represented in the average score, tends to be less subjective due to the “wisdom of the crowd”. We randomly group five propositions into a task.
Comparison Tasks As an alternative, we use a second task design based on pairwise comparisons. Comparisons are known to be easier to make and more consistent (Belz and Kow, 2010),
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
but also more expensive, as the number of pairs grows quadratically with the number of objects.3 To reduce the cost, we group five propositions into a task and ask workers to order them by importance per drag-and-drop. From the results, we derive pairwise comparisons and use TrueSkill (Herbrich et al., 2007), a powerful Bayesian rank induction model (Zhang et al., 2016), to obtain importance estimates for each proposition.
3.2 Pilot Study
To verify the proposed approach, we conducted a pilot study on Amazon Mechanical Turk using data from TAC2008 (Dang and Owczarzak, 2008). We collected importance estimates for 474 propositions extracted from the first three clusters4 using both task designs. Each Likert-scale task was assigned to 5 different workers and awarded $0.06. For comparison tasks, we also collected 5 labels each, paid $0.05 and sampled around 7% of all possible pairs. We submitted them in batches of 100 pairs and selected pairs for subsequent batches based on the confidence of the TrueSkill model.
Quality Control Following the observations of Lloret et al. (2013), we established several measures for quality control. First, we restricted our tasks to workers from the US with an approval rate of at least 95%. Second, we identified low quality workers by measuring the correlation of each worker’s Likert-scores with the average of the other four scores. The worst workers (at most 5% of all labels) were removed. In addition, we included trap sentences, similar as in (Lloret et al., 2013), in around 80 of the tasks. In contrast to Lloret et al.’s findings, both an obvious trap sentence (This sentence is not important) and a less obvious but unimportant one (Barack Obama graduated from Harvard Law) were consistently labeled as unimportant (1.08 and 1.14), indicating that the workers paid attention to the task.
Agreement and Reliability For Likert-scale tasks, we follow Snow et al. (2008) and calculate agreement as the average Pearson correlation of a worker’s Likert-score with the average score of the remaining workers.5 This measure is less
3Even with intelligent sampling strategies, such as the active learning in CrowdBT (Chen et al., 2013), the number of pairs is only reduced by a constant factor (Zhang et al., 2016).
4D0801A-A, D0802A-A, D0803A-A 5As workers are not consistent across all items, we create
five meta-workers by sorting the labels per proposition.
strict than exact label agreement and can account for close labels and high- or low-scoring workers. We observe a correlation of 0.81, indicating substantial agreement. For comparisons, the majority agreement is 0.73. To further examine the reliability of the collected data, we followed the approach of Kiritchenko and Mohammed (2016) and simply repeated the crowdsourcing for one of the three topics. Between the importance estimates calculated from the first and second run, we found a Pearson correlation of 0.82 (Spearman 0.78) for Likert-scale tasks and 0.69 (Spearman 0.66) for comparison tasks. This shows that the approach, despite the high subjectiveness of the task, allows us to collect reliable annotations.
Peer Evaluation In addition to the reliability studies, we extrinsically evaluated the annotations in the task of summary evaluation. For each of the 58 peer summaries in TAC2008, we calculated a score as the sum of the importance estimates of the propositions it contains. Table 1 shows how these peer scores, averaged over the three topics, correlate with the manual responsiveness scores assigned during TAC in comparison to ROUGE-2 and Pyramid scores.6 The results demonstrate that with both task designs, we obtain importance annotations that are similarly useful for summary evaluation as pyramid annotations or gold-standard summaries (used for ROUGE).
Conclusion Based on the pilot study results, we conclude that the proposed crowdsourcing scheme allows us to obtain proper annotations for the importance of propositions. Thus, the approach can be used to construct gold-standard summaries, evaluate systems or generate training data.
4 Corpus Creation
To carry out comparable evaluations, the proposed task requires a gold-standard corpus. We
6Correlations for ROUGE and Pyramid are lower than reported in TAC since we only use 3 topics instead of all 48.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Proposition Extraction
Proposition Filtering
Importance Annotation
Proposition Revision
Concept Map Construction
§4.2 §4.3 §4.4 §4.5 §4.6
Figure 4: Steps of the corpus creation (with references to the corresponding sections).
present our corpus construction process, outlined in Figure 4, that combines automatic preprocessing, scalable crowdsourcing and high-quality expert annotations to efficiently create such a dataset.
4.1 Source Data
As a starting point, we used the DIP corpus (Habernal et al., 2016), a collection of 49 clusters of 100 web pages on educational topics (e.g. bullying, homeschooling, drugs). It was created from a large web crawl using state-of-the-art information retrieval. Web pages are boilerplate-cleaned, segmented and all sentences have binary labels for topic relevance. For our corpus, we selected 30 of the topics. From the list of 100 ranked web pages per topic, we choose the top-ranked ones that were from distinct websites and had at least one relevant sentence, stopping at 50 documents.
4.2 Proposition Extraction
As concept maps consist of propositions expressing the relation between concepts (see Figure 1), we need to impose such a structure upon the plain text in the document clusters. This could be done by manually annotating spans representing concepts and relations in every sentence, however, the size of our clusters makes this a huge effort: Even if we restrict the annotation to sentences known to be relevant, 2288 sentences per topic (69k in total) need to be processed. Therefore, we resort to an automatic approach for this step.
The Open Information Extraction paradigm (Banko et al., 2007) offers a representation very similar to the desired one. For instance, from
Specifically, eligible students may borrow the Direct Stafford Loans and eligible parents may borrow the Direct PLUS Loan on behalf of their dependent students.
Open IE systems extract tuples of two arguments and a relation phrase representing propositions:
(eligible students, may borrow, Dir. Stafford Loan) (eligible parents, may borrow, Direct PLUS Loan)
While the relation phrase is similar to a relation
in a concept map, many arguments in these tuples represent useful concepts. We used Open IE 47, a state-of-the-art Open IE system (Stanovsky and Dagan, 2016), to process the relevant sentences of all topics. After removing duplicates, we obtained 4137 tuples per topic.
Since we want to create a gold-standard corpus, we have to ensure that we produce high-quality data. We therefore made use of the confidence assigned to every extracted tuple to filter out low quality ones. To ensure that we do not filter too aggressively (and miss important aspects in the final summary), we manually annotated 500 tuples sampled from all topics for correctness. On the first 250 of them, we tuned the filter threshold to 0.5, which keeps 98.7% of the correct extractions in the unseen second half. After filtering, a topic had on average 2850 propositions (85k in total).
4.3 Proposition Filtering
Despite the similarity of the Open IE paradigm, not every extracted tuple is a suitable proposition for a concept map. To reduce the effort in the subsequent steps, we therefore want to filter out unsuitable ones. A tuple is suitable if it (1) is a correct extraction, (2) is meaningful without any context and (3) has arguments that represent proper concepts. To better understand these criteria, we performed a small annotation study.
Based on a set of examples, we created a guideline explaining when to label a tuple as suitable for a concept map. We found tuples to be unsuitable mostly because they had unresolvable pronouns, conflicting with (2), or arguments that were full clauses or propositions, conflicting with (3), while (1) was mostly taken care of by the confidence filtering in §4.2. Three annotators independently labeled a set of 500 randomly sampled tuples. We observed an agreement of 82% (κ = 0.60).
Due to the high number of tuples we decided to automate the filtering step. We trained a linear SVM on the majority voted annotations. As features, we used the extraction confidence, length of arguments and relations as well as part-of-speech
7github.com/knowitall/openie
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
tags, among others. To ensure that the automatic classification does not remove suitable propositions, we tuned the classifier to avoid false negatives. In particular, we introduced class weights, improving precision on the negative class at the cost of a higher fraction of positive classifications. Further, we assumed that we can manually verify a certain number of the most uncertain negative classifications. Under these constraints, a model trained on 350 instances achieves 93% precision on negative classifications on the unseen 150 instances, assuming that 20% are manually verified. We found this to be a reasonable trade-off of automation and data quality and applied the model to the full dataset.
The classifier filtered out 43% of the propositions, leaving 1622 per topic. We manually examined the 17k least confident negative classifications and corrected 955 of them. We also corrected positive classifications for certain types of tuples for which we knew the classifier to be imprecise. Finally, each topic was left with an average of 1554 propositions (47k in total).
4.4 Importance Annotation
Given the propositions identified in the previous step, we now applied our crowdsourcing scheme as described in §3 to determine their importance. To cope with the large number of propositions, we combine the two task designs: First, we collect 5 Likert-scores for each proposition, clean the data and calculate averages. Then, using only the top 100 propositions8, we crowdsource 10% of the pairwise comparisons among them. Using TrueSkill, we obtain a fine-grained ranking of the 100 most important propositions. We use the topic descriptions from the underlying DIP corpus.
For Likert-scores, the average agreement over all topics is 0.80, while the majority agreement for comparisons is 0.78. We repeated the data collection for three randomly selected topics and found the Pearson correlation between both runs to be 0.73 (Spearman 0.73) for Likert-scores and 0.72 (Spearman 0.71) for comparisons. These figures show that the crowdsourcing approach works on this dataset as reliably as on the TAC documents.
In total, we uploaded 53k scoring and 12k comparison tasks to Mechanical Turk, spending $4425.45 including fees. From the fine-grained
8We also add all propositions with the same score as the 100th, yielding 112 propositions on average.
ranking of the 100 most important propositions, we select the top 50 per topic to construct a summary concept map in the subsequent steps.
4.5 Proposition Revision
Having a manageable number of propositions per topic, an expert annotator then applied a few straightforward transformations that correct common errors of the Open IE system. First, we break down propositions with conjunctions in either of the arguments into separate propositions per conjunct, which the Open IE system sometimes fails to do. And second, we correct span errors that might occur in the arguments or relation phrase, especially when sentences were not properly segmented. As a result, we have a set of high quality propositions for our concept map, consisting of, due to the first transformation, 56.1 propositions per topic on average.
4.6 Concept Map Construction
In this final step, we connect the set of important propositions to form a graph. For instance, given the following two propositions
(student, may borrow, Stafford Loan) (the student, does not have, a credit history)
one can easily see, although the first arguments differ slightly, that both labels describe the concept student, allowing us to build a concept map with three concepts. The annotation task thus involves deciding which of the available propositions to include in the map, which of their concepts to join and which of the available labels to use. As these decisions highly depend upon each other and require context, we decided to use expert annotators rather than crowdsource the subtasks.
Annotators were given the topic description and the most important, ranked propositions. They could connect them step by step in a simple annotation tool that visualized the map constructed so far. They were instructed to reach a size of 25 concepts, the recommended maximum size for a concept map (Novak and Cañas, 2007). Further, they should prefer more important propositions and ensure connectedness. When connecting two propositions, they were asked to keep the concept label that was appropriate for both propositions. To support the annotator, the tool used ADW (Pilehvar et al., 2013), a state-of-the-art approach for semantic similarity, to suggest possible connections. If
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Corpus Cluster Cluster Size Docs Doc. Size Rel. Std. This work 30 97,880 ± 50,086.2 40.5 ± 6.8 2,412.8 ± 3,764.1 1.56 DUC 2006 50 17,461 ± 6,627.8 25.0 ± 0.0 729.2 ± 542.3 0.74 DUC 2004 50 6,721 ± 3,017.9 10.0 ± 0.0 672.1 ± 506.3 0.75 TAC 2008A 48 5,892 ± 2,832.4 10.0 ± 0.0 589.2 ± 480.3 0.82
Table 2: Topic clusters in comparison to classic corpora (size in token, mean with standard deviation).
an annotator was not able to connect 25 concepts, she was allowed to create up to three synthetic relations with freely defined labels, making the maps slightly abstractive. On average, the constructed maps have 0.77 synthetic relations, mostly connecting concepts whose relation is too obvious to be explicitly stated in text (e.g. between Montessori teacher and Montessori education).
To assess the reliability of this annotation step, we had the first three maps created by two annotators. We casted the task of selecting propositions to be included in the map as a binary decision task and observed an agreement of 84% (κ = 0.66). Second, we modeled the decision which concepts to join as a binary decision on all pairs of common concepts, observing an agreement of 95% (κ = 0.70). And finally, we compared which concept labels the annotators decided to include in the final map, observing 85% (κ = 0.69) agreement. Hence, the annotation shows substantial agreement (Landis and Koch, 1977).
5 Corpus Analysis
In this section, we describe our newly created corpus, which, in addition to having summaries in the form of concept maps, differs from traditional summarization corpora in several aspects.
5.1 Document Clusters
Size The corpus consists of document clusters for 30 different topics. Each of them contains around 40 documents with on average 2413 tokens, which leads to an average cluster size of 97,880 token. With these characteristics, the document clusters are 15 times larger than typical DUC clusters of ten documents and five times larger than the 25-document-clusters (Table 2). In addition, the documents are also more variable in terms of length, as the (length-adjusted) standard deviation is twice as high as in the other corpora.
Genres Because we used a large web crawl as the source for our corpus, it contains docu-
per Map Token Character Concepts 25.0± 0.0 3.2± 0.5 22.0± 4.1 Relations 25.2± 1.3 3.2± 0.5 17.1± 2.6
Table 3: Size of concept maps (mean with std).
ments from a variety of genres. To further analyze this property, we categorized a sample of 50 documents from the corpus. Among them, we found professionally written articles and blog posts (28%), educational material for parents and kids (26%), personal blog posts (16%), forum discussions and comments (12%), commented link collections (12%) and scientific articles (6%). Systems summarizing the corpus can therefore not rely on genre-specific characteristics and have to deal with the full variety of text types.
Textual Heterogeneity In addition to the variety of genres, the documents also differ in terms of language use. To capture this property, we follow Zopf et al. (2016) and compute, for every topic, the average Jensen-Shannon divergence between the word distribution of one document and the word distribution in the remaining documents. The higher this value is, the more the language differs between documents. We found the average divergence over all topics to be 0.3490, whereas it is 0.3019 in DUC 2004 and 0.3188 in TAC 2008A.
5.2 Concept Maps
As Table 3 shows, each of the 30 reference concept maps has exactly 25 concepts and between 24 and 28 relations. Labels for both concepts and relations consist on average of 3.2 tokens, whereas the latter are a bit shorter in characters.
To obtain a better picture of what kind of text spans have been used as labels, we automatically tagged them with their part-of-speech and determined their head with a dependency parser. Concept labels tend to be headed by nouns (82%) or verbs (15%), while they also contain adjectives, prepositions and determiners. Relation labels, on
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
the other hand, are almost always headed by a verb (94%) and contain prepositions, nouns and particles in addition. These distributions are very similar to those reported by Villalon et al. (2010) for their (single-document) concept map corpus.
Analyzing the graph structure of the maps, we found that all of them are connected. They have on average 7.2 central concepts with more than one relation, while the remaining ones occur in only one proposition. During the annotation, we found that achieving a higher number of connections would mean compromising importance, i.e. including less important propositions, and we decided against it. For experiments, we provide a split into 15 topics each for training and testing.
6 Baseline Experiments
In this section, we briefly describe a baseline and evaluation scripts that we release, with a detailed documentation, along with the corpus.
Baseline Method We implemented a simple approach inspired by previous work on concept map generation and keyphrase extraction. For a document cluster, it performs the following steps:
1. Extract all NPs as potential concepts.
2. Merge potential concepts whose labels match after stemming into a single concept.
3. For each pair of concepts co-occurring in a sentence, select the tokens in between as a potential relation if they contain a verb.
4. If a pair of concepts has more than one relation, select the one with the shortest label.
5. Rank all concepts by importance.
6. Connect concepts to a graph of 25 concepts.
For (5), we trained a binary classifier to identify the important concepts in the set of all potential concepts. We used common features for keyphrase extraction, including position, frequency and length, and Weka’s Random Forest (Hall et al., 2009). At inference time, the classifier provides an importance score for each concept. In (6), we start with the full graph of all extracted concepts and relations and use a heuristic to find a subgraph that is connected, satisfies the size limit of 25 concepts and has many high-scoring concepts: We iteratively remove the weakest concept until only one connected component of 25 concepts or less remains, which is the final concept map.
Evaluation Metrics In order to automatically compare generated concept maps with reference maps, we propose three metrics.9 As a concept map is fully defined by the set of its propositions, we can compute precision, recall and F1scores between the two sets, with propositions represented as the concatenation of concept and relation labels. Strict Match compares them after stemming and only counts complete matches. Using METEOR (Denkowski and Lavie, 2014), we offer a second metric that takes synonyms and paraphrases into account and also scores partial matches. And finally, we compute ROUGE-2 (Lin, 2004) between the concatenation of all propositions from the maps. These automatic measures might be complemented with a human evaluation.
Results Table 4 shows the performance of the baseline. An analysis of the single pipeline steps revealed major bottlenecks of the method and challenges of the task. First, we observed that around 76% of gold concepts are covered with the extraction (step 1+2), while the top 25 concepts (step 5) only contain 17%. Hence, content selection is a major challenge, stemming from the large cluster sizes. Second, while also 17% of gold concepts are contained in the final maps (step 6), scores for strict proposition matching are low, indicating a poor performance of the relation extraction (step 3). The propagation of these errors along the pipeline contributes to overall low scores.
7 Conclusion
In this work, we presented low-context importance annotation, a novel crowdsourcing scheme that we used to create a new benchmark corpus for concept-map-based MDS. The corpus has largescale document clusters of heterogeneous web documents, posing a challenging summarization task. Together with the corpus, we provide implementations of a baseline method and evaluation scripts and hope that our efforts facilitate future research on this variant of summarization.
9For exact definitions, see the published scripts and docs.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths:  Detailed guidelines and explicit illustrations.  - Weaknesses:  The document-independent crowdsourcing annotation is unreliable.   - General Discussion:  This work creates a new benchmark corpus for concept-map-based MDS. It is well organized and written clearly. The supplement materials are sufficient. I have two questions here. 1)              Is it necessary to treat concept map extraction as a separate task? On the one hand, many generic summarization systems build a similar knowledge graph and then generate summaries accordingly. On the other hand, with the increase of the node number, the concept map becomes growing hard to distinguish. Thus, the general summaries should be more readable. 2)              How can you determine the importance of a concept independent of the documents? The definition of summarization is to reserve the main concepts of documents. Therefore, the importance of a concept highly depends on the documents. For example, in the given topic of coal mining accidents, assume there are two concepts: A) an instance of coal mining accidents and B) a cause of coal mining accidents. Then, if the document describes a series of coal mining accidents, A is more important than B. In comparison, if the document explores why coal mining accidents happen, B is more significant than A. Therefore, just given the topic and two concepts A&B, it is impossible to judge their relative importance.  I appreciate the great effort spent by authors to build this dataset. However, this dataset is more like a knowledge graph based on common sense rather than summary.