Event Factuality Identification via Deep Neural Networks
Event factuality identification plays an important role in deep NLP applications. In this paper, we propose a deep learning framework for this task which first extracts essential information from raw texts as the inputs and then identifies the factuality of events via a deep neural network with a proper combination of Bidirectional Long Short-Term Memory (BiLSTM) neural network and Convolutional Neural Network (CNN). The experimental results on FactBank show that our framework significantly outperforms several state-of-the-art baselines.
1 Introduction
Event factuality is the information expressing the commitment of relevant sources towards the factual nature of events. That is, event factuality conveys whether an event is characterized as a fact, a certain possibility, or an impossible situation, and thus is helpful for various NLP applications, e.g., opinion detection (Wiebe et al., 2005), QA (Pustejovsky et al., 2005), textual entailment (de Marneffe et al., 2006; Hickl and Bensley, 2007), rumor identification (Qazvinian et al., 2011).
In principle, the value of event factuality is related to some elements, e.g., predicates, speculative and negative cues. Consider following sentences as examples1:
(S1) McCulley, a famous economist, doubts that the tax rate will increase soon.
(S2) He knows they are not able to go to the village due to the flood.
In sentence S1, the event increase is a possibility according to the predicate doubts, while in
1In this paper, events are in bold and sources are underlined in example sentences.
S2, the event go is regarded as an impossible situation due to the negation word not. From these sentences, we can learn that predicates and cues can determine the event factuality to a great degree. Such event factuality provides the way to distinguish fact events from speculative and negative ones.
Traditional methods for event factuality identification usually employed either hand-crafted rules (Saurı́, 2008; Saurı́ and Pustejovsky, 2012) relying on expert knowledge, or features (Prabhakaran et al., 2010; de Marneffe et al., 2012) relying on various kinds of annotated information, such as source introducing predicates, predicate classes, speculative and negative cues. Due to the recent success of deep learning on various NLP tasks in learning useful representations from ordinary sentences (Socher et al., 2012; Zeng et al., 2014; Cheng et al., 2016) and syntactic paths (Xu et al., 2015a,b; Roth and Lapata, 2016) and the attention mechanism on capturing the focus change in sequence modeling (Chen et al., 2016; Wang et al., 2016; Zhou et al., 2016), this paper proposes a framework to identify event factuality in raw texts with neural networks. Our main contributions can be summarized as follows:
1) the proposal of a two-step supervised framework for identifying event factuality in raw texts.
2) the utilization of an attention-based CNN to detect source introducing predicates (SIPs), the most important factors to identify event factuality.
3) the proposal of an attention-based deep neural network model with a proper combination of BiLSTM and CNN to identify the factuality of events.
2 Background
This section presents the concepts and basic factors of event factuality.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
187
188
189
190
191
192
193
194
195
196
197
198
199
2.1 Event Factuality
Factuality can be characterized by the combination of two dimensions: epistemic modality and polarity (Saurı́, 2008). While modality conveys the certainty degree of events, such as certain (CT), probable (PR), possible (PS) and underspecified (U), polarity expresses whether the event happened, including positive (+), negative (-) and underspecified (u). Table 1 shows various values of event factuality. For example, CT+ means it is certain that the event happened, while PR- denotes it is probable that the event did not happen. Notice that some values are non-application (NA) grammatically, e.g., PRu, PSu, U+/-.
+ - u CT CT+ CT- CTu PR PR+ PR- (NA) PS PS+ PS- (NA) U (NA) (NA) Uu
Table 1: Various values of event factuality.
2.2 Basic Factors
Considering that FactBank is built on top of TimeBank (Pustejovsky et al., 2003b), we adopt the definition of events proposed by TimeML (Pustejovsky et al., 2003a) from a more grammatical and generalized perspective. We consider the events which can be critical for computing the factuality. Such events can be detected by feature-based methods (Chambers, 2013).
In the literature, three kinds of factors play critical roles in event factuality identification: Source Introducing Predicate (SIP), source and cue.
SIP. Source Introducing Predicates (SIPs) are events that can not only introduce additional sources to assess the factuality of embedded events, but also influence their factuality. For example, the SIP doubts introduces McCulley as a new source in S1, and McCulley evaluates event increase as a possibility according to doubts.
Source. By default, AUTHOR is considered the only source if the sentence contains no SIPs. Further sources can be incorporated by SIPs. In S1, the event increase has two sources: AUTHOR and McCulley. The factuality value is not the inherent property of the event but always relevant to the sources. McCulley commits to the event increase as a possibility, while AUTHOR remains uncommitted. Strictly speaking, we as readers know about McCulley’s perspective only accord-
ing to AUTHOR. Hence, we appeal to the notion of the nested source structure (Wiebe et al., 2005) and represent the sources in the chain form: McCulley AUTHOR, which means McCulley is Embedded in AUTHOR. Only AUTHOR can be displayed by a simple source.
Cues are words that have speculative or negative meanings. The modality of a Non-Uu event is PR/PS if this event is modified by a PR/PS cue, while the polarity of a Non-Uu event is negative if this event is modified by a negative (NEG) cue. So cues are key signals to identify the event factuality. For instance, the factuality of event go in S1 is CT- due to the NEG cue not.
3 Baseline
As a baseline, we present a two-step framework shown in Figure 1 to identify event factuality in raw texts. Firstly, various useful factors, such as SIPs, sources, and cues are extracted from raw texts. Then, a classification algorithm is employed to identify event factuality.
3.1 Event and Cue Detection
For event detection task, we only consider nouns, verbs, adjectives as the candidate tokens to reduce the complexity, and we utilize a maximum entropy classification model with the features used by Chambers (2013).
Øvrelid et al. (2010) and Velldal et al. (2012) concluded that syntactic features are not necessary and lexical sequence-oriented n-gram features can achieve excellent performances on cue detection
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
task. Hence, we employ the lexical features developed by Velldal et al. (2012) to classify each token into PR cue, PS cue, Neg cue, or not cue.
3.2 SIP Detection
Whether an event is a SIP has relation to its semantic information. For instance, verbs state, say, tell are usually used to express opinions which are promoted by certain participants of events, so they can all introduce additional sources. Therefore, they are in the same synonym set {state, say, tell} and share the same hypernym in WordNet2. Hence, we consider token, part-of-speech (POS) and hypernym of the token as lexical level features, and concatenate them into the vector l.
According to the definition of SIPs, if there is no event in the syntactic scope of a word, this word is not a SIP. For example:
(S3) Tom suggests we should have a rest now. In S3, although the event suggests can introduce a new source Tom for the clause “we should have a rest now”, however, this clause expresses a suggestion and is not an event. So the event suggests is NOT a SIP in S3. This example demonstrates the structure of a sentence is also able to determine whether a token is a SIP. Therefore, we also consider the sentence level feature. Instead of original sentences, we propose the following Pruned Sentence structure:
Pruned Sentence (PSen): If a clause of the candidate token contains events, this clause is replaced by the tag 〈event〉; Nouns, pronouns and the current candidate token are unchanged, while other tokens are replaced by the tag 〈O〉.
We argue that PSen can characterize clearly whether there are events in the clause of the token. Besides, in PSen only the candidate token and the tokens that are possible to be new sources are reserved, and effects of other tokens are omitted.
For example, in S4, the current candidate event says governs the clause “the manager will attend a meeting later”, which contains the event attend. Hence this clause is replaced by the tag 〈event〉. While the tokens which are impossible to be new sources are replaced by 〈O〉. Sentence S5 is the PSen structure of S4.
(S4) Tom, who is the secretary of the manager, says the manager will attend a meeting later .
(S5) Tom 〈O〉 who 〈O〉 〈O〉 secretary 〈O〉 〈O〉 manager 〈O〉 says 〈event〉 〈O〉
2http://wordnet.princeton.edu
Considering that PSen is a simplified and pruned structure, we extract sentence level features through an attention-based CNN illustrated in Figure 2 instead of RNN model. A pruned sentence S = (t0, t1, . . . , tn−1) is transferred to a matrix X0 ∈ Rd0×n according to pre-trained word embeddings. Our CNN is computed as follows:
Y1 =WcX0 + bc (1)
Ym = tanh (Y1) (2)
α = softmax(vTc Ym) (3) c = tanh (Y1α T ) (4)
where Wc ∈ Rnc×d0 , bc,vc, c ∈ Rnc . The lexical feature l includes the embeddings of the token, the POS tag and the hypernym with the dimension of d0, dpos and dhyp, respectively. We concatenate c and l into f0 = [lT , cT ]T for each token. Finally, f0 is fed into the softmax layer:
o = softmax(Ws0f0 + bs0) (5)
where o ∈ R2. To train the network, we exploit the following objective function:
J(θ) = − 1 m m−1∑ i=0 log p(y(i)|x(i), θ)+ λ 2 ‖θ‖2 (6)
where p(y(i)|x(i), θ) is the confidence score of the golden label y(i) of the training instance x(i), m is the number of the training instances, λ is the regularization coefficient and θ is the parameter set.
The grammatical subjects of SIPs are chosen as the New Sources introduced by corresponding
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
SIPs. Since we have identified events and SIPs, we identify relevant sources RS, which is initialized as RS = {AUTHOR}, for each event according to the definition. When traversing from the root of the dependency parse tree of the sentence to the current event, RS is updated as:
RSn = RSn−1 ∪ {ns s} (7)
where ns is a new source introduced by the corresponding SIP and s ∈ RSn−1. This is a recursive algorithm defined by Saurı́ (2008).
4 Deep Neural Network Framework
This section describes our neural networks for event factuality identification in details shown in Figure 3. The inputs are developed according to the outputs of Section 3. We learn effective representations from Shortest Dependency Paths (SDP) by means of neural networks with attention. Different from previous studies, our model combines both BiLSTM and CNN properly which can learn meaningful features from SDPs and words, respectively.
SIP_Path RS_Path Cue_Path Auxiliary
Words
BiLSTM CNN
softmax(Ws1fu+bs1)
Input Layer
Neural Network
Layer
Softmax
Layer
softmax(Ws2fcue+bs2)
Feature Layer
Lexical Features
lp lt
fu fcue
Figure 3: The architecture of our neural networks for event factuality identification.
Our model has two outputs: one represents whether the event is Uu, Non-Uu or other, and the other shows whether the event is modified by the cue and then Non-Uu events are further classified into CT+/-, PR+/-, PS+/-. We have two main reasons for the design of the two outputs: 1) We can identify negative and speculative values (e.g., CT-, PR+/-, PS+/-) more precisely with the assistance of cues, and 2) this design can tackle the imbalance among instances, because the negative and speculative factuality values are usually in the minority. Finally, the event factuality is determined by these two outputs directly.
4.1 Inputs
We develop the following inputs for our neural network model:
Relevant Source Path (RS Path): The SDP from the root of the dependency tree to the relevant sources of the event. Notice that this path contains all the sources in the chain form.
SIP Path: The SDP from the SIP that introduces the current source to the event.
Cue Path: The SDP from the cue to the event. In addition, we consider the Lexical Features to judge whether the event is modified by the cue: 1) Relative Position is defined as the distance from the cue to the event in the sentence, and is mapped into the vector lp with the dimension dp; 2) Type of Cue includes PR, PS and NEG and is mapped into the vector lt with the dimension dt.
Auxiliary Words (Aux Words) include auxiliary and marker words that share the dependency relations aux or mark with the current event. Here, a marker is the word introducing a finite clause subordinate to the event. We argue that Aux Words can describe the syntactic structures of sentences.
An example sentence and its inputs for our neural networks are shown in Figure 4.
4.2 LSTM with Attention
Traditional RNNs have problems called vanishing gradients during the gradient back-propagation phase. Overcoming these problems is the motivation behind the LSTM model (Hochreiter and Schmidhuber, 1997) which introduces a memory cell including an input gate it, a neuron with a self-recurrent connection ct, a forget gate ft and an output gate ot. The values of components depend on the previous state ht−1 and the current input xt. Formally, LSTM is computed according to the following equations, where xt ∈ X , X ∈ Rd0×n is the matrix of the SDP:
it = σ(Wixt +Uiht−1 + bi) (8)
ft = σ(Wfxt +Ufht−1 + bf ) (9)
ot = σ(Woxt +Uoht−1 + bo) (10)
c̃t = tanh(Wcxt +Ucht−1 + bc) (11)
ct = ft ct−1 + it c̃t (12) ht = ot tanh(ct) (13)
where is the element-wise multiplication and σ is the sigmoid function. The dimension of hidden units in LSTM is set as d0. We can obtain H ∈ Rd0×n via the LSTM layer, where H = (h0,h1, . . . ,hn−1), ht ∈ Rd0(0 ≤ t ≤ n− 1).
Many sequence modeling tasks are beneficial from the access to the future as well as past con-
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
mark
The journalist reports that a famous economist claims that there is a high possibility that the economy will recover soon .
nsubj
debt
ROOT
root
mark
det
amod nsubj
ccomp
expl
ccomp
det
amod
nsubj
mark
det nsubj
aux
ccomp
advmod
Current event: recover Current Relevant Source:
economist_journalist_AUTHOR
Current SIP: claims Current cue: possibility
RS_Path: “ROOT root reports nsubj journalist nsubj reports ccomp claims nsubj economist” SIP_Path: “claims ccomp is ccomp recover” Cue_Path: “possibility nsubj is ccomp recover”
Relative Position from cue to event: -5 Type of Cue: PS Aux_Words: “that will”
Figure 4: An example sentence and its inputs.
LSTM LSTM LSTM LSTM LSTM
LSTM LSTM LSTM LSTM LSTM
Input Layer
LSTM Layer
Attention
Layer
Output Layer
x0 x1 x2 x3 x4
h0 h1 h2 h3 h4
×α0 ×α1 ×α2 ×α3 ×α4
tanh
Figure 5: The bidirectional LSTM network with attention mechanism.
text. Therefore, to model the representations of syntactic paths, we utilize the bidirectional LSTM network shown in Figure 5 which processes the syntactic path in both directions. It produces the forward hidden sequence ~H , the backward hidden sequence ~H and the output sequenceHp:
Hp = ~H + ~H (14)
To capture the most important information from the syntactic path effectively, we adopt the attention mechanism and get the output hp:
M = tanh(Hp) (15)
α = softmax(vTM) (16)
hp = tanh(Hpα T ) (17)
4.3 Outputs
The feature representations hsp, hrp and hcp are extracted from SIP Path, RS Path and Cue Path,
respectively. Noticing that the auxiliary words are just a collection of tokens instead of a sequence with specific meanings, we employ the CNN in Section 3, i.e., Equation (1) to (4), to learn the representations of auxiliary words:
fw = CNN(Xaux words) (18)
We concatenate hsp, hrp and fw into fu to judge whether the event is Uu, Non-Uu or other:
fu = [h T sp,h T rp,f T w ] T (19)
To determine whether a Non-Uu event is governed by a cue, we consider not only hcp but also the lexical features lp (Relative Position) and lt (Type of Cue) described above:
fcue = [l T p , l T t ,h T cp] T (20)
Finally, these feature representations are fed into the softmax layer:
o1 = softmax(Ws1fu + bs1) (21)
o2 = softmax(Ws2fcue + bs2) (22)
whereWs1, bs1,Ws2, bs2 are the parameters. The dimension of o1 is 3, which is equal to the number of labels representing whether the event is Uu, Non-Uu or other (label y1). While o2 is used to determine whether the event is governed by the cue, and Non-Uu events are classified into CT+/-, PR+/-, PS+/- according to o2. The dimension of o2 is 3 (label y2), considering that a sentence may have no cues. The objective function with l2-norm is designed as:
J(θ) = [− 1 m m−1∑ i=0 log p(y (i) 1 |x (i), θ)]+ (23)
(1− )[− 1 m m−1∑ i=0 log p(y (i) 2 |x (i), θ)] + λ 2 ‖θ‖2
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
where given the training instance (x, y1) and (x, y2), p(y (i) 1 |x(i), θ) and p(y (i) 2 |x(i), θ) are the confidence scores of the golden label y1 and y2 in o1 and o2, respectively.
5 Experimentation
This section first introduces experimental settings and then gives the detailed results and analysis of event factuality identification.
5.1 Experimental Settings
We evaluate our models on FactBank (Saurı́ and Pustejovsky, 2009), which contains 3864 sentences, 9492 events and 13506 factuality values. Table 2 presents the distribution of factuality values in FactBank. Following previous studies (Saurı́, 2008; de Marneffe et al., 2012), we only take into account the five main categories of values, i.e., CT+, CT-, PR+, PS+, and Uu, which make up 99.05% of all the instances.
Counts Percentage(%) CT+ 7749 53.37 CT- 433 3.21 PR+ 363 2.69 PS+ 226 1.67 Uu 4607 34.11
Other 128 0.95 Total 13506 100
Table 2: Distribution of factuality values in FactBank.
We divide the corpus into 5 folds to perform 5-fold cross-validation. Precision, Recall, and F1-measure are employed to report the performance of each factuality value. To obtain the performance for the whole corpus, macro-averaging and micro-averaging are applied (Manning and Schütze, 2001). We choose two-sample two-tailed t-test for significance test.
For the SIP detection task, we set d0 = 100, dpos = dhyp = 50, nc = 150. For the event factuality identification task, we set d0 = 100, dp = dt = 10, nc = 50, = 0.25. We initialize word embeddings via Word2Vec (Mikolov et al., 2013) and randomly initialize all the other parameters. The Stochastic Gradient Descent with momentum is applied to optimize our models.
If there is more than one cue in the sentence, we consider whether the event is modified by each cue separately. If a Non-Uu event is affected by both PR and PS cues, we adopt the cue with the
highest confidence score in o2 to decide whether the modality of the event is PR or PS.
We employ the following baselines, whose inputs are the outputs of our basic factor extraction for the fair comparison with our model:
Rules: The rule-based model developed by Saurı́ (2008) is a top-down algorithm traversing a dependency parse tree.
MaxEnt: We employ a maximum entropy classification model using the features developed by de Marneffe et al. (2012). Some features are not available (i.e., predicate classes and general classes of events) because they rely on annotated information which are not outputs of our basic factor extraction tasks.
CNN+CNN: This is a variant of our model whose LSTM is replaced by the CNN in Section 3 with the size of convolutional layer nc = 100.
5.2 Results and Analysis: Basic Factor Extraction
Table 3 presents the performances of basic factor extraction tasks. It is worth noting that a SIP is correctly identified means both this SIP and the new source introduced by it are correctly detected.
P(%) R(%) F Event 86.67 82.86 84.68 SIP 73.98 73.01 73.42 Source of events 79.30 76.53 77.82
Cue All 66.52 65.68 66.04 NEG 72.13 75.13 73.48 PR 55.92 47.97 51.36 PS 62.41 65.90 63.47
Table 3: The performances of basic factor extraction tasks.
We get a better performance on event detection task (F1=84.68) than Chambers (2013) (F1=80.30). Remember that we only consider nouns, verbs, adjectives, and rule out the effects of other tokens. We also employ the features developed by Chambers (2013) to identify SIPs and obtain F1=72.56, while our attention-based CNN for SIP detection achieves a higher F1=73.42, proving that our CNN is beneficial from pruned sentences (PSen) structures. Although the improvement of F1 is slight, however, we argue that one SIP can determine all the sources of events embedded in it. Therefore, we adopt the attention-based CNN model for SIP detection.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Systems Sources CT+ CT- PR+ PS+ Uu Micro-A Macro-A
Rules All 61.83 54.52 20.75 39.89 26.08 50.71 40.62
Author 64.83 48.83 13.35 31.93 26.17 53.49 37.02 Embed 53.19 61.94 25.11 47.01 25.95 44.20 42.64
MaxEnt All 64.37 47.57 35.74 17.08 58.86 61.00 44.87
Author 70.93 48.36 28.25 14.67 67.19 68.41 50.30 Embed 50.61 45.78 38.49 24.97 24.26 43.69 36.82
CNN+CNN(Avg) All 60.15 51.53 32.88 34.06 56.67 57.66 47.06
Author 66.11 54.67 29.25 34.67 63.44 64.14 49.69 Embed 47.88 48.35 34.92 36.49 27.09 42.59 38.95
CNN+CNN(Att) All 62.50 52.56 42.56 40.02 56.69 59.43 50.87
Author 68.97 53.86 39.26 35.91 64.16 66.23 52.43 Embed 49.37 50.76 43.57 41.10 25.32 43.69 42.03
BiLSTM+CNN(Avg) All 61.74 49.19 34.09 40.19 55.59 58.23 48.16
Author 67.67 50.33 34.55 31.10 62.94 64.79 49.32 Embed 49.68 47.95 34.48 43.13 22.86 42.94 39.62
BiLSTM+CNN(Att) All 64.17 53.89 41.83 42.97 58.46 61.06 52.26
Author 70.16 55.39 36.84 31.75 65.90 67.59 52.01 Embed 51.68 52.51 43.39 47.45 26.26 45.81 44.26
Table 4: The performances (F1-measures) of all the systems on event factuality identification. “Embed” are the sources in the chain form embedded in AUTHOR. “Avg” means using average pooling layers, and “Att” denotes the attention mechanism.
5.3 Results and Analysis: Models
Table 4 displays the performances of various models on event factuality identification. CT-, PR+ and PS+ only cover 7.57% of all the factuality values. Hence, it is challenging to identify them. Compared with Rule-based and MaxEnt model, our attention-based neural network models achieve better results on CT-, PR+ and PS+. The higher macro-averaging indicates the performances are more balanced, and BiLSTM+CNN(Att) is superior to all the other baselines in terms of microand macro-averaging on All sources. However, if we concatenate fu and fcue in Equation (19)(20) into ONE vector and design only ONE output in BiLSTM+CNN(Att), we obtain good results on CT+ and Uu (F1=63.64 and 57.80), but results of CT-, PR+, PS+ are lower (F1=49.43, 33.06, 37.07). With the design of two outputs, BiLSTM+CNN(Att) achieves the highest F1 of PS+ (42.97) and the second highest F1 of CT-, PR+ (53.89 and 41.83), which can prove the advantages of the design of two outputs in our model and the usefulness of speculative and negative cues.
Particularly, our BiLSTM model can obtain better results than CNN. Compared with the simple CNN which ignores the contexts, BiLSTM can learn features from both future and past contexts. Hence BiLSTM is able to ex-
tract more effective representations from syntactic paths than CNN. For BiLSTM+CNN(Att) and CNN+CNN(Att) models, we get p < 0.05.
The performances of both CNN+CNN (p < 0.001) and BiLSTM+CNN (p < 0.05) models are improved after using attention, indicating that some factors (e.g., SIPs, cues, auxiliary words) in syntactic paths play the key roles in computing the factuality, while others have fewer effects. Therefore, attention is helpful to improve the results.
The F1 of BiLSTM+CNN(Att) on CT+ and Uu events whose sources are AUTHOR are slightly lower than those of MaxEnt model. CT+ and Uu values cover 95.45% among events with AUTHOR, so MaxEnt can also perform well on identifying them. But BiLSTM+CNN(Att) achieves higher F1 on CT- (p < 0.001), PR+ (p < 0.05) and PS+ (p < 0.001) on AUTHOR.
However, F1 of Uu on embedded sources of BiLSTM+CNN(Att) is quite low (26.26), which is similar to other models. On one hand, Uu only covers 23.99% among events with embedded sources. On the other hand, the events embedded in other sources usually have complex syntactic structures. Therefore, it is difficult to judge whether these events are Uu or Non-Uu. Compared with MaxEnt model, however, BiLSTM+CNN(Att) achieves better performances on
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
CT-, PR+, PS+ events with AUTHOR (CT-, PS+: p < 0.001) and on all the five categories of values with embedded sources (p < 0.001). These results can prove the effectiveness of our model, especially cue-related features which are helpful to identify speculative and negative values.
5.4 Results and Analysis: Effects of Inputs
We explore the effects of different inputs on the performances of our BiLSTM+CNN(Att) model and obtain the results in Figure 6. We can conclude that only RS Path is not sufficient to identify Uu and Non-Uu events. SIP Path can greatly improve the results (p < 0.001), proving that SIPs are key factors to determine the event factuality and can offer meaningful syntactic information to discriminate Non-Uu events from Uu ones. Finally, auxiliary words make further improvement on results, because they can reflect the pragmatic structures of sentences, just as described in Section 4. For example:
(S6) He also is charged in the 1996 Olympic bombing, which killed one person.
(S7) The environmental commission must adopt regulations to ensure people are not exposed to radioactive waste.
In S6, the auxiliary word “which” leads the relative clause “which killed one person” containing the CT+ event killed. Events in non-restrictive clauses are commonly evaluated as facts. In S7, event ensure is in the purpose clause and is presented as Uu grammatically.
R: RS_Path S: SIP_Path AW: Auxiliary Words C: Cue_Info = Cue_Path + Lexical Features lp and lt
0
10
20
30
40
50
60
70
R+C S+C S+R+C S+R+C+AW
CT+ CT- PR+ PS+ Uu Micro-Ave Macro-Ave
Figure 6: F1-measures of our BiLSTM+CNN(Att) model using different inputs.
6 Related Work
Currently, only a few studies focused on event factuality identification. Diab et al. (2009) and Prabhakaran et al. (2010) presented a preliminary pilot study of belief annotation and automatic tagging,
and limited the source to AUTHOR. They classified predicates into Committed Belief (CB), NonCB or Not Applicable under a supervised framework using lexical and syntactic features.
Saurı́ (2008) proposed a rule-based top-down algorithm traversing dependency trees which can achieve excellent results (F1 of macro- and microaveraging are 73.59 and 85.45, respectively), and developed FactBank corpus (Saurı́ and Pustejovsky, 2009). Then de Marneffe et al. (2012) employed lexical, syntactic and pragmatic features to identify factuality of events in some sentences of FactBank. These studies utilized annotated events and relative factors directly.
Neural networks, emerging in recent years, can not only extract effective features from sentences (Socher et al., 2012; Zeng et al., 2014; Zhou and Xu, 2015), but also capture important information from syntactic paths and improve performances of many NLP applications, such as relation classification (Xu et al., 2015a,b) and semantic role labeling (Roth and Lapata, 2016).
Many recent researches show that attentionbased models can improve the performances of NLP tasks. Chen et al. (2016) built a hierarchical LSTM model with attention to generate sentence and document representations for sentiment classification. Zhou et al. (2016) utilized a bidirectional LSTM with attention to model sentences, and Wang et al. (2016) proposed a novel CNN relying on two levels of attention in terms of relation classification. In this paper we utilize attentionbased neural networks which consider both LSTM and CNN to learn effective features from syntactic paths and words, respectively.
7 Conclusions
This paper presents a supervised framework to identify factuality of events from raw texts. We firstly extract events, SIPs, relevant sources and cues in texts, and then employ an attention-based neural network model combining BiLSTM and CNN for event factuality identification. Our model can identify negative and speculative factuality values more effectively with the help of corresponding cues. Experimental results show that our model can outperform the state-of-the-art ones. We will develop more useful features for basic factor extraction tasks and design better neural network models to improve the results of event factuality identification in the future work.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
Comments after author response  - Thank you for clarifying that the unclear  two-step framework  reference was not about the two facets. I still do not find this use of a pipeline to be a particularly interesting contribution. - You state that  5. de Marneffe (2012) used additional annotated features in their system. For fair comparison, we re-implement their system with annotated information in FactBank.  But the de Marneffe et al. feature cited in the paper,  Predicate Classes  requires only a dependency parser and vocabulary lists from Roser Saur u00ed's PhD thesis;  general classes of event  might be referring to FactML event classes, and while I admit it is not particularly clear in their work, I am sure they could clarify. - I continue to find the use of  combined properly  to be obscure. I agree that using LSTM and CNN where respectively appropriate is valuable, but you seem to imply that some prior work has been improper, and that it is their combination which must be proper. - Thank you for reporting on separate LSTMs for each of the paths. I am curious as to why this combination may less effective. In any case, experiments with this kind of alternative structure deserve to be reported.  ---  This paper introduces deep neural net technologies to the task of factuality classification as defined by FactBank, with performance exceeding alternative neural net models and baselines reimplemented from the literature.  - Strengths:  This paper is very clear in its presentation of a sophisticated model for factuality classification and of its evaluation.  It shows that the use of attentional features and BiLSTM clearly provide benefit over alternative pooling strategies, and that the model also exceeds the performance of a more traditional feature-based log-linear model.  Given the small amount of training data in FactBank, this kind of highly-engineered model seems appropriate. It is interesting to see that the BiLSTM CNN model is able to provide benefit despite little training data.  - Weaknesses:  My main concerns with this work regard its (a) apparent departure from the evaluation procedure in the prior literature; (b) failure to present prior work as a strong baseline; and (c) novelty.  While I feel that the work is original in engineering deep neural nets for the factuality classification task, and that such work is valuable, its approach is not particularly novel, and  the proposal of a two-step supervised framework  (line 087) is not particularly interesting given that FactBank was always described in terms of two facets (assuming I am correct to interpret  two-step  as referring to these facets, which I may not be).  The work cites Saur u00ed and Pustejovsky (2012), but presents their much earlier (2008) and weaker system as a baseline; nor does it consider Qian et al.'s (IALP 2015) work which compares to the former.              Both these works are developed on the TimeBank portion of FactBank and evaluated on a held-out ACQUAINT TimeBank section, while the present work does not report results on a held-out set.  de Marneffe et al.'s (2012) system is also chosen as a baseline, but not all their features are implemented, nor is the present system evaluated on their PragBank corpus (or other alternative representations of factuality proposed in Prabhakaran et al. (*SEM 2015) and Lee et al. (EMNLP 2015)).  The evaluation is therefore somewhat lacking in comparability to prior work.  There were also important questions left unanswered in evaluation, such as the effect of using gold standard events or SIPs.  Given the famed success of BiLSTMs with little feature engineering, it is somewhat disappointing that this work does not attempt to consider a more minimal system employing deep neural nets on this task with, for instance, only the dependency path from a candidate event to its SIP plus a bag of modifiers to that path. The inclusion of heterogeneous information in one BiLSTM was an interesting feature, which deserved more experimentation: what if the order of inputs were permuted? what if delimiters were used in concatenating the dependency paths in RS instead of the strange second  nsubj  in the RS chain of line 456? What if each of SIP_path, RS_path, Cue_path were input to a separate LSTM and combined? The attentional features were evaluated together for the CNN and BiLSTM components, but it might be worth reporting whether it was beneficial for each of these components. Could you benefit from providing path information for the aux words? Could you benefit from character-level embeddings to account for morphology's impact on factuality via tense aspect? Proposed future work is lacking in specificity seeing as there are many questions raised by this model and a number of related tasks to consider applying it to.  - General Discussion:  194: Into what classes are you classifying events?  280: Please state which are parameters of the model.  321: What do you mean by  properly ? You use the same term in 092 and it's not clear which work you consider improper nor why.  353: Is  the chain form  defined anywhere? Citation? The repetition of nsubj in the example of line 456 seems an unusual feature for the LSTM to learn.  356: It may be worth footnoting here that each cue is classified separately.  359:  distance  to  surface distance   514: How many SIPs? Cues? Perhaps add to Table 3.  Table 2. Would be good if augmented by the counts for embedded and author events. Percentages can be removed if necessary.  532: Why 5-fold? Given the small amount of training data, surely 10-fold would be more useful and not substantially increase training costs.  594: It's not clear that this benefit comes from PSen, nor that the increase is significant or substantial.  Does it affect overall results substantially?  674: Is this significance across all metrics?  683: Is the drop of F1 due to precision, recall or both?  686: Not clear what this sentence is trying to say.  Table 4: From the corpus sizes, it seems you should only report 2 significant figures for most columns (except CT+, Uu and Micro-A).  711: It seems unsurprising that RS_path is insufficient given that the task is with respect to a SIP and other inputs do not encode that information. It would be more interesting to see performance of SIP_path alone.  761: This claim is not precise, to my understanding. de Marneffe et al (2012) evaluates on PragBank, not FactBank.  Minor issues in English usage:  112:  non-application  to  not applicable   145: I think you mean  relevant  to  relative   154:  can be displayed by a simple source  is unclear  166: Not sure what you mean by  basline . Do you mean  pipeline ?