Event Factuality Identification via Deep Neural Networks
Event factuality identification plays an important role in deep NLP applications. In this paper, we propose a deep learning framework for this task which first extracts essential information from raw texts as the inputs and then identifies the factuality of events via a deep neural network with a proper combination of Bidirectional Long Short-Term Memory (BiLSTM) neural network and Convolutional Neural Network (CNN). The experimental results on FactBank show that our framework significantly outperforms several state-of-the-art baselines.
1 Introduction
Event factuality is the information expressing the commitment of relevant sources towards the factual nature of events. That is, event factuality conveys whether an event is characterized as a fact, a certain possibility, or an impossible situation, and thus is helpful for various NLP applications, e.g., opinion detection (Wiebe et al., 2005), QA (Pustejovsky et al., 2005), textual entailment (de Marneffe et al., 2006; Hickl and Bensley, 2007), rumor identification (Qazvinian et al., 2011).
In principle, the value of event factuality is related to some elements, e.g., predicates, speculative and negative cues. Consider following sentences as examples1:
(S1) McCulley, a famous economist, doubts that the tax rate will increase soon.
(S2) He knows they are not able to go to the village due to the flood.
In sentence S1, the event increase is a possibility according to the predicate doubts, while in
1In this paper, events are in bold and sources are underlined in example sentences.
S2, the event go is regarded as an impossible situation due to the negation word not. From these sentences, we can learn that predicates and cues can determine the event factuality to a great degree. Such event factuality provides the way to distinguish fact events from speculative and negative ones.
Traditional methods for event factuality identification usually employed either hand-crafted rules (Saurı́, 2008; Saurı́ and Pustejovsky, 2012) relying on expert knowledge, or features (Prabhakaran et al., 2010; de Marneffe et al., 2012) relying on various kinds of annotated information, such as source introducing predicates, predicate classes, speculative and negative cues. Due to the recent success of deep learning on various NLP tasks in learning useful representations from ordinary sentences (Socher et al., 2012; Zeng et al., 2014; Cheng et al., 2016) and syntactic paths (Xu et al., 2015a,b; Roth and Lapata, 2016) and the attention mechanism on capturing the focus change in sequence modeling (Chen et al., 2016; Wang et al., 2016; Zhou et al., 2016), this paper proposes a framework to identify event factuality in raw texts with neural networks. Our main contributions can be summarized as follows:
1) the proposal of a two-step supervised framework for identifying event factuality in raw texts.
2) the utilization of an attention-based CNN to detect source introducing predicates (SIPs), the most important factors to identify event factuality.
3) the proposal of an attention-based deep neural network model with a proper combination of BiLSTM and CNN to identify the factuality of events.
2 Background
This section presents the concepts and basic factors of event factuality.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
187
188
189
190
191
192
193
194
195
196
197
198
199
2.1 Event Factuality
Factuality can be characterized by the combination of two dimensions: epistemic modality and polarity (Saurı́, 2008). While modality conveys the certainty degree of events, such as certain (CT), probable (PR), possible (PS) and underspecified (U), polarity expresses whether the event happened, including positive (+), negative (-) and underspecified (u). Table 1 shows various values of event factuality. For example, CT+ means it is certain that the event happened, while PR- denotes it is probable that the event did not happen. Notice that some values are non-application (NA) grammatically, e.g., PRu, PSu, U+/-.
+ - u CT CT+ CT- CTu PR PR+ PR- (NA) PS PS+ PS- (NA) U (NA) (NA) Uu
Table 1: Various values of event factuality.
2.2 Basic Factors
Considering that FactBank is built on top of TimeBank (Pustejovsky et al., 2003b), we adopt the definition of events proposed by TimeML (Pustejovsky et al., 2003a) from a more grammatical and generalized perspective. We consider the events which can be critical for computing the factuality. Such events can be detected by feature-based methods (Chambers, 2013).
In the literature, three kinds of factors play critical roles in event factuality identification: Source Introducing Predicate (SIP), source and cue.
SIP. Source Introducing Predicates (SIPs) are events that can not only introduce additional sources to assess the factuality of embedded events, but also influence their factuality. For example, the SIP doubts introduces McCulley as a new source in S1, and McCulley evaluates event increase as a possibility according to doubts.
Source. By default, AUTHOR is considered the only source if the sentence contains no SIPs. Further sources can be incorporated by SIPs. In S1, the event increase has two sources: AUTHOR and McCulley. The factuality value is not the inherent property of the event but always relevant to the sources. McCulley commits to the event increase as a possibility, while AUTHOR remains uncommitted. Strictly speaking, we as readers know about McCulley’s perspective only accord-
ing to AUTHOR. Hence, we appeal to the notion of the nested source structure (Wiebe et al., 2005) and represent the sources in the chain form: McCulley AUTHOR, which means McCulley is Embedded in AUTHOR. Only AUTHOR can be displayed by a simple source.
Cues are words that have speculative or negative meanings. The modality of a Non-Uu event is PR/PS if this event is modified by a PR/PS cue, while the polarity of a Non-Uu event is negative if this event is modified by a negative (NEG) cue. So cues are key signals to identify the event factuality. For instance, the factuality of event go in S1 is CT- due to the NEG cue not.
3 Baseline
As a baseline, we present a two-step framework shown in Figure 1 to identify event factuality in raw texts. Firstly, various useful factors, such as SIPs, sources, and cues are extracted from raw texts. Then, a classification algorithm is employed to identify event factuality.
3.1 Event and Cue Detection
For event detection task, we only consider nouns, verbs, adjectives as the candidate tokens to reduce the complexity, and we utilize a maximum entropy classification model with the features used by Chambers (2013).
Øvrelid et al. (2010) and Velldal et al. (2012) concluded that syntactic features are not necessary and lexical sequence-oriented n-gram features can achieve excellent performances on cue detection
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
task. Hence, we employ the lexical features developed by Velldal et al. (2012) to classify each token into PR cue, PS cue, Neg cue, or not cue.
3.2 SIP Detection
Whether an event is a SIP has relation to its semantic information. For instance, verbs state, say, tell are usually used to express opinions which are promoted by certain participants of events, so they can all introduce additional sources. Therefore, they are in the same synonym set {state, say, tell} and share the same hypernym in WordNet2. Hence, we consider token, part-of-speech (POS) and hypernym of the token as lexical level features, and concatenate them into the vector l.
According to the definition of SIPs, if there is no event in the syntactic scope of a word, this word is not a SIP. For example:
(S3) Tom suggests we should have a rest now. In S3, although the event suggests can introduce a new source Tom for the clause “we should have a rest now”, however, this clause expresses a suggestion and is not an event. So the event suggests is NOT a SIP in S3. This example demonstrates the structure of a sentence is also able to determine whether a token is a SIP. Therefore, we also consider the sentence level feature. Instead of original sentences, we propose the following Pruned Sentence structure:
Pruned Sentence (PSen): If a clause of the candidate token contains events, this clause is replaced by the tag 〈event〉; Nouns, pronouns and the current candidate token are unchanged, while other tokens are replaced by the tag 〈O〉.
We argue that PSen can characterize clearly whether there are events in the clause of the token. Besides, in PSen only the candidate token and the tokens that are possible to be new sources are reserved, and effects of other tokens are omitted.
For example, in S4, the current candidate event says governs the clause “the manager will attend a meeting later”, which contains the event attend. Hence this clause is replaced by the tag 〈event〉. While the tokens which are impossible to be new sources are replaced by 〈O〉. Sentence S5 is the PSen structure of S4.
(S4) Tom, who is the secretary of the manager, says the manager will attend a meeting later .
(S5) Tom 〈O〉 who 〈O〉 〈O〉 secretary 〈O〉 〈O〉 manager 〈O〉 says 〈event〉 〈O〉
2http://wordnet.princeton.edu
Considering that PSen is a simplified and pruned structure, we extract sentence level features through an attention-based CNN illustrated in Figure 2 instead of RNN model. A pruned sentence S = (t0, t1, . . . , tn−1) is transferred to a matrix X0 ∈ Rd0×n according to pre-trained word embeddings. Our CNN is computed as follows:
Y1 =WcX0 + bc (1)
Ym = tanh (Y1) (2)
α = softmax(vTc Ym) (3) c = tanh (Y1α T ) (4)
where Wc ∈ Rnc×d0 , bc,vc, c ∈ Rnc . The lexical feature l includes the embeddings of the token, the POS tag and the hypernym with the dimension of d0, dpos and dhyp, respectively. We concatenate c and l into f0 = [lT , cT ]T for each token. Finally, f0 is fed into the softmax layer:
o = softmax(Ws0f0 + bs0) (5)
where o ∈ R2. To train the network, we exploit the following objective function:
J(θ) = − 1 m m−1∑ i=0 log p(y(i)|x(i), θ)+ λ 2 ‖θ‖2 (6)
where p(y(i)|x(i), θ) is the confidence score of the golden label y(i) of the training instance x(i), m is the number of the training instances, λ is the regularization coefficient and θ is the parameter set.
The grammatical subjects of SIPs are chosen as the New Sources introduced by corresponding
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
SIPs. Since we have identified events and SIPs, we identify relevant sources RS, which is initialized as RS = {AUTHOR}, for each event according to the definition. When traversing from the root of the dependency parse tree of the sentence to the current event, RS is updated as:
RSn = RSn−1 ∪ {ns s} (7)
where ns is a new source introduced by the corresponding SIP and s ∈ RSn−1. This is a recursive algorithm defined by Saurı́ (2008).
4 Deep Neural Network Framework
This section describes our neural networks for event factuality identification in details shown in Figure 3. The inputs are developed according to the outputs of Section 3. We learn effective representations from Shortest Dependency Paths (SDP) by means of neural networks with attention. Different from previous studies, our model combines both BiLSTM and CNN properly which can learn meaningful features from SDPs and words, respectively.
SIP_Path RS_Path Cue_Path Auxiliary
Words
BiLSTM CNN
softmax(Ws1fu+bs1)
Input Layer
Neural Network
Layer
Softmax
Layer
softmax(Ws2fcue+bs2)
Feature Layer
Lexical Features
lp lt
fu fcue
Figure 3: The architecture of our neural networks for event factuality identification.
Our model has two outputs: one represents whether the event is Uu, Non-Uu or other, and the other shows whether the event is modified by the cue and then Non-Uu events are further classified into CT+/-, PR+/-, PS+/-. We have two main reasons for the design of the two outputs: 1) We can identify negative and speculative values (e.g., CT-, PR+/-, PS+/-) more precisely with the assistance of cues, and 2) this design can tackle the imbalance among instances, because the negative and speculative factuality values are usually in the minority. Finally, the event factuality is determined by these two outputs directly.
4.1 Inputs
We develop the following inputs for our neural network model:
Relevant Source Path (RS Path): The SDP from the root of the dependency tree to the relevant sources of the event. Notice that this path contains all the sources in the chain form.
SIP Path: The SDP from the SIP that introduces the current source to the event.
Cue Path: The SDP from the cue to the event. In addition, we consider the Lexical Features to judge whether the event is modified by the cue: 1) Relative Position is defined as the distance from the cue to the event in the sentence, and is mapped into the vector lp with the dimension dp; 2) Type of Cue includes PR, PS and NEG and is mapped into the vector lt with the dimension dt.
Auxiliary Words (Aux Words) include auxiliary and marker words that share the dependency relations aux or mark with the current event. Here, a marker is the word introducing a finite clause subordinate to the event. We argue that Aux Words can describe the syntactic structures of sentences.
An example sentence and its inputs for our neural networks are shown in Figure 4.
4.2 LSTM with Attention
Traditional RNNs have problems called vanishing gradients during the gradient back-propagation phase. Overcoming these problems is the motivation behind the LSTM model (Hochreiter and Schmidhuber, 1997) which introduces a memory cell including an input gate it, a neuron with a self-recurrent connection ct, a forget gate ft and an output gate ot. The values of components depend on the previous state ht−1 and the current input xt. Formally, LSTM is computed according to the following equations, where xt ∈ X , X ∈ Rd0×n is the matrix of the SDP:
it = σ(Wixt +Uiht−1 + bi) (8)
ft = σ(Wfxt +Ufht−1 + bf ) (9)
ot = σ(Woxt +Uoht−1 + bo) (10)
c̃t = tanh(Wcxt +Ucht−1 + bc) (11)
ct = ft ct−1 + it c̃t (12) ht = ot tanh(ct) (13)
where is the element-wise multiplication and σ is the sigmoid function. The dimension of hidden units in LSTM is set as d0. We can obtain H ∈ Rd0×n via the LSTM layer, where H = (h0,h1, . . . ,hn−1), ht ∈ Rd0(0 ≤ t ≤ n− 1).
Many sequence modeling tasks are beneficial from the access to the future as well as past con-
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
mark
The journalist reports that a famous economist claims that there is a high possibility that the economy will recover soon .
nsubj
debt
ROOT
root
mark
det
amod nsubj
ccomp
expl
ccomp
det
amod
nsubj
mark
det nsubj
aux
ccomp
advmod
Current event: recover Current Relevant Source:
economist_journalist_AUTHOR
Current SIP: claims Current cue: possibility
RS_Path: “ROOT root reports nsubj journalist nsubj reports ccomp claims nsubj economist” SIP_Path: “claims ccomp is ccomp recover” Cue_Path: “possibility nsubj is ccomp recover”
Relative Position from cue to event: -5 Type of Cue: PS Aux_Words: “that will”
Figure 4: An example sentence and its inputs.
LSTM LSTM LSTM LSTM LSTM
LSTM LSTM LSTM LSTM LSTM
Input Layer
LSTM Layer
Attention
Layer
Output Layer
x0 x1 x2 x3 x4
h0 h1 h2 h3 h4
×α0 ×α1 ×α2 ×α3 ×α4
tanh
Figure 5: The bidirectional LSTM network with attention mechanism.
text. Therefore, to model the representations of syntactic paths, we utilize the bidirectional LSTM network shown in Figure 5 which processes the syntactic path in both directions. It produces the forward hidden sequence ~H , the backward hidden sequence ~H and the output sequenceHp:
Hp = ~H + ~H (14)
To capture the most important information from the syntactic path effectively, we adopt the attention mechanism and get the output hp:
M = tanh(Hp) (15)
α = softmax(vTM) (16)
hp = tanh(Hpα T ) (17)
4.3 Outputs
The feature representations hsp, hrp and hcp are extracted from SIP Path, RS Path and Cue Path,
respectively. Noticing that the auxiliary words are just a collection of tokens instead of a sequence with specific meanings, we employ the CNN in Section 3, i.e., Equation (1) to (4), to learn the representations of auxiliary words:
fw = CNN(Xaux words) (18)
We concatenate hsp, hrp and fw into fu to judge whether the event is Uu, Non-Uu or other:
fu = [h T sp,h T rp,f T w ] T (19)
To determine whether a Non-Uu event is governed by a cue, we consider not only hcp but also the lexical features lp (Relative Position) and lt (Type of Cue) described above:
fcue = [l T p , l T t ,h T cp] T (20)
Finally, these feature representations are fed into the softmax layer:
o1 = softmax(Ws1fu + bs1) (21)
o2 = softmax(Ws2fcue + bs2) (22)
whereWs1, bs1,Ws2, bs2 are the parameters. The dimension of o1 is 3, which is equal to the number of labels representing whether the event is Uu, Non-Uu or other (label y1). While o2 is used to determine whether the event is governed by the cue, and Non-Uu events are classified into CT+/-, PR+/-, PS+/- according to o2. The dimension of o2 is 3 (label y2), considering that a sentence may have no cues. The objective function with l2-norm is designed as:
J(θ) = [− 1 m m−1∑ i=0 log p(y (i) 1 |x (i), θ)]+ (23)
(1− )[− 1 m m−1∑ i=0 log p(y (i) 2 |x (i), θ)] + λ 2 ‖θ‖2
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
where given the training instance (x, y1) and (x, y2), p(y (i) 1 |x(i), θ) and p(y (i) 2 |x(i), θ) are the confidence scores of the golden label y1 and y2 in o1 and o2, respectively.
5 Experimentation
This section first introduces experimental settings and then gives the detailed results and analysis of event factuality identification.
5.1 Experimental Settings
We evaluate our models on FactBank (Saurı́ and Pustejovsky, 2009), which contains 3864 sentences, 9492 events and 13506 factuality values. Table 2 presents the distribution of factuality values in FactBank. Following previous studies (Saurı́, 2008; de Marneffe et al., 2012), we only take into account the five main categories of values, i.e., CT+, CT-, PR+, PS+, and Uu, which make up 99.05% of all the instances.
Counts Percentage(%) CT+ 7749 53.37 CT- 433 3.21 PR+ 363 2.69 PS+ 226 1.67 Uu 4607 34.11
Other 128 0.95 Total 13506 100
Table 2: Distribution of factuality values in FactBank.
We divide the corpus into 5 folds to perform 5-fold cross-validation. Precision, Recall, and F1-measure are employed to report the performance of each factuality value. To obtain the performance for the whole corpus, macro-averaging and micro-averaging are applied (Manning and Schütze, 2001). We choose two-sample two-tailed t-test for significance test.
For the SIP detection task, we set d0 = 100, dpos = dhyp = 50, nc = 150. For the event factuality identification task, we set d0 = 100, dp = dt = 10, nc = 50, = 0.25. We initialize word embeddings via Word2Vec (Mikolov et al., 2013) and randomly initialize all the other parameters. The Stochastic Gradient Descent with momentum is applied to optimize our models.
If there is more than one cue in the sentence, we consider whether the event is modified by each cue separately. If a Non-Uu event is affected by both PR and PS cues, we adopt the cue with the
highest confidence score in o2 to decide whether the modality of the event is PR or PS.
We employ the following baselines, whose inputs are the outputs of our basic factor extraction for the fair comparison with our model:
Rules: The rule-based model developed by Saurı́ (2008) is a top-down algorithm traversing a dependency parse tree.
MaxEnt: We employ a maximum entropy classification model using the features developed by de Marneffe et al. (2012). Some features are not available (i.e., predicate classes and general classes of events) because they rely on annotated information which are not outputs of our basic factor extraction tasks.
CNN+CNN: This is a variant of our model whose LSTM is replaced by the CNN in Section 3 with the size of convolutional layer nc = 100.
5.2 Results and Analysis: Basic Factor Extraction
Table 3 presents the performances of basic factor extraction tasks. It is worth noting that a SIP is correctly identified means both this SIP and the new source introduced by it are correctly detected.
P(%) R(%) F Event 86.67 82.86 84.68 SIP 73.98 73.01 73.42 Source of events 79.30 76.53 77.82
Cue All 66.52 65.68 66.04 NEG 72.13 75.13 73.48 PR 55.92 47.97 51.36 PS 62.41 65.90 63.47
Table 3: The performances of basic factor extraction tasks.
We get a better performance on event detection task (F1=84.68) than Chambers (2013) (F1=80.30). Remember that we only consider nouns, verbs, adjectives, and rule out the effects of other tokens. We also employ the features developed by Chambers (2013) to identify SIPs and obtain F1=72.56, while our attention-based CNN for SIP detection achieves a higher F1=73.42, proving that our CNN is beneficial from pruned sentences (PSen) structures. Although the improvement of F1 is slight, however, we argue that one SIP can determine all the sources of events embedded in it. Therefore, we adopt the attention-based CNN model for SIP detection.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Systems Sources CT+ CT- PR+ PS+ Uu Micro-A Macro-A
Rules All 61.83 54.52 20.75 39.89 26.08 50.71 40.62
Author 64.83 48.83 13.35 31.93 26.17 53.49 37.02 Embed 53.19 61.94 25.11 47.01 25.95 44.20 42.64
MaxEnt All 64.37 47.57 35.74 17.08 58.86 61.00 44.87
Author 70.93 48.36 28.25 14.67 67.19 68.41 50.30 Embed 50.61 45.78 38.49 24.97 24.26 43.69 36.82
CNN+CNN(Avg) All 60.15 51.53 32.88 34.06 56.67 57.66 47.06
Author 66.11 54.67 29.25 34.67 63.44 64.14 49.69 Embed 47.88 48.35 34.92 36.49 27.09 42.59 38.95
CNN+CNN(Att) All 62.50 52.56 42.56 40.02 56.69 59.43 50.87
Author 68.97 53.86 39.26 35.91 64.16 66.23 52.43 Embed 49.37 50.76 43.57 41.10 25.32 43.69 42.03
BiLSTM+CNN(Avg) All 61.74 49.19 34.09 40.19 55.59 58.23 48.16
Author 67.67 50.33 34.55 31.10 62.94 64.79 49.32 Embed 49.68 47.95 34.48 43.13 22.86 42.94 39.62
BiLSTM+CNN(Att) All 64.17 53.89 41.83 42.97 58.46 61.06 52.26
Author 70.16 55.39 36.84 31.75 65.90 67.59 52.01 Embed 51.68 52.51 43.39 47.45 26.26 45.81 44.26
Table 4: The performances (F1-measures) of all the systems on event factuality identification. “Embed” are the sources in the chain form embedded in AUTHOR. “Avg” means using average pooling layers, and “Att” denotes the attention mechanism.
5.3 Results and Analysis: Models
Table 4 displays the performances of various models on event factuality identification. CT-, PR+ and PS+ only cover 7.57% of all the factuality values. Hence, it is challenging to identify them. Compared with Rule-based and MaxEnt model, our attention-based neural network models achieve better results on CT-, PR+ and PS+. The higher macro-averaging indicates the performances are more balanced, and BiLSTM+CNN(Att) is superior to all the other baselines in terms of microand macro-averaging on All sources. However, if we concatenate fu and fcue in Equation (19)(20) into ONE vector and design only ONE output in BiLSTM+CNN(Att), we obtain good results on CT+ and Uu (F1=63.64 and 57.80), but results of CT-, PR+, PS+ are lower (F1=49.43, 33.06, 37.07). With the design of two outputs, BiLSTM+CNN(Att) achieves the highest F1 of PS+ (42.97) and the second highest F1 of CT-, PR+ (53.89 and 41.83), which can prove the advantages of the design of two outputs in our model and the usefulness of speculative and negative cues.
Particularly, our BiLSTM model can obtain better results than CNN. Compared with the simple CNN which ignores the contexts, BiLSTM can learn features from both future and past contexts. Hence BiLSTM is able to ex-
tract more effective representations from syntactic paths than CNN. For BiLSTM+CNN(Att) and CNN+CNN(Att) models, we get p < 0.05.
The performances of both CNN+CNN (p < 0.001) and BiLSTM+CNN (p < 0.05) models are improved after using attention, indicating that some factors (e.g., SIPs, cues, auxiliary words) in syntactic paths play the key roles in computing the factuality, while others have fewer effects. Therefore, attention is helpful to improve the results.
The F1 of BiLSTM+CNN(Att) on CT+ and Uu events whose sources are AUTHOR are slightly lower than those of MaxEnt model. CT+ and Uu values cover 95.45% among events with AUTHOR, so MaxEnt can also perform well on identifying them. But BiLSTM+CNN(Att) achieves higher F1 on CT- (p < 0.001), PR+ (p < 0.05) and PS+ (p < 0.001) on AUTHOR.
However, F1 of Uu on embedded sources of BiLSTM+CNN(Att) is quite low (26.26), which is similar to other models. On one hand, Uu only covers 23.99% among events with embedded sources. On the other hand, the events embedded in other sources usually have complex syntactic structures. Therefore, it is difficult to judge whether these events are Uu or Non-Uu. Compared with MaxEnt model, however, BiLSTM+CNN(Att) achieves better performances on
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
CT-, PR+, PS+ events with AUTHOR (CT-, PS+: p < 0.001) and on all the five categories of values with embedded sources (p < 0.001). These results can prove the effectiveness of our model, especially cue-related features which are helpful to identify speculative and negative values.
5.4 Results and Analysis: Effects of Inputs
We explore the effects of different inputs on the performances of our BiLSTM+CNN(Att) model and obtain the results in Figure 6. We can conclude that only RS Path is not sufficient to identify Uu and Non-Uu events. SIP Path can greatly improve the results (p < 0.001), proving that SIPs are key factors to determine the event factuality and can offer meaningful syntactic information to discriminate Non-Uu events from Uu ones. Finally, auxiliary words make further improvement on results, because they can reflect the pragmatic structures of sentences, just as described in Section 4. For example:
(S6) He also is charged in the 1996 Olympic bombing, which killed one person.
(S7) The environmental commission must adopt regulations to ensure people are not exposed to radioactive waste.
In S6, the auxiliary word “which” leads the relative clause “which killed one person” containing the CT+ event killed. Events in non-restrictive clauses are commonly evaluated as facts. In S7, event ensure is in the purpose clause and is presented as Uu grammatically.
R: RS_Path S: SIP_Path AW: Auxiliary Words C: Cue_Info = Cue_Path + Lexical Features lp and lt
0
10
20
30
40
50
60
70
R+C S+C S+R+C S+R+C+AW
CT+ CT- PR+ PS+ Uu Micro-Ave Macro-Ave
Figure 6: F1-measures of our BiLSTM+CNN(Att) model using different inputs.
6 Related Work
Currently, only a few studies focused on event factuality identification. Diab et al. (2009) and Prabhakaran et al. (2010) presented a preliminary pilot study of belief annotation and automatic tagging,
and limited the source to AUTHOR. They classified predicates into Committed Belief (CB), NonCB or Not Applicable under a supervised framework using lexical and syntactic features.
Saurı́ (2008) proposed a rule-based top-down algorithm traversing dependency trees which can achieve excellent results (F1 of macro- and microaveraging are 73.59 and 85.45, respectively), and developed FactBank corpus (Saurı́ and Pustejovsky, 2009). Then de Marneffe et al. (2012) employed lexical, syntactic and pragmatic features to identify factuality of events in some sentences of FactBank. These studies utilized annotated events and relative factors directly.
Neural networks, emerging in recent years, can not only extract effective features from sentences (Socher et al., 2012; Zeng et al., 2014; Zhou and Xu, 2015), but also capture important information from syntactic paths and improve performances of many NLP applications, such as relation classification (Xu et al., 2015a,b) and semantic role labeling (Roth and Lapata, 2016).
Many recent researches show that attentionbased models can improve the performances of NLP tasks. Chen et al. (2016) built a hierarchical LSTM model with attention to generate sentence and document representations for sentiment classification. Zhou et al. (2016) utilized a bidirectional LSTM with attention to model sentences, and Wang et al. (2016) proposed a novel CNN relying on two levels of attention in terms of relation classification. In this paper we utilize attentionbased neural networks which consider both LSTM and CNN to learn effective features from syntactic paths and words, respectively.
7 Conclusions
This paper presents a supervised framework to identify factuality of events from raw texts. We firstly extract events, SIPs, relevant sources and cues in texts, and then employ an attention-based neural network model combining BiLSTM and CNN for event factuality identification. Our model can identify negative and speculative factuality values more effectively with the help of corresponding cues. Experimental results show that our model can outperform the state-of-the-art ones. We will develop more useful features for basic factor extraction tasks and design better neural network models to improve the results of event factuality identification in the future work.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

Comments after author response  - Thank you for clarifying that the unclear  two-step framework  reference was not about the two facets. I still do not find this use of a pipeline to be a particularly interesting contribution. - You state that  5. de Marneffe (2012) used additional annotated features in their system. For fair comparison, we re-implement their system with annotated information in FactBank.  But the de Marneffe et al. feature cited in the paper,  Predicate Classes  requires only a dependency parser and vocabulary lists from Roser Saur u00ed's PhD thesis;  general classes of event  might be referring to FactML event classes, and while I admit it is not particularly clear in their work, I am sure they could clarify. - I continue to find the use of  combined properly  to be obscure. I agree that using LSTM and CNN where respectively appropriate is valuable, but you seem to imply that some prior work has been improper, and that it is their combination which must be proper. - Thank you for reporting on separate LSTMs for each of the paths. I am curious as to why this combination may less effective. In any case, experiments with this kind of alternative structure deserve to be reported.  ---  This paper introduces deep neural net technologies to the task of factuality classification as defined by FactBank, with performance exceeding alternative neural net models and baselines reimplemented from the literature.  - Strengths:  This paper is very clear in its presentation of a sophisticated model for factuality classification and of its evaluation.  It shows that the use of attentional features and BiLSTM clearly provide benefit over alternative pooling strategies, and that the model also exceeds the performance of a more traditional feature-based log-linear model.  Given the small amount of training data in FactBank, this kind of highly-engineered model seems appropriate. It is interesting to see that the BiLSTM CNN model is able to provide benefit despite little training data.  - Weaknesses:  My main concerns with this work regard its (a) apparent departure from the evaluation procedure in the prior literature; (b) failure to present prior work as a strong baseline; and (c) novelty.  While I feel that the work is original in engineering deep neural nets for the factuality classification task, and that such work is valuable, its approach is not particularly novel, and  the proposal of a two-step supervised framework  (line 087) is not particularly interesting given that FactBank was always described in terms of two facets (assuming I am correct to interpret  two-step  as referring to these facets, which I may not be).  The work cites Saur u00ed and Pustejovsky (2012), but presents their much earlier (2008) and weaker system as a baseline; nor does it consider Qian et al.'s (IALP 2015) work which compares to the former.              Both these works are developed on the TimeBank portion of FactBank and evaluated on a held-out ACQUAINT TimeBank section, while the present work does not report results on a held-out set.  de Marneffe et al.'s (2012) system is also chosen as a baseline, but not all their features are implemented, nor is the present system evaluated on their PragBank corpus (or other alternative representations of factuality proposed in Prabhakaran et al. (*SEM 2015) and Lee et al. (EMNLP 2015)).  The evaluation is therefore somewhat lacking in comparability to prior work.  There were also important questions left unanswered in evaluation, such as the effect of using gold standard events or SIPs.  Given the famed success of BiLSTMs with little feature engineering, it is somewhat disappointing that this work does not attempt to consider a more minimal system employing deep neural nets on this task with, for instance, only the dependency path from a candidate event to its SIP plus a bag of modifiers to that path. The inclusion of heterogeneous information in one BiLSTM was an interesting feature, which deserved more experimentation: what if the order of inputs were permuted? what if delimiters were used in concatenating the dependency paths in RS instead of the strange second  nsubj  in the RS chain of line 456? What if each of SIP_path, RS_path, Cue_path were input to a separate LSTM and combined? The attentional features were evaluated together for the CNN and BiLSTM components, but it might be worth reporting whether it was beneficial for each of these components. Could you benefit from providing path information for the aux words? Could you benefit from character-level embeddings to account for morphology's impact on factuality via tense aspect? Proposed future work is lacking in specificity seeing as there are many questions raised by this model and a number of related tasks to consider applying it to.  - General Discussion:  194: Into what classes are you classifying events?  280: Please state which are parameters of the model.  321: What do you mean by  properly ? You use the same term in 092 and it's not clear which work you consider improper nor why.  353: Is  the chain form  defined anywhere? Citation? The repetition of nsubj in the example of line 456 seems an unusual feature for the LSTM to learn.  356: It may be worth footnoting here that each cue is classified separately.  359:  distance  to  surface distance   514: How many SIPs? Cues? Perhaps add to Table 3.  Table 2. Would be good if augmented by the counts for embedded and author events. Percentages can be removed if necessary.  532: Why 5-fold? Given the small amount of training data, surely 10-fold would be more useful and not substantially increase training costs.  594: It's not clear that this benefit comes from PSen, nor that the increase is significant or substantial.  Does it affect overall results substantially?  674: Is this significance across all metrics?  683: Is the drop of F1 due to precision, recall or both?  686: Not clear what this sentence is trying to say.  Table 4: From the corpus sizes, it seems you should only report 2 significant figures for most columns (except CT+, Uu and Micro-A).  711: It seems unsurprising that RS_path is insufficient given that the task is with respect to a SIP and other inputs do not encode that information. It would be more interesting to see performance of SIP_path alone.  761: This claim is not precise, to my understanding. de Marneffe et al (2012) evaluates on PragBank, not FactBank.  Minor issues in English usage:  112:  non-application  to  not applicable   145: I think you mean  relevant  to  relative   154:  can be displayed by a simple source  is unclear  166: Not sure what you mean by  basline . Do you mean  pipeline ?


Learning Symmetric Collaborative Dialogue Agents with Dynamic Knowledge Graph Embeddings
We study a symmetric collaborative dialogue setting in which two agents, each with private knowledge, must strategically communicate to achieve a common goal. The open-ended dialogue state in this setting poses new challenges for existing dialogue systems. We collected a dataset of 11K human-human dialogues, which exhibits interesting lexical, semantic, and strategic elements. To model both structured knowledge and unstructured language, we propose a neural model with dynamic knowledge graph embeddings that evolve as the dialogue progresses. Automatic and human evaluations show that our model is both more effective at achieving the goal and more human-like than baseline neural and rule-based models.
1 Introduction
Current task-oriented dialogue systems (Young et al., 2013; Wen et al., 2017; Dhingra et al., 2016) require a pre-defined dialogue state (e.g., slots such as food type and price range for a restaurant searching task) and a fixed set of dialogue acts (e.g., request, inform). However, human conversation often requires richer dialogue states and more nuanced, pragmatic dialogue acts. In contrast, recent open-domain chat systems (Shang et al., 2015; Serban et al., 2015b; Sordoni et al., 2015; Li et al., 2016a) learn a mapping directly from previous utterances to the next utterance. While these models capture open-ended aspects of dialogue, the lack of structured dialogue state prevents them from being directly applied to settings that require interfacing with structured knowledge.
In order to bridge the gap between the two types of systems, we focus on a symmetric collabora-
tive dialogue setting, which is task-oriented but encourages open-ended dialogue acts. In our setting, two agents, each with a private list of items with attributes, must communicate to identify the unique shared item. Consider the dialogue in Figure 1, in which two people are trying to find their mutual friend. When B asks “do you have anyone who went to columbia?”, it suggests that she has some Columbia friends and they probably work at Google. Such conversational implicature is lost when interpreting it as simply requesting information about Columbia. In addition, it is hard to define a state that captures the diverse semantics in these utterances (e.g., defining “most of”, “might be”; see details in Table 1).
To model both structured and open-ended context, we propose the Dynamic Knowledge Graph Network (DynoNet), in which the dialogue state is modeled as a knowledge graph with an embedding for each node (Section 3). Our model is similar
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
to EntNet (Henaff et al., 2017) in that node/entity embeddings are updated recurrently given new utterances. The difference is that we structure entities as a knowledge graph; as the dialogue proceeds, new nodes are added and new context is propagated on the graph. An attention-based mechanism (Bahdanau et al., 2015) over the node embeddings drives generation of new utterances. Our model’s use of knowledge graphs captures the grounding capability of classic task-oriented systems and the graph embedding provides the representational flexibility of neural models.
The naturalness of communication in the symmetric collaborative setting enables large-scale data collection: We were able to crowd-source around 11K human-human dialogues on Amazon Mechanical Turk (AMT) in less than 15 hours.1 We show that the new dataset calls for more powerful representation beyond fully-structured states (Section 2.2).
Aside from the third-party human evaluation adopted by most work (Liu et al., 2016; Li et al., 2016b,c), we also conduct partner evaluation (Wen et al., 2017) where AMT workers rate their conversational partners (our models) based on fluency, correctness, cooperation, and human-likeness. We compare DynoNet with baseline neural models and a strong rule-based system. The results show that DynoNet can perform the task with humans efficiently and naturally; it also captures some strategic aspects of human-human dialogues.
The contributions of this work are: (i) a new symmetric collaborative dialogue setting and a large dialogue corpus that push the boundaries of existing dialogue systems; (ii) DynoNet that integrates semantically rich utterances with structured knowledge to represent open-ended dialogue states; (iii) multiple automatic metrics based on bot-bot chat and a comparison of third-party and partner evaluation.
2 Symmetric Collaborative Dialogue
We introduce a collaborative task between two agents and describe the human-human dialogue collection process. We show that our data exhibits diverse, interesting language phenomena.
1The dataset is available publicly at http: //anonymous.
2.1 Task Definition
In the symmetric collaborative dialogue setting, there are two agents, A and B, each with a private knowledge base—KBA and KBB, respectively. Each knowledge base includes a list of items, where each item has a value for each attribute. For example, in the MutualFriends setting, Figure 1, items are friends and attributes are name, school, etc. There is a shared item that A and B both have; their goal is to converse with each other to determine the shared item and select it. Formally, an agent is a mapping from its private KB and the dialogue thus far (sequence of utterances) to the next utterance to generate. A dialogue is considered successful when both agents correctly select the shared item. This setting has parallels in human-computer collaboration where each agent has different expertise.
2.2 Data collection
We created a schema containing 7 attributes and more than 3K entities (attribute values) in total. To elicit linguistic and strategic variants, we generate a random scenario for each task by varying the number of items, the number attributes, and the distribution of values for each attribute. See Appendix B for details of scenario generation.
We crowd-sourced dialogues on AMT by randomly pairing up users to perform the task within 5 minutes.2 To discourage random guessing, we prevent users from selecting more than once every 10 seconds. The chat interface is shown in Appendix C. Our task was very popular and we collected 11K dialogues over a period of 13.5 hours.3
2.3 Dataset statistics
We show the basic statistics of our dataset in Table 3. An utterance is defined as a message sent by the user. The average utterance length is shorter due to the informality of the chat, however, a user usually sends multiple utterances in one turn. Some example dialogues are shown in Table 6 and Appendix H.
We coarsely categorize utterances into inform, ask, answer, greeting, apology by pattern matching (Appendix E). There are 7.4% multi-type utterances and 30.9% multi-entity utterances. In Ta-
2If users exceed the time limit, the dialogue is marked as unsuccessful (but still logged).
3Tasks are put up in batches; the total time excludes intervals between batches.
4Entity names are replaced by entity types.
3
206
207
208
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
256
257
258
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
# dialogues 11439 # completed dialogues 9041 Vocabulary size 4548 Average # of utterances 11.41 Average time taken per task (sec.) 91.18 Average utterance length (tokens) 4.84 Number of linguistic templates4 30370
Table 3: Statistics of the MutualFriends dataset.
ble 1, we show example utterances where the rich semantics cannot be sufficiently represented by traditional slot-value pairs. Even the more standard ones can be challenging due to coreference and conjunctions.
Our dataset also exhibits interesting communication phenomena that add to its complexity. Coreference occurs frequently when people check attributes jointly. Sometimes the mentions are dropped: it simply continues from the previous partner utterance. People occasionally use external knowledge to group items with out-of-schema attributes (e.g., gender based on names, location based on schools). We summarize these phenomena in Table 2. In addition, we find 30% utterances involve cross-talk where the conversation does not progress linearly (e.g., italic utterances in Figure 1). Cross-talk is a common characteristic of online chat that creates ambiguity (Ivanovic, 2005).
One strategic aspect of this task is the order in which attributes are mentioned. We find that peo-
ple tend to start from attributes with fewer unique values, e.g., “all my friends like morning” given the KBB in Table 6, as intuitively it would help exclude items quickly given fewer values to check.5 We provide a more detailed analysis of strategy in Section 4.2 and Appendix F.
3 Dynamic Knowledge Graph Network
The diverse semantics in our data motivates us to combine rich but unstructured representation of the dialogue history with structured knowledge. Our model consists of three components shown in Figure 2: (i) a dynamic knowledge graph, which represents the private KB and dialogue history in a graph (Section 3.1), (ii) a graph embedding over the nodes (Section 3.2), and (iii) an utterance generator (Section 3.3).
The knowledge graph represents entities and relations in the agent’s private KB, e.g., google belongs to company. As the conversation unrolls, utterances are embedded and incorporated to node embeddings of mentioned entities. In Figure 2, “Most of my friends work for Google” updates the embedding of google. Further, mentioned nodes pass on this new information to their neighbors so that related entities (e.g., those in the same row or column) also receive the information. In our example, jessica and josh both receive new context when google and columbia are mentioned.
5Our goal is to model human behavior thus we do not discuss the optimal strategy here.
4
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
The utterance generator, an LSTM, then produces the next utterance by attending to the node embeddings that represent the dialogue state.
3.1 Knowledge Graph
Given a dialogue of T utterances, we construct an evolving graph Gt over the KB and dialogue history for agent A.6 We consider three types of nodes: item nodes, attribute nodes, and entity nodes. Edges between nodes represent their relations. For example, (item-1, hasSchool, columbia) means that the first item has attribute school whose value is columbia. An example graph is shown in Figure 2. The graph is updated at the end of an utterance if new entities (not in KBA) are mentioned; they are added as dangling nodes.7
3.2 Graph Embedding
Given a knowledge graph, we are interested in computing a vector representation for each node v that captures both its unstructured context from the dialogue history and its structured context in the KB. The node embedding V is built from three parts: structural properties of an entity defined by the KB, embeddings of utterances in the dialogue history, and message passing between neighboring nodes.
Node Features. Simple structural properties of the KB often govern what is talked about; e.g., a high-frequency entity is often interesting (consider
6 It is important to differentiate perspectives of the two agents as they have different KBs. Thereafter we assume the perspective of agent A, i.e., accessing KBA and generating utterances for A only, and refer to B as the partner.
7 We use a rule-based lexicon to link text spans to entities. See details in Appendix D.
“All my friends like dancing.”). We represent this type of information as an indicator vector F (v), including the frequency and type (item, attribute, or entity type) of entity v.
Mention Vectors. A mention vector Mt(v) contains unstructured context from utterances relevant to node v up to turn t. Given embedding ut of the last utterance (Section 3.3), we define
Mt(v) = Mt 1(v) + (1 )ut,
=
( W [Mt 1(v), ut] if ut mentions v
1 otherwise
where is the sigmoid function and W is a projection matrix. If no entity is mentioned in ut, we update v in ut 1. This is useful when ut answers a question, e.g., “do you have any google friends?” “No.” To differentiate between the agent’s and the partner’s utterances, Mt(v) consists of two separate components for utterances from each source.
Recursive Node Embeddings. We propagate information between nodes according to the structure of the knowledge graph. In Figure 2, given “anyone went to columbia?”, the agent should focus on her Google friends who went to Columbia. Therefore, we want this information to be sent to item nodes connected to columbia, and one step further to other attributes of these items because they might be implicitly referred to (google which was mentioned previously) or mentioned next (jessica and josh which will be generated).
We compute the embeddings recursively, analogous to belief propagation:
V kt (v) = max v02N(v) tanh
⇣ Wm h V k 1t (v 0 ), R(ev!v0) i⌘ ,
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
where V kt (v) is the depth-k node embedding at turn t, Wm is a projection matrix, R is an relation embedding function, ev!v0 denotes the relation from v to v0, and max is an element-wise max operation.8 Example message passing paths are shown in Figure 2.
The final node embedding is the concatenation of embeddings at each depth:
Vt(v) = ⇥ V 0t (v), . . . , V K t (v) ⇤ ,
where K is a hyperparameter (we experiment with K 2 {0, 1, 2}) and V 0t (v) = [F (v),Mt(v)]. The node embedding is updated periodically whenever Mt(v) is updated by a new utterance.
3.3 Utterance Generation
In this section, we describe how an utterance is embedded and generated. Our model is based on a recurrent neural network (RNN). On turn t (we elide the subscript t for notational convenience), upon receiving an utterance of n tokens, {xn}Nn=1, the RNN maps it to a vector: hn = LSTMenc(hn 1, xn). We use the last hidden state hn as the utterance embedding u which updates the mention vectors.
Given (updated) embeddings of each node, we use another RNN to generate the next utterance, where each token is either copied from a node or generated from the vocabulary. Formally,
hn = LSTMdec(hn 1, [xn, ct]),
where ct is a weighted sum of node embeddings in the current turn: ct = P i ↵n,iVt(vi). As shown in Figure 2, entities relevant to generating the next utterance should have high weights. We compute the weights through standard attention mechanism: ↵n = softmax(an) and an,i = wa tanh (W a [hn 1, Vt(vi)]), where wa is a scoring vector and W a is a projection matrix.
The RNN outputs a distribution over words in the vocabulary and the nodes in Gt as in the copying mechanism of Jia and Liang (2016) :
p(xn+1 = w |xn, Gt) / exp (Uwhcn + b) , p(xn+1 = r(vi) |xn, Gt) / exp (an,i) ,
where w is a word in the vocabulary, Uw and b are the model parameters, and r(vi) is the realization of the entity represented by node i, e.g., generating “Google” given the node google.9
8Using sum slightly hurts performance. 9 We realize an entity by sampling from the empirical dis-
tribution of its surface forms in the training set.
3.4 Entity Abstraction
In our setting, the role of an entity is governed by its relation to other entities and relevant utterances. For example, replacing google with alphabet in Figure 1 should make very little difference to the conversation. Note that we do not use an embedding matrix for any entity when computing V (v). Furthermore, in the input embedding layer of the LSTM, we represent an entity by its type embedding concatenated with its node embedding. Accordingly, a regular word is represented as its word embedding concatenated with a zero vector of the same dimension as V . This way, the models’ representation of an entity only depends on its structural property in the KB and the dialogue context.
4 Experiments
4.1 Setup
We use a one-layer LSTM with 100 hidden units and 100-dimensional word vectors for both the encoder and the decoder. We maximize the likelihood of each utterance from perspectives of both agents. The parameters are optimized by AdaGrad (Duchi et al., 2010) with a learning rate of 0.5. We ran at least 10 epochs; after that, training stops if there is no improvement for 5 epochs. By default, we perform K = 2 iterations of message passing to compute node embeddings. We randomly split the data into train, dev, and test sets and only train on successful dialogues. For decoding, we sequentially sample from the output distribution with a softmax temperature of 0.5.10 Hyperparameters are tuned on the dev set. We compare DynoNet with its static cousion (StanoNet) and a rule-based system (Rule). StanoNet uses G0 throughout the dialogue, thus the dialogue context is completely contained in the RNN hidden states instead of being structured around the knowledge graph. Rule maintains weights for each entity and each item in the KB to decide what to talk about and which item to select. It has a pattern-matching semantic parser, a rule-based policy, and a templated generator. See Appendix G for details.
10 Since selection is a common ‘utterance’ in our dataset and neural generation models are susceptible to overgenerating common sentences, we halve its probability during sampling.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
4.2 Evaluation
We test our systems in two interactive settings: bot-bot chat and bot-human chat. We perform both automatic evaluation and human evaluation to test syntactic, semantic, and pragmatic competence of each system.
Automatic Evaluation. First, we compute the cross-entropy (`) of a model on test data. As shown in Table 4, DynoNet has the lowest test loss. Next, we have a model chat with itself on the scenarios from the test set.11 We evaluate the chats with respect to language variation, effectiveness and strategy.
For language variation, we report the average utterance length Lu and the unigram entropy H in Table 4. Compared to Rule, the neural models tend to generate shorter utterances (Li et al., 2016b; Serban et al., 2017). However, they are more diverse; for example, questions are asked in multiple ways such as “Do you have ...”, “Any friends like ...”, “What about ...”.
At the discourse level, we expect the distribution of a bot’s utterance types to match the distribution of human’s. We show percentages of each utterance type in Table 4. For Rule, the decision about which action to take is written in the rules, while StanoNet and DynoNet learned to behave in a more human-like way, frequently informing and asking questions.
To measure effectiveness, we compute the success rate per turn (CT ) and per selection (CS). In Table 4, humans are the best at this game, followed by Rule which is comparable to DynoNet. Next, we investigate the strategies leading to these results.
An agent needs to decide which entity/attribute to check first to quickly reduce the search space. We hypothesize that humans tend to first focus on a majority entity and an attribute with fewer unique values (Section 2.3). We show the average frequency of first-mentioned entities (#Ent1) and the average number of unique values for first-mentioned attributes (|Attr1|) in Table 4.12 Clearly, both DynoNet and StanoNet successfully matched human’s starting strategy by favoring more frequent entities and attributes of smaller do-
11 We limit the number of turns in bot-bot chat to be the maximum number of turns humans took in the test set (46 turns).
12 Both numbers are normalized to [0, 1] with respect to all entities/attributes in the corresponding KB.
main size. To examine the overall strategy, we show the average number of attributes and entities mentioned during the conversation in Table 4. Humans and DynoNet strategically focus on a few attributes and entities, whereas Rule needs almost twice entities to achieve similar success rates. This suggests that the effectiveness of Rule mainly comes from large amounts of unselective information, which is consistent with feedback from their human partners.
Partner Evaluation. We generated 200 new scenarios and put up the bots on AMT using the same chat interface that used for data collection. Each AMT worker is randomly paired with Rule, StanoNet, DynoNet, or another human (but they don’t know which), and we make sure that all four types of agents are tested in each scenario (800 dialogues in total). At the end of each task, humans are asked to rate their partner in terms of fluency, correctness, cooperation, and humanlikeness from 1 (very bad) to 5 (very good), along with optional comments.
As shown in the example dialogues in Table 6, DynoNet cooperates smoothly with the human partner, e.g., replying with relevant information about morning/indoor friends when the partner mentioned that all her friends prefer morning and most like indoor. However, it “lied” when saying that it had a morning friend who likes outdoor. StanoNet starts well but doesn’t continue with the morning friend, because the morning node is not updated dynamically when mentioned by the partner. Rule always informs true facts but poorly follows the partner. In the comments, most complaint about Rule is that it was not ‘listening’ or ‘understanding’. We show the average ratings in Table 5 and the histograms in Appendix H. Overall, DynoNet achieves better partner satisfaction, especially in cooperation.
Third-party Evaluation. Since the evaluator cannot see the partner’s KB, judgment on correctness is a mere guess. Therefore, we also created a third-party evaluation task, where an independent AMT worker is shown a conversation and the KB of one of the agents; she is asked to rate the same aspects of the agent as in the partner eval-
13We only show the most frequent speech acts therefore the numbers do not sum to 1.
14For third-party evaluation, we first take mean of each question then average the ratings.
7
606
607
608
609
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
656
657
658
659
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
uation and provide justifications.15 The average ratings and histograms are shown in Table 5 and Appendix H respectively.
Surprisingly, there is a discrepancy between the two evaluation modes. Manual analysis of the comments indicates that while a participant in the dialogue considers the partner fluent as long as there is no communication overhead, a third person may have higher standards, e.g., thinking short sentences do not fully reveal an agent’s fluency. For human-likeness, partner evaluation is largely correlated with coherence (e.g., not repeating or ignoring past information) and task success, whereas third-party evaluators often rely on informality (e.g., usage of colloquia like “hiya”, capitalization, and abbreviation) or intuition.16 Interestingly, third-party evaluators noted most phenomena listed in Table 2 as indicators of humanbeings, e.g., correcting oneself, making chit-chat other than simply finishing the task.
4.3 Ablation Studies
Our model has two novel designs: entity abstraction and message passing for node embeddings.
15Each agent in a dialogue is rated by at least 5 people. 16 Nevertheless, informality can be easily handcrafted or
learned.
Table 7 shows what happens if we ablate these. When the number of message passing iterations, K, is reduced from 2 to 0, the loss consistently increases. Removing entity abstraction—meaning adding entity embeddings to node embeddings and the LSTM input embeddings—also degrades performance. This shows that DynoNet benefits from contextually-defined, structural node embeddings rather than ones based on a classic lookup table.
Model `
DynoNet (K = 2) 2.16 DynoNet (K = 1) 2.20 DynoNet (K = 0) 2.26 DynoNet - entity abstraction 2.21
Table 7: Ablations of our model on the dev set.
5 Discussion
There has been a recent surge of interest in endto-end task-oriented dialogue systems. Unfortunately, the progress has been limited by the size of available datasets (Serban et al., 2015a). Most work uses a small corpus (< 5K dialogues) through Wizard-of-Oz data collection (Williams et al., 2016; Asri et al., 2016) or simulators (Bordes and Weston, 2017; Li et al., 2016d), which
8
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
mainly focuses on information-query tasks. Current strategic dialogue datasets include Settlers of Catan (Afantenos et al., 2012) (2K turns) and the cards corpus (Potts, 2012) (1.3K dialogues), though neither has offered an end-to-end dialogue system.
Most task-oriented dialogue systems follow the POMDP-based approach (Young et al., 2013). Despite their success (Wen et al., 2017; Dhingra et al., 2016), the requirement for handcrafted slots limits their scalability to new domains and burdens data collection with extra state labeling. To break this limit, Bordes and Weston (2017) propose a Memory-Networks-based approach without domain-specific features. However, the memory is unstructured and interfacing with KBs relies on API calls, whereas our model embeds both the dialogue history and the KB structurally. Williams and Zweig (2016) use an LSTM to automatically infer the dialogue state, but as they focus on dialogue control rather than the full prob-
lem, the LSTM input contains only recognized entities, which restricts the representational power for richer utterances. Our network architecture is most similar to EntNet (Henaff et al., 2017), where memories are also updated by input sentences recurrently. The main difference is that our model allows information to be propagated between structured entities, which is shown to be crucial in our setting (Section 4.3).
In conclusion, we believe the symmetric collaborative dialogue setting and our dataset provide unique opportunities at the interface of traditional task-oriented dialogue and open-domain chat. We also offered DynoNet as a promising means for open-ended dialogue state representation. Looking forward, our dataset facilitates the study of pragmatics and human strategies in dialogue—a good stepping stone towards learning more complex dialogues such as negotiation. It would be interesting to integrate supervised learning with game-theoretic approaches in future.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

This paper proposes a method for building dialogue agents involved in a symmetric collaborative task, in which the agents need to strategically communicate to achieve a common goal.    I do like this paper.  I am very interested in how much data-driven techniques can be used for dialogue management.  However, I am concerned that the approach that this paper proposes, is actually not specific to symmetric collaborative tasks, but to tasks that can be represented as graph operations, such as finding an intersection between objects that the two people know about.  In Section 2.1, the authors introduce symmetric collaborative dialogue setting.  However, such dialogs have been studied before, such as Clark and Wilkes-Gibbs explored (Cognition '86), and Walker's furniture layout task (Journal of Artificial Research '00).  On line 229, the authors say that this domain is too rich for slot-value semantics.  However, their domain is based on attribute value pairs, so their domain could use a semantics represenation based on attribute value-pairs, such as first order logic.  Section 3.2 is hard to follow.        The authors often refer to Figure 2, but I didn't find this example that helpful.        For example, for section 3.1, at what point of the dialogue does this represent?  Is this the same after `anyone went to columbia?'