Exploiting Argument Information to Improve Event Detection via Supervised Attention Mechanisms
This paper tackles the task of event detection (ED), which involves identifying and categorizing events. We argue that arguments provide significant clues to this task, but they are either completely ignored or exploited in an indirect manner in existing detection approaches. In this work, we propose to exploit argument information explicitly for ED via supervised attention mechanisms. In specific, we systematically investigate the proposed model under the supervision of different attention strategies. Experimental results show that our approach advances state-ofthe-arts and achieves the best F1 score on ACE 2005 dataset.
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033
034
035
036
037
038
039
040
041
042
043
044
045
046
047
048
049
062
063
064
065
066
067
068
069
070
071
072
073
074
075
076
077
078
079
080
081
082
083
084
085
086
087
088
089
090
091
092
093
094
095
096
097
098
099
This paper tackles the task of event detection (ED), which involves identifying and categorizing events. We argue that arguments provide significant clues to this task, but they are either completely ignored or exploited in an indirect manner in existing detection approaches. In this work, we propose to exploit argument information explicitly for ED via supervised attention mechanisms. In specific, we systematically investigate the proposed model under the supervision of different attention strategies. Experimental results show that our approach advances state-ofthe-arts and achieves the best F1 score on ACE 2005 dataset.
1 Introduction
In the ACE (Automatic Context Extraction) event extraction program, an event is represented as a structure comprising an event trigger and a set of arguments. This work tackles event detection (ED) task, which is a crucial part of event extraction (EE) and focuses on identifying event triggers and categorizing them. For instance, in the sentence “He died in the hospital”, an ED system is expected to detect a Die event along with the trigger word “died”. Besides, the task of EE also includes event argument extraction (AE), which involves event argument identification and role classification. In the above sentence, the arguments of the event include “He”(Role = Person) and “hospital”(Role = Place). However, this paper does not focus on AE and only tackles the former task.
According to the above definitions, event arguments seem to be not essentially necessary to ED. However, we argue that they are capable of providing significant clues for identifying and categorizing events. They are especially useful for ambiguous trigger words. For example, consider a sentence in ACE 2005 dataset:
Mohamad fired Anwar, his former protege, in 1998.
In this sentence, “fired” is the trigger word and the other bold words are event arguments. The correct type of the event triggered by “fired” in this case is End-Position . However, it might be easily misidentified as Attack because “fired” is a multivocal word. In this case, if we consider the phrase “former protege”, which serves as an argument (Role = Position) of the target event, we would have more confidence in predicting it as an End-Position event.
Unfortunately, most exiting methods performed event detection individually, where the annotated arguments in training set are totally ignored (Ji and Grishman, 2008; Gupta and Ji, 2009; Hong et al., 2011; Chen et al., 2015; Nguyen and Grishman, 2015; Liu et al., 2016a,b; Nguyen and Grishman, 2016). Although some joint learning based methods have been proposed, which tackled event detection and argument extraction simultaneously (Riedel et al., 2009; Li et al., 2013; Venugopal et al., 2014; Nguyen et al., 2016), these approaches usually only make remarkable improvements to AE, but insignificant to ED. Table 1 illustrates our observations. Li et al. (2013) and Nguyen et al. (2016) are state-of-the-art joint models in symbolic and embedding methods for event extraction, respectively. Compared with state-of-the-art pipeline systems, both join-
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
Methods ED AE
Symbolic Hong’s pipeline (2011) 68.3 48.3
Methods Li’s joint (2013) 67.5 52.7
Embedding Chen’s pipeline (2015) 69.1 53.5
Methods Nguyen’s joint (2016) 69.3 55.4
Table 1: Performances of pipeline and joint approaches on ACE 2005 dataset. The pipeline method in each group was the state-of-the-art system when the corresponding joint method was proposed.
t methods achieved remarkable improvements on AE (over 1.9 points), whereas achieved insignificant improvements on ED (less than 0.2 points). The symbolic joint method even performed worse (67.5 vs. 68.3) than pipeline system on ED.
We believe that this phenomenon may be caused by the following two reasons. On the one hand, since joint methods simultaneously solve ED and AE, methods following this paradigm usually combine the loss functions of these two tasks and are jointly trained under the supervision of annotated triggers and arguments. However, training corpus contains much more annotated arguments than triggers (about 9800 arguments and 5300 triggers in ACE 2005 dataset) because each trigger may be along with multiple event arguments. Thus, the unbalanced data may cause joint models to favor AE task. On the other hand, in implementation, joint models usually pre-predict several potential triggers and arguments first and then make global inference to select correct items. When pre-predicting potential triggers, almost all existing approaches do not leverage any argument information. In this way, ED does hardly benefit from the annotated arguments. By contrast, the component for pre-prediction of arguments always exploits the extracted trigger information. Thus, we argue that annotated arguments are actually used for AE, not for ED in existing joint methods, which is also the reason we call it an indirect way to use arguments for ED.
Contrast to joint methods, this paper proposes to exploit argument information explicitly for ED. We have analyzed that arguments are capable of providing significant clues to ED, which gives us an enlightenment that ar-
guments should be focused on when performing this task. Therefore, we propose a neural network based approach to detect events in texts. And in the proposed approach, we adopt a supervised attention mechanism to achieve this goal, where argument words are expected to acquire more attention than other words. The attention value of each word in a given sentence is calculated by an operation between the current word and the target trigger candidate. Specifically, in training procedure, we first construct gold attentions for each trigger candidate based on annotated arguments. Then, treating gold attentions as the supervision to train the attention mechanism, we learn attention and event detector jointly both in supervised manner. In testing procedure, we use the ED model with learned attention mechanisms to detect events.
In the experiment section, we systematically conduct comparisons on a widely used benchmark dataset ACE20051. In order to further demonstrate the effectiveness of our approach, we also use events from FrameNet (FN) (F. Baker et al., 1998) as extra training data, as the same as Liu et al. (2016a) to alleviate the data-sparseness problem for ED to augment the performance of the proposed approach. The experimental results demonstrate that the proposed approach is effective for ED task, and it outperforms state-of-the-art approaches with remarkable gains.
To sum up, our main contributions are: (1) we analyze the problem of joint models on the task of ED, and propose to use the annotated argument information explicitly for this task. (2) to achieve this goal, we introduce a supervised attention based ED model. Furthermore, we systematically investigate different attention strategies for the proposed model. (3) we improve the performance of ED and achieve the best performance on the widely used benchmark dataset ACE 2005.
2 Task Description
The ED task is a subtask of ACE event evaluations where an event is defined as a specific occurrence involving one or more participants. Event extraction task requires certain specified types of events, which are mentioned
1https://catalog.ldc.upenn.edu/LDC2006T06
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
in the source language data, be detected. We firstly introduce some ACE terminologies to facilitate the understanding of this task:
Entity: an object or a set of objects in one of the semantic categories of interests.
Entity mention: a reference to an entity (typically, a noun phrase).
Event trigger: the main word that most clearly expresses an event occurrence.
Event arguments: the mentions that are involved in an event (participants).
Event mention: a phrase or sentence within which an event is described, including the trigger and arguments.
The goal of ED is to identify event triggers and categorize their event types. For instance, in the sentence “He died in the hospital”, an ED system is expected to detect a Die event along with the trigger word “died”. The detection of event arguments “He”(Role = Person) and “hospital”(Role = Place) is not involved in the ED task. The 2005 ACE evaluation included 8 super types of events, with 33 subtypes. Following previous work, we treat these simply as 33 separate event types and ignore the hierarchical structure among them.
3 The Proposed Approach
Similar to existing work, we model ED as a multi-class classification task. In detail, given a sentence, we treat every token in that sentence as a trigger candidate, and our goal is to classify each of these candidates into one of 34 classes (33 event types plus an NA class).
In our approach, every word along with its context, which includes the contextual words and entities, constitute an event trigger candidate. Figure 1 describes the architecture of the proposed approach, which involves two components: (i) Context Representation Learning (CRL), which reveals the representation of both contextual words and entities via attention mechanisms; (ii) Event Detector (ED), which assigns an event type (including the NA type) to each candidate based on the learned contextual representations.
3.1 Context Representation Learning
In order to prepare for Context Representation Learning (CRL), we limit the context to a fixed length by trimming longer sen-
tences and padding shorter sentences with a special token when necessary. Let n be the fixed length and w0 be the current candidate trigger word, then its contextual words Cw is [w−n
2 , w−n 2 +1, ..., w−1, w1, ..., wn 2 −1, wn 2 ]2, and its contextual entities, which is the corresponding entity types (including an NA type) of Cw, is [e−n
2 , e−n 2 +1, ..., e−1, e1, ..., en 2 −1, en 2 ].
For convenience, we use w to denote the current word, [w1, w2, ..., wn] to denote the contextual words Cw and [e1, e2, ..., en] to denote the contextual entities Ce in figure 1. Note that, both w, Cw and Ce mentioned above are originally in symbolic representation. Before entering CRL component, we transform them into real-valued vector by looking up word embedding table and entity type embedding table. Then we calculate attention vectors for both contextual words and entities by performing operations between the current word w and its contexts. Finally, the contextual words representation cw and contextual entities representation ce are formed by the weighted sum of the corresponding embeddings of each word and entity in Cw and Ce, respectively. We will give the details in the fol-
2The current candidate trigger word w0 is not included in the context.
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
lowing subsections.
3.1.1 Word Embedding Table
Word embeddings learned from a large amount of unlabeled data have been shown to be able to capture the meaningful semantic regularities of words (Bengio et al., 2003; Erhan et al., 2010). This paper uses the learned word embeddings as the source of basic features. Specifically, we use the Skip-gram model (Mikolov et al., 2013) to learn word embeddings on the NYT corpus3.
3.1.2 Entity Type Embedding Table
The ACE 2005 corpus annotated not only events but also entities for each given sentence. Following existing work (Li et al., 2013; Chen et al., 2015; Nguyen and Grishman, 2015), we exploit the annotated entity information in our ED system. We randomly initialize embedding vector for each entity type (including the NA type) and update it in training procedure.
3.1.3 Representation Learning
In this subsection, we illustrate our proposed approach to learn representations of both contextual words and entities, which serve as inputs to the following event detector component. Recall that, we use the matrix Cw and Ce to denote contextual words and contextual entities, respectively.
As illustrated in figure 1, the CRL component needs three inputs: the current candidate trigger word w, the contextual words Cw and the contextual entities Ce. Then, two attention vectors, which reflect different aspects of the context, are calculated in the next step. The contextual word attention vector αw is computed based on the current word w and its contextual words Cw. We firstly transform each word wk (including w and every word in Cw) into a hidden representation wk by the following equation:
wk = f(wk Ww) (1)
where f(·) is a non-linear function such as the hyperbolic tangent, and Ww is the transformation matrix. Then, we use the hidden representations to compute the attention value for each
3https://catalog.ldc.upenn.edu/LDC2008T19
word in Cw:
αkw = exp(w wTk )∑ i exp(w w T i )
(2)
The contextual entity attention vector αe is calculated with a similar method to αw.
αke = exp(we eTk )∑ i exp(we e T i )
(3)
Note that, we do not use the entity information of the current candidate token to compute the attention vector. The reason is that only a small percentage of true event triggers are entities4. Therefore, the entity type of a candidate trigger is meaningless for ED. Instead, we use we, which is calculated by transforming w from the word space into the entity type space, as the attention source.
We combine αw and αe to obtain the final attention vector, α = αw+αe. Finally, the contextual words representation cw and the contextual entities representation ce are formed by weighted sum of Cw and Ce, respectively:
cw = Cwα T (4)
ce = Ceα T (5)
3.2 Event Detector
As illustrated in figure 1, we employ a threelayer (an input layer, a hidden layer and a softmax output layer) Artificial Neural Networks (ANNs) (Hagan et al., 1996) to model the ED task, which has been demonstrated very effective for event detection by Liu et al. (2016a).
3.2.1 Basic ED Model
Given a sentence, as illustrated in figure 1, we concatenate the embedding vectors of the context (including contextual words and entities) and the current candidate trigger to serve as the input to ED model. Then, for a given input sample x, ANN with parameter θ outputs a vector O, where the i-th value oi of O is the confident score for classifying x to the i-th event type. To obtain the conditional probability p(i|x, θ), we apply a softmax operation over all event types:
p(i|x, θ) = e oi∑m
k=1 e ok
(6)
4Only 10% of triggers in ACE 2005 are entities.
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Given all of our (suppose T) training instances (x(i); y(i)), we can then define the negative loglikelihood loss function:
J(θ) = − T∑ i=1 log p(y(i)|x(i), θ) (7)
We train the model by using a simple optimization technique called stochastic gradient descent (SGD) over shuffled mini-batches with the Adadelta rule (Zeiler, 2012). Regularization is implemented by a dropout (Kim, 2014; Hinton et al., 2012) and L2 norm.
3.2.2 Supervised Attention
In this subsection, we introduce supervised attention to explicitly use annotated argument information to improve ED. Our basic idea is simple: argument words should acquire more attention than other words. To achieve this goal, we first construct vectors using annotated arguments as the gold attentions. Then, we employ them as supervision to train the attention mechanism.
Constructing Gold Attention Vectors
Our goal is to encourage argument words to obtain more attention than other words. To achieve this goal, we propose two strategies to construct gold attention vectors:
S1: only pay attention to argument
words. That is, all argument words in the given context obtain the same attention, whereas other words get no attention. For candidates without any annotated arguments in context, we force all entities to average the whole attention. Figure 2 illustrates the details, where α∗ is the final gold attention vector.
Figure 2: An example of S1 to construct gold attention vector. The word fired is the trigger candidate, and underline words are arguments of fired annotated in the corpus.
S2: pay attention to both arguments
and the words around them. The assumption is that, not only arguments are important
to ED, the words around them are also helpful. And the nearer a word is to arguments, the more attention it should obtain. Inspired by Mi et al. (2016), we use a gaussian distribution g(·) to model the attention distribution of words around arguments. In detail, given an instance, we first obtain the raw attention vector α in the same manner as S1 (see figure 2). Then, we create a new vector α ′ with all points initialized with zero, and for each αi = 1, we update α ′ by adding g(d, µ, σ) with window size w, where d is the distance from a word to the argument (positioned at i) and µ, σ are hyper-parameters of the gaussian distribution. Finally, similar to S1, we normalize α ′ to obtain the target attention vector α∗.
Jointly Training ED and Attention Given the gold attention α∗ (see subsection 3.2.2) and the machine attention α produced by our model (see subsection 3.1.3), we employ the square error as the loss function of attentions:
D(θ) = T∑ i=1 n∑ j=1 (α∗ij − αij)2 (8)
Combining equation 7 and equation 8, we define the joint loss function of our proposed model as follows:
J ′ (θ) = J(θ) + λD(θ) (9)
where λ is a hyper-parameter for trade-off between J and D. Similar to basic ED model, we minimize the loss function J ′ (θ) by using SGD over shuffled mini-batches with the Adadelta update rule.
4 Experiments
4.1 Dataset and Experimental Setup
Dataset
We conducted experiments on ACE 2005 dataset. For the purpose of comparison, we followed the evaluation of (Li et al., 2013; Chen et al., 2015; Liu et al., 2016b): randomly selected 30 articles from different genres as the development set, and subsequently conducted a blind test on a separate set of 40 ACE 2005 newswire documents. We used the remaining 529 articles as our training set.
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Hyper-parameter Setting
Hyper-parameters are tuned on the development dataset. We set the dimension of word embeddings to 200, the dimension of entity type embeddings to 50, the size of hidden layer to 300, the output size of word transformation matrix Ww in equation 1 to 200, the batch size to 100, the hyper-parameter for the L2 norm to 10
−6 and the dropout rate to 0.6. In addition, we use the standard normal distribution to model attention distributions of words around arguments, which means that µ = 0.0, σ = 1.0, and the window size is set to 3 (see Subsection 3.2.2). The hyper-parameter λ in equation 9 is various for different attention strategies, we will give its setting in the next section.
4.2 Correctness of Our Assumption
In this section, we conduct experiments on ACE 2005 corpus to demonstrate the correctness of our assumption that argument information is crucial to ED. To achieve this goal, we design a series of systems for comparison.
ANN is the basic event detection model, in which the hyper-parameter λ is set to 0. This system does not employ argument information and computes attentions without supervision (see Subsection 3.1.3).
ANN-ENT assigns λ with 0, too. The difference is that it constructs the attention vector α by forcing all entities in the context to average the attention instead of computing it in the manner introduced in Subsection 3.1.3. Since all arguments are entities, this system is designed to investigate the effects of entities.
ANN-Gold1 uses the gold attentions constructed by strategy S1 in both training and testing procedure.
ANN-Gold2 is akin to ANN-Gold1, but uses the second strategy to construct its gold attentions.
Note that, in order to avoid the interference of attention mechanisms, the last two systems are designed to use argument information (via gold attentions) in both training and testing procedure.
Table 2 compares these systems on ACE 2005 corpus. From the table, we observe that systems with argument information (the last two systems) significantly outperform system-
s without argument information (the first two systems), which demonstrates that argument information is very useful for this task. Moreover, since all arguments are entities, for preciseness we also investigate that whether ANN-Gold1/2 on earth benefits from entities or arguments. Compared with ANN-ENT (revising that this system only uses entity information), ANN-Gold1/2 performs much better, which illustrates that entity information is not enough and further demonstrates that argument information is necessary for ED.
4.3 Results on ACE 2005 Corpus
In this section, we conduct experiments on ACE 2005 corpus to demonstrate the effectiveness of the proposed approach. Firstly, we introduce systems implemented in this work.
ANN-S1 uses gold attentions constructed by strategy S1 as supervision to learn attention. In our experiments, λ is set to 1.0.
ANN-S2 is akin to ANN-S1, but use strategy S2 to construct gold attentions and the hyper-parameter λ is set to 5.0.
These two systems both employ supervised attention mechanisms. For comparison, we use an unsupervised-attention system ANN as our baseline, which is introduced in Subsection 4.2. In addition, we select the following state-ofthe-art methods for comparison.
1). Li’s joint model (Li et al., 2013) extracts events based on structure prediction. It is the best structure-based system.
2). Liu’s PSL (Liu et al., 2016b) employs both latent local and global information for event detection. It is the best-reported featurebased system.
3). Liu’s FN-Based approach (Liu et al., 2016a) leverages the annotated corpus of FrameNet to alleviate data sparseness problem of ED based on the observation that frames in
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Methods P R F1 Li’s joint model (2013) 73.7 62.3 67.5
Liu’s PSL (2016) 75.3 64.4 69.4 Liu’s FN-Based (2016) 77.6 65.2 70.7
Ngyuen’s joint (2016) 66.0 73.0 69.3
Skip-CNN (2016) N/A 71.3
ANN 73.2 57.9 64.6 ANN-S1† 81.4 62.4 70.8 ANN-S2† 78.0 66.3 71.7
Table 3: Experimental results on ACE 2005. The first group illustrates the performances of state-of-the-art approaches. The second group illustrates the performances of the proposed approach. † designates the systems that employ arguments information.
FN are analogous to events in ACE.
4). Ngyen’s joint model (Nguyen et al., 2016) employs a bi-directional RNN to jointly extract event triggers and arguments. It is the best-reported representation-based joint approach proposed on this task.
5). Skip-CNN (Nguyen and Grishman, 2016) introduces the non-consecutive convolution to capture non-consecutive k-grams for event detection. It is the best reported representation-based approach on this task.
Table 3 presents the experimental results on ACE 2005 corpus. From the table, we make the following observations:
1). ANN performs unexpectedly poorly, which indicates that unsupervised-attention mechanisms do not work well for ED. We believe the reason is that the training data of ACE 2005 corpus is insufficient to train a precise attention in an unsupervised manner, considering that data sparseness is an important issue of ED (Zhu et al., 2014; Liu et al., 2016a).
2). With argument information employed via supervised attention mechanisms, both ANN-S1 and ANN-S2 outperform ANN with remarkable gains, which illustrates the effectiveness of the proposed approach.
3). ANN-S2 outperforms ANN-S1, but the latter achieves higher precision. It is not difficult to understand. On the one hand, strategy S1 only focuses on argument words, which provides accurate information to identify event type, thus ANN-S1 could achieve higher precision. On the other hand, S2 focus-
es on both arguments and words around them, which provides more general but noised clues. Thus, ANN-S2 achieves higher recall with a little loss of precision.
4). Compared with state-of-the-art approaches, our method ANN-S2 achieves the best performance. We also perform a t-test (p 6 0.05), which indicates that our method significantly outperforms all of the compared methods. Furthermore, another noticeable advantage of our approach is that it achieves much higher precision than state-of-the-arts.
4.4 Augmentation with FrameNet
Recently, Liu et al. (2016a) used events automatically detected from FN as extra training data to alleviate the data-sparseness problem for event detection. To further demonstrate the effectiveness of the proposed approach, we also use the events from FN to augment the performance of our approach.
In this work, we use the events published by Liu et al. (2016a)5 as extra training data. However, their data can not be used in the proposed approach without further processing, because it lacks of both argument and entity information. Figure 3 shows several examples of this data.
Processing of Events from FN
Liu et al. (2016a) detected events from FrameNet based on the observation that frames in FN are analogous to events in ACE (lexical unit of a frame ↔ trigger of an event, frame elements of a frame ↔ arguments of an event). All events they published are also frames in FN. Thus, we treat frame elements annotated in FN corpus as event arguments. Since frames generally contain more frame elements than events, we only use core6 elements
5https://github.com/subacl/acl16 6FrameNet classifies frame elements into three
groups: core, peripheral and extra-thematic.
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
in this work. Moreover, to obtain entity information, we use RPI Joint Information Extraction System7 (Li et al., 2013, 2014; Li and Ji, 2014) to label ACE entity mentions.
Experimental Results
We use the events from FN as extra training data and keep the development and test datasets unchanged.Table 4 presents the experimental results.
Methods P R F1 ANN 73.2 57.9 64.6
ANN-S1 81.4 62.4 70.8 ANN-S2 78.0 66.3 71.7
ANN +FrameNet 75.1 59.2 66.2 ANN-S1 +FrameNet 80.1 63.6 70.9 ANN-S2 +FrameNet 76.8 67.5 71.9
Table 4: Experimental results on ACE 2005 corpus. “+FrameNet” designates the systems that are augmented by events from FrameNet.
From the results, we observe that: 1). With extra training data, ANN achieves significant improvements on F1 measure (66.2 vs. 64.6). This result, to some extent, demonstrates the correctness of our assumption that the data sparseness problem is the reason that causes unsupervised attention mechanisms to be ineffective to ED.
2). Augmented with external data, both ANN-S1 and ANN-S2 achieve higher recall with a little loss of precision. This is to be expected. On the one hand, more positive training samples consequently make higher recall. On the other hand, the extra event samples are automatically extracted from FN, thus false-positive samples are inevitable to be involved, which may result in hurting the precision. Anyhow, with events from FN, our approach achieves higher F1 score.
5 Related Work
Event detection is an increasingly hot and challenging research topic in NLP. Generally, existing approaches could roughly be divided into two groups.
The first kind of approach tackled this task under the supervision of annotated triggers and entities, but totally ignored anno-
7http://nlp.cs.rpi.edu/software/
tated arguments. The majority of existing work followed this paradigm, which includes feature-based methods and representationbased methods. Feature-based methods exploited a diverse set of strategies to convert classification clues (i.e., POS tags, dependency relations) into feature vectors (Ahn, 2006; Ji and Grishman, 2008; Patwardhan and Riloff, 2009; Gupta and Ji, 2009; Liao and Grishman, 2010; Hong et al., 2011; Liu et al., 2016b). Representation-based methods typically represent candidate event mentions by embeddings and feed them into neural networks (Chen et al., 2015; Nguyen and Grishman, 2015; Liu et al., 2016a; Nguyen and Grishman, 2016).
The second kind of approach, on the contrast, tackled event detection and argument extraction simultaneously, which is called joint approach (Riedel et al., 2009; Poon and Vanderwende, 2010; Li et al., 2013, 2014; Venugopal et al., 2014; Nguyen et al., 2016). Joint approach is proposed to capture internal and external dependencies of events, including trigger-trigger, argument-argument and trigger-argument dependencies. Theoretically, both ED and AE are expected to benefit from joint methods because triggers and arguments are jointly considered. However, in practice, existing joint methods usually only make remarkable improvements to AE, but insignificant to ED. Different from them, this work investigates the exploitation of argument information to improve the performance of ED.
6 Conclusions
In this work, we propose a novel approach to model argument information explicitly for ED via supervised attention mechanisms. Besides, we also investigate two strategies to construct gold attentions using the annotated arguments. To demonstrate the effectiveness of the proposed method, we systematically conduct a series of experiments on the widely used benchmark dataset ACE 2005. Moreover, we also use events from FN to augment the performance of the proposed approach. Experimental results show that our approach outperforms state-of-the-art methods, which demonstrates that the proposed approach is effective for event detection.
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths: This paper tries to use the information from arguments, which is usually ignored yet actually quite important, to improve the performance of event detection. The framework is clear and simple. With the help of the supervised attention mechanism, an important method that has been used in many tasks such as machine translation, the performance of their system outperforms the baseline significantly.  - Weaknesses:  The attention vector is simply the summation of two attention vectors of each part. Maybe the attention vector could be calculated in a more appropriate approach. For the supervised attention mechanism, two strategies are proposed. Both of them are quite straightforward. Some more complicated strategies can work better and can be tried.  - General Discussion:  Although there are some places that can be improved, this paper proposed a quite effective framework, and the performance is good. The experiment is solid. It can be considered to be accepted.


Tandem Anchoring: a Multiword Anchor Approach for Interactive Topic Modeling
Interactive topic models are powerful tools for those seeking to understand large collections of text. However, existing sampling-based interactive topic modeling approaches scale poorly to large data sets. Anchor methods, which use a single word to uniquely identify a topic, offer the speed needed for interactive work but lack both a mechanism to inject prior knowledge and lack the intuitive semantics needed for user-facing applications. We propose combinations of words as anchors, going beyond existing single word anchor algorithms—an approach we call “Tandem Anchors”. We begin with a synthetic investigation of this approach then apply the approach to interactive topic modeling in a user study and compare it to interactive and non-interactive approaches. Tandem anchors are faster and more intuitive than existing interactive approaches. Topic models distill large collections of text into topics, giving a high-level summary of the thematic structure of the data without manual annotation. In addition to facilitating discovery of topical trends (Gardner et al., 2010), topic modeling is used for a wide variety of problems including document classification (Rubin et al., 2012), information retrieval (Wei and Croft, 2006), author identification (Rosen-Zvi et al., 2004), and sentiment analysis (Titov and McDonald, 2008). However, the most compelling use of topic models is to help users understand large datasets (Chuang et al., 2012). Interactive topic modeling (Hu et al., 2014) allows non-experts to refine automatically generated topics, making topic models less of a “take it or leave it” proposition. Including humans input during training improves the quality of the model and allows users to guide topics in a specific way, custom tailoring the model for a specific downstream task or analysis. The downside is that interactive topic modeling is slow—algorithms typically scale with the size of the corpus—and requires non-intuitive information from the user in the form of must-link and cannot-link constraints (Andrzejewski et al., 2009). We address these shortcomings of interactive topic modeling by using an interactive version of the anchor words algorithm for topic models. The anchor algorithm (Arora et al., 2013) is an alternative topic modeling algorithm which scales with the number of unique word types in the data rather than the number of documents or tokens (Section 1). This makes the anchor algorithm fast enough for interactive use, even in web-scale document collections. A drawback of the anchor method is that anchor words—words that have high probability of being in a single topic—are not intuitive. We extend the anchor algorithm to use multiple anchor words in tandem (Section 2). Tandem anchors not only improve interactive refinement, but also make the underlying anchor-based method more intuitive. For interactive topic modeling, tandem anchors produce higher quality topics than single word anchors (Section 3). Tandem anchors provide a framework for fast interactive topic modeling: users improve and refine an existing model through multiword anchors (Section 4). Compared to existing methods such as Interactive Topic Models (Hu et al., 2014), our method is much faster.
Topic models distill large collections of text into topics, giving a high-level summary of the thematic structure of the data without manual annotation. In addition to facilitating discovery of topical trends (Gardner et al., 2010), topic modeling is used for a wide variety of problems including document classification (Rubin et al., 2012), information retrieval (Wei and Croft, 2006), author identification (Rosen-Zvi et al., 2004), and sentiment analysis (Titov and McDonald, 2008). However, the most compelling use of topic models is to help users understand large datasets (Chuang et al., 2012).
Interactive topic modeling (Hu et al., 2014) allows non-experts to refine automatically generated
topics, making topic models less of a “take it or leave it” proposition. Including humans input during training improves the quality of the model and allows users to guide topics in a specific way, custom tailoring the model for a specific downstream task or analysis.
The downside is that interactive topic modeling is slow—algorithms typically scale with the size of the corpus—and requires non-intuitive information from the user in the form of must-link and cannot-link constraints (Andrzejewski et al., 2009). We address these shortcomings of interactive topic modeling by using an interactive version of the anchor words algorithm for topic models.
The anchor algorithm (Arora et al., 2013) is an alternative topic modeling algorithm which scales with the number of unique word types in the data rather than the number of documents or tokens (Section 1). This makes the anchor algorithm fast enough for interactive use, even in web-scale document collections.
A drawback of the anchor method is that anchor words—words that have high probability of being in a single topic—are not intuitive. We extend the anchor algorithm to use multiple anchor words in tandem (Section 2). Tandem anchors not only improve interactive refinement, but also make the underlying anchor-based method more intuitive.
For interactive topic modeling, tandem anchors produce higher quality topics than single word anchors (Section 3). Tandem anchors provide a framework for fast interactive topic modeling: users improve and refine an existing model through multiword anchors (Section 4). Compared to existing methods such as Interactive Topic Models (Hu et al., 2014), our method is much faster.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
1 Vanilla Anchor Algorithm
The anchor algorithm computes the topic matrix A, where Av,k is the conditional probability of observing word v given topic k, e.g., the probability of seeing the word “lens” given the camera topic in a corpus of Amazon product reviews. Arora et al. (2012) finds these probabilities by assuming that every topic contains at least one ‘anchor’ word which has a non-zero probability only in that topic. Anchor words make computing the topic matrix A tractable because the occurrence pattern of the anchor word mirrors the occurrence pattern of the topic itself.
To recover the topic matrix A using anchor words, we first compute a V × V cooccurrence matrix Q, where Qi,j is the conditional probability p(wj |wi) of seeing word type wj after having seen wi in the same document. A form of the Gram-Schmidt process on Q finds anchor words {g1 . . . gk} (Arora et al., 2013).
Once we have the set of anchor words, we can compute the probability of a topic given a word (the inverse of the conditioning in A). This coefficient matrix C is defined row-wise for each word i
C∗i,· = argmin Ci,· DKL
( Qi,· ∥∥∥∥ K∑ k=1 Ci,kQgk,· ) ,
(1) which gives the best reconstruction (based on Kullback-Leibler divergence DKL) of non-anchor words given anchor words’ conditional probabilities. For example, in our product review data, a word such as “battery” is a convex combination of the anchor words’ contexts (Qgk,·) such as “camera”, “phone”, and “car”. Solving each row of C is fast and is embarrassingly parallel. Finally, we apply Bayes’ rule to recover the topic matrix A from the coefficient matrix C.
The anchor algorithm can be orders of magnitude faster than probabilistic inference. The construction of Q requires only a single pass over the data and can be pre-computed for interactive use-cases. Once Q is constructed, topic inference scales with the size of Q which, in turn, depends on the square of the vocabulary size V . In contrast, traditional topic model inference typically requires multiple passes over the entire data. Techniques such as Online LDA (Hoffman et al., 2010) or Stochastic Variation Inference (Hoffman et al., 2013) could improve this to a single pass over the
Anchor Top Words in Topics backpack backpack camera lens bag room carry fit cameras equipment comfortable camera camera lens pictures canon digital lenses batteries filter mm photos bag bag camera diaper lens bags genie smell
room diapers odor
Table 1: Three separate attempts to construct a topic concerning camera bags in Amazon product reviews with single word anchors. The term “backpack” is a good anchor because it uniquely identifies the topic. However, both “camera” and “bag” are poor anchors for this topic.
entire data. However, even if such techniques were to be adapted to incorporate human guidance, a single pass is not tractable for interactive use.
2 Tandem Anchor Extension
Single word anchors can be opaque to users. For an example of bewildering anchor words, consider a camera bag topic from a collection of Amazon product reviews (Table 1). The anchor word “backpack” may seem strange. However, this dataset contains nothing about regular backpacks; thus, “backpack” is unique to camera bags. Bizarre, low-to-mid frequency words are often anchors because anchor words must be unique to a topic; intuitive or high-frequency words cannot be anchors if they have probability in any other topic.
If we instead ask users to give us representative words for this topic, we would expect combinations of words like “camera” and “bag.” However, with single word anchors we must choose a single word to anchor each topic. Unfortunately, because these words might appear in multiple topics, individually they are not suitable as anchor words. The anchor word “camera” generates a general camera topic instead of camera bags, and the topic anchored by “bag” includes bags for diaper pails (Table 1).
Instead, we need to use sets of representative terms as an interpretable, parsimonious description of a topic. This section discusses strategies to build anchors from multiple words and the implications of using multiword anchors to recover topics. This extension not only makes anchors more interpretable but also enables users to manually construct effective anchors in interactive topic modeling settings.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
2.1 Anchor Facets
We first need to turn words into an anchor. If we interpret the anchor algorithm geometrically, each row of Q represents a word as a point in V -dimensional space. We then model each point as a convex combination of anchor words to reconstruct the topic matrix A (Equation 1). Instead of individual anchor words (one anchor word per topic), we use anchor facets, or sets of words that describe a topic. The facets for each anchor form a new pseudoword, or an invented point in V -dimensional space (described in more detail in Section 2.2).
While these new points do not correspond to words in the vocabulary, we can express nonanchor words as convex combinations of pseudowords. To construct these pseudowords from their facets, we combine the co-occurrence profiles of the facets. These pseudowords then augment the original cooccurrence matrix Q with K additional rows corresponding to synthetic pseudowords forming each of K multiword anchors. We refer to this augmented matrix as S. The rest of the anchor algorithm proceeds unmodified.
Our augmented matrix S is therefore a (V + K) × V matrix. As before, V is the number of token types in the data and K is the number of topics. The first V rows of S correspond to the V token types observed in the data, while the additionalK rows correspond to the pseudowords constructed from anchor facets. Each entry of S encodes conditional probabilities so that Si,j is equal to p(wi |wj). For the additionalK rows, we invent a cooccurrence pattern that can effectively explain the other words’ conditional probabilities.
2.2 Combining Facets into Pseudowords
We now describe more concretely how to combine an anchor facets to describe the cooccurrence pattern of our new pseudoword anchor. In tandem anchors, we create vector representations that combine the information from anchor facets. Our anchor facets are G1 . . .GK , where Gk is a set of anchor facets which will form the kth pseudoword anchor. The pseudowords are g1 . . . gK , where gk is the pseudoword from Gk. These pseudowords form the new rows of S. We give several candidates for combining anchors facets into a single multiword anchor; we compare their performance in Section 3.
Vector Average An obvious function for com-
puting the central tendency is the vector average. For each anchor facet,
Sgk,j = ∑ i∈Gk Si,j |Gk| , (2)
where |Gk| is the cardinality of Gk. Vector average makes the pseudoword Sgk,j more central, which is intuitive but inconsistent with the interpretation from Arora et al. (2013) that anchors should be extreme points whose linear combinations explain more central words.
Or-operator An alternative approach is to consider a cooccurrence with any anchor facet in Gk. For word j, we use De Morgan’s laws to set
Sgk,j = 1− ∏ i∈Gk (1− Si,j). (3)
Unlike the average, which pulls the pseudoword inward, this or-operator pushes the word outward, increasing each of the dimensions. Increasing the volume of the simplex spanned by the anchors explains more words.
Element-wise Min Vector average and oroperator are both sensitive to outliers and cannot account for polysemous anchor facets. Returning to our previous example, both “camera” and “bag” are bad anchors for camera bags because they appear in documents discussing other products. However, if both “camera” and “bag” are anchor facets, we can look at an intersection of their contexts: words that appear with both. Using the intersection, the cooccurrence pattern of our anchor facet will only include terms relevant to camera bags.
Mathematically, this is an element-wise min operator,
Sgk,j = min i∈Gk Si,j . (4)
This construction, while perhaps not as simple as the previous two, is robust to words which have cooccurrences which are not unique to a single topic.
Harmonic Mean Leveraging the intuition that we should use a combination function which is both centralizing (like vector average) and ignores large outliers (like element-wise min), the final combination function is the element-wise harmonic mean. Thus, for each anchor facet
Sgk,j = ∑ i∈Gk ( S−1i,j |Gk| )−1 . (5)
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Since the harmonic mean tends towards the lowest values in the set, it is not sensitive to large outliers, giving us robustness to polysemous words.
2.3 Finding Topics
After constructing the pseudowords of S we then need to find the coefficients Ci,k which describe each word in our vocabulary as a convex combination of the multiword anchors. Like standard anchor methods, we solve the following for each token type:
C∗i,· = argmin Ci,· DKL
( Si,· ∥∥∥∥ K∑ k=1 Ci,kSgk,· ) .
(6) Finally, we appeal to Bayes’ rule, we recover the topic-word matrix A from the coefficients of C.
3 High Water Mark for Tandem Anchors
Before addressing interactivity, we apply tandem anchors to real world data, but with anchors gleaned from metadata. Our purpose is twofold. First, we determine which combiner from Section 2.2 to use in our interactive experiments in Section 4 and second, we confirm that well-chosen tandem anchors can improve topics. In addition, we examine running time of tandem anchors and compare to traditional model-based interactive topic modeling techniques. We cannot assume that we will have metadata available to build tandem anchors, but we use them here because they provide a high water mark without the variance introduced by study participants.
3.1 Experimental Setup
We use the well-known 20 Newsgroups dataset (20NEWS) used in previous interactive topic modeling work: 18,846 usenet postings from 20 different newgroups in the early 1990s. We remove stopwords and words which appear in fewer than 100 documents or more than 1,500 documents.
To seed the tandem anchors, we use the titles of newsgroups. To build each multiword anchor facet, we split the title on word boundaries and expand any abbreviations or acronyms. For example, the newsgroup title ‘comp.os.mswindows.misc’ becomes {“computer”, “operating”, “system”, “microsoft”, “windows”, “miscellaneous”}. We do not fully specify the topic; the title gives some intuition, but the topic modeling algorithm must still recover the complete
topic-word distributions. This is akin to knowing the names of the categories used but nothing else. Critically, the topic modeling algorithm has no knowledge of document-label relationships.
3.2 Experimental Results
Our first evaluation is a classification task to predict documents’ newsgroup membership. We do not aim for state-of-the-art accuracy, but the experiment shows title-based tandem anchors yield topics closer to the underlying classes than GramSchmidt anchors. After randomly splitting the data into test and training sets we learn topics from the test data using both the title-based tandem anchors and the Gram-Schmidt single word anchors.1 For multiword anchors, we use each of the proposed combiner functions from Section 2.2. The anchor algorithm only gives the topic-word distributions and not word-level topic assignments, we infer token-level topic assignments using Latent Dirichlet Allocation (Blei et al., 2003) with fixed topics discovered by the anchor method. We use our own implementation of Gibbs sampling with fixed topics and a symmetric document-topic Dirichlet prior with concentration α = .01. Since the topics are fixed, this inference is very fast and can be parallelized on a perdocument basis. We then train a hinge-loss linear classifier on the newsgroup labels using Vowpal Wabbit2 with topic-word pairs as features. Finally, we infer topic assignments in the test data and evaluate the classification using those topicword features. For both training and test, we exclude words outside the LDA vocabulary.
The topics created from multiword anchor facets are more accurate than Gram-Schmidt topics (Figure 1). This is true regardless of the combiner function. However, harmonic mean is more accurate than the other functions.3
Since 20NEWS has twenty classes, accuracy alone does not capture confusion between closely related newsgroups. For example, accuracy penalizes a classifier just as much for label-
1With fixed anchors and data the anchor algorithm is deterministic, so we use random splits instead of the standard train/test splits so that we can compute variance.
2http://hunch.net/˜vw/ 3Significant at p < 0.01/4 when using two-tailed t-tests with a Bonferroni correction. For each of our evaluations, we verify the normality of our data (D’Agostino and Pearson, 1973) and use two-tailed t-tests with Bonferroni correction to determine whether the differences between the different methods are significant.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
●●
● ●● ●
● ●
●
● ● ●
●●
● ●
●
● ●● ●
● ●●
Accuracy ARI F−Measure VI Coherence
Gram−Schmidt
Title+Average
Title+Or
Title+Min
Title+HMean
0. 55
0. 60
0. 65
0. 70
0. 30 0. 35 0. 40 0. 45 0. 50
0. 56 0. 60 0. 64 0. 68 0. 72 2. 4 2. 7 3. 0 3. 3 3. 6 − 23 0 − 22 5 − 22 0 − 21 5 − 21 0
Figure 1: Using metadata can improve anchor-based topic models. For all metrics, the unsupervised Gram-Schmidt anchors do worse than creating anchors based on Newsgroup titles (for all metrics except VI, higher is better). For coherence, Gram-Schmidt does better than two functions for combining anchor words, but not the element-wise min or harmonic mean.
ing a document from ‘rec.sport.baseball’ with ‘rec.sport.hockey’ as with ‘alt.atheism’ despite the similarity between sports newsgroups. Consequently, after building a confusion matrix between the predicted and true classes, external clustering metrics reveal confusion between classes.
The first clustering metric is the adjusted Rand index (Yeung and Ruzzo, 2001), which is akin to accuracy for clustering, as it gives the percentage of correct pairing decisions from a reference clustering. Adjusted Rand index also accounts for chance groupings of documents. Next we use Fmeasure, which also considers pairwise groups, balancing the contribution of false negatives, but without the true negatives. Finally, we use variation of information. This metric measures the amount of information lost by switching from the gold standard labels to the predicted labels (Meilă, 2003). Since we are measuring the amount of information lost, lower variation of information is better.
Based on these clustering metrics, tandem anchors can yield superior topics to those created using single word anchors (Figure 1). As with accuracy, this is true regardless of which combination function we use. Furthermore, harmonic mean produces the least confusion between classes.4
The final evaluation is topic coherence by Newman et al. (2010), which measures whether the topics make sense, and correlates with human judgments of topic quality. Given V , the set of
4Significant at p < 0.01/4 when using two-tailed t-tests with a Bonferroni correction. For each of our evaluations, we verify the normality of our data (D’Agostino and Pearson, 1973) and use two-tailed t-tests with Bonferroni correction to determine whether the differences between the different methods are significant.
the n most probable words of a topic, coherence is
∑ v1,v2∈V log D(v1, v2) + D(v2) (7)
where D(v1, v2) is the co-document frequency of word types v1 and v2, and D(v2) is the document frequency of word type v2. A smoothing parameter prevents zero logarithms.
Figure 1 also shows topic coherence. Although title-based anchor facets produce better classification features, topics from Gram-Schmidt anchors have better coherence than title-based anchors with the vector average or the or-operator. However, when using the harmonic mean combiner, title-based anchors produce the most human interpretable topics.4
Harmonic mean beats other combiner functions because it is robust to ambiguous or irrelevant term cooccurrences an anchor facet. Both the vector average and the or-operator are swayed by large outliers, making them sensitive to ambiguous terms in an anchor facet. Element-wise min also has this robustness, but harmonic mean is also able to better characterize anchor facets as it has more centralizing tendency than the min.
3.3 Runtime Considerations
Tandem anchors will enable users to direct topic inference to improve topic quality. However, for the algorithm to be interactive we must also consider runtime. Cook and Thomas (2005) argue that for interactive applications with user-initiated actions like ours the response time should be less than ten seconds. Longer waits can increase the cognitive load on the user and harm the user interaction.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Algorithm #Doc Runtime (seconds) Tandem Anchors 18828 2.13
Fast ITM 13284 24.8 Utopian 515 48.0
Table 2: Reported run times for various interactive topic modeling algorithms. Only tandem anchors is fast enough to be interactive. For Tandem Anchors, we ran on a single core of an AMD Phemon II X6 1090T processor. For Fast ITM, we use the time reported by Hu and Boyd-Graber (2012). For Utopian, we use the time published by Choo et al. (2013).
Fortunately, the runtime of tandem anchors is amenable to interactive topic modeling. Our run time results are summarized in Table 2. On 20NEWS, interactive updates take roughly two seconds. Furthermore, larger datasets typically have a sublinear increase in distinct word types, so we can expect to see similar run times, even on much larger datasets.
Compared to other interactive topic modeling algorithms, tandem anchors has a very attractive run time. For example, using an optimized version of the sampler for the Interactive Topic Model described by Hu and Boyd-Graber (2012), and the recommended 30 iterations of sampling, the Interactive Topic Model is well beyond our desired update time for interactive use and an order of magnitude slower than tandem anchors.
Another promising interactive topic modeling approach is Utopian (Choo et al., 2013), which uses non-negative factorization, albeit without the benefit of anchor words. Utopian is much slower than tandem anchors. Even on the small InfoVisVAST dataset, Utopian takes nearly a minute to converge. Furthermore, since Utopian scales linearly with the size of the data, for moderately sized datasets such as 20NEWS, Utopian is infeasible for interactive topic modeling due to run time.
While each of these interactive topic modeling algorithms do achieve reasonable topics, only our algorithm fits the run time requirements for interactivity. Furthermore, since tandem anchors scales with the size of the vocabulary rather than the size of the data, this trend will only become more pronounced as we increase the amount of data.
4 Interactive Anchor Words
Given high quality anchor facets, the tandem anchor algorithm can produce high quality topic
models (particularly when the harmonic mean combiner is used). Furthermore, the tandem anchor algorithm is fast enough to be interactive (as opposed to model-based approaches such as the Interactive Topic Model). We now turn our attention to our main experiment: tandem anchors applied to the problem of interactive topic modeling. We compare both single word and tandem anchors in our study. We do not include the Interactive Topic Model or Utopian, as their run times are too slow for our users.
4.1 Interface and User Study
To show that interactive tandem anchor words are fast, effective, and intuitive, we ask users to understand a dataset using the anchor word algorithm. For this user study, we recruit twenty participants drawn from a university student body. The student median age is twenty-two. Seven are female, and thirteen are male. None of the students had any prior familiarity with topic modeling or the 20NEWS dataset.
Each participant sees a simple user interface (Figure 2) with topic given as a row with two columns. The left column allows users to view and edit topics’ anchor words; the right column lists the most probable words in each topic.5 The user can remove an anchor word or drag words from the topic word lists (right column) to become an anchor word. Users can also add additional topics by clicking the “Add Anchor” to create additional anchors. If the user wants to add a word to a tandem anchor set that does not appear in the interface, they manually type the word (restricted to the model’s vocabulary). When the user wants to see the updated topics for their newly refined anchors,
5While we use topics generated using harmonic mean for our final analysis, users were shown topics generated using the min combiner. However, this does not change our result.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
they click “Update Topics”. We give each a participant a high level overview of topic modeling. We also describe common problems with topic models including intruding topic words, duplicate topics, and ambiguous topics. Users are instructed to use their best judgement to determine if topics are useful. The task is to edit the anchor words to improve the topics. We asked that users spend at least twenty minutes, but no more than thirty minutes. We repeat the task twice: once with tandem anchors, and once with single word anchors.6
4.2 Quantitative Results
We now validate our main result that for interactive topic modeling, tandem anchors yields better topics than single word anchors. Like our titlebased experiments in Section 3, topics generated from users become features to train and test a classifier for the 20NEWS dataset. Based on our results with title-based anchors, we use the harmonic mean combiner in our analysis. As before, we report not only accuracy, but also multiple clustering metrics using the confusion matrix from the classification task. Finally, we report topic coherence.
Figure 3 summarizes the results of our quantitative evaluation. While we only compare user generated anchors in our analysis, we include the unsupervised Gram-Schmidt anchors as a baseline. Some of the data violate assumptions of normality. Therefore, we use Wilcoxon’s signed-rank test (Wilcoxon, 1945) to determine if the differences between multiword anchors and single word anchors are significant.
Topics from user generated multiword anchors yield higher classification accuracy (Figure 3). Not only is our approach more scalable than the Interactive Topic Model, but we also achieve higher classification accuracy than Hu et al. (2014).7 Tandem anchors also improve clustering metrics.8
While user selected tandem anchors produce better classification features than single word anchors, users selected single word anchors produce topics with similar topic coherence scores.9
6The order in which users complete these tasks is counterbalanced.
7However, the values are not strictly comparable, as Hu et al. (2014) use the standard chronological test/train fold, and we use random splits.
8Significant at p < 0.01 when using Wilcoxon’s signedrank test.
9The difference between coherence scores was not statis-
To understand this phenomenon, we use quality metrics (AlSumait et al., 2009) for ranking topics by their correspondence to genuine themes in the data. Significant topics are likely skewed towards a few related words, so we measure the distance of each topic-word distribution from the uniform distribution over words. Topics which are close to the underlying word distribution of the entire data are likely to be vacuous, so we also measure the distance of each topic-word distribution from the underlying word distribution. Finally, background topics are likely to appear in a wide range of documents, while meaningful topics will appear in a smaller subset of the data.
Figure 4 reports our topic significance findings. For all three significance metrics, multiword anchors produce more significant topics than single word anchors.8Topic coherence is based solely on the top n words of a topic, while both accuracy and topic significance depend on the entire topicword distributions. With single word anchors, topics with good coherence may still be too general. Tandem anchors enables users to produce topics with more specific word distributions which are better features for classification.
4.3 Qualitative Results
We examine the qualitative differences between how users select multiword anchor facets versus single word anchors. Table 3 gives examples of topics generated using different anchor strategies. In a follow-up survey with our users, 75% find it easier to affect individual changes in the topics using tandem anchors compared to single word anchors. Users who prefer editing multiword anchors over single word anchors often report that multiword anchors make it easier to merge similar topics into a single focused topic by combining anchors. For example, by combining multiple words related to Christianity, users were able to create a topic which is highly specific, and differentiated from general religion themes which included terms about Atheism and Judaism.
While users find that use tandem anchors is easier, only 55% of our users say that they prefer the final topics produced by tandem anchors compared to single word anchors. This is in harmony with our quantitative measurements of topic coherence, and may be the result of our stopping criteria: when users judged the topics to be useful.
tically significant using Wilcoxon’s signed-rank test.
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
● ●● ● ●●
●
Accuracy ARI F−Measure VI Coherence
Tandem
Singleword
Gram−Schmidt
0. 55
0. 60
0. 65
0. 70 0. 30 0. 35 0. 40 0. 45 0. 50 0. 55
0. 60
0. 65
0. 70 2. 7 3. 0 3. 3 3. 6 − 24 0 − 23 0 − 22 0 − 21 0 − 20 0 − 19 0
Figure 3: Classification accuracy and coherence using topic features gleaned from user provided multiword and single word anchors. Grahm-Schmidt anchors are provided as a baseline. For all metrics except VI, higher is better. Except for coherence, multiword anchors are best.
●
●●●
●
●●●
●
●●●
uniform vacuous background
Tandem
Singleword
Gram−Schmidt
1 2 3
0. 5
1. 0
1. 5
2. 0
0. 5
1. 0
1. 5
2. 0
Figure 4: Topic significance for both single word and multiword anchors. In all cases higher is better. Multiword anchors produce topics which are more significant than single word anchors.
Anchor Top Words in Topic Automatic Gram Schmidt love love god evolution romans heard car game game games team hockey baseball heard Interactive Single-word evolution evolution theory science faith quote facts religion religion god government state jesus israel baseball baseball games players word teams car hockey hockey team play games season players Interactive Tandem atheism god exists prove god science evidence reason faith objective christian jesus jesus christian christ church bible christians jew israel israel jews jewish israeli state religion baseball bat ball hit baseball ball player games call hockey nhl team hockey player nhl win play
Table 3: Comparison of topics generated for 20NEWS using various types of anchor words. Users are able to combine words to create more specific topics with tandem anchors.
However, 100% of our users feel that the topics created through interaction were better than those generated from Gram-Schmidt anchors. This was true regardless of whether we used tandem anchors or single word anchors.
Our participants also produce fewer topics when using multiword anchors. The mean difference between topics under single word anchors and multiple word anchors is 9.35. In follow up interviews, participants indicate that the easiest way to resolve an ambiguous topic with single word anchors was to create a new anchor for each of the ambiguous terms, thus explaining the proliferation of topics for single word anchors. In contrast, fixing an ambiguous tandem anchor is simple: users just add more terms to the anchor facet.
5 Conclusion
Tandem anchors extend the anchor words algorithm to allow multiple words to be combined into anchor facets. For interactive topic modeling, using anchor facets in place of single word anchors produces higher quality topic models and are more intuitive to use. Furthermore, our approach scales much better than existing interactive topic modeling techniques, allowing interactivity on large datasets for which interactivity was previous impossible.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths: Clear description of methods and evaluation Successfully employs and interprets a variety of evaluations Solid demonstration of practicality of technique in real-world interactive topic modeling  - Weaknesses: Missing related work on anchor words Evaluation on 20 Newsgroups is not ideal Theoretical contribution itself is small   - General Discussion: The authors propose a new method of interactive user specification of topics called Tandem Anchors. The approach leverages the anchor words algorithm, a matrix-factorization approach to learning topic models, by replacing the individual anchors inferred from the Gram-Schmidt algorithm with constructed anchor pseudowords created by combining the sparse vector representations of multiple words that for a topic facet. The authors determine that the use of a harmonic mean function to construct pseudowords is optimal by demonstrating that classification accuracy of document-topic distribution vectors using these anchors produces the most improvement over Gram-Schmidt. They also demonstrate that their work is faster than existing interactive methods, allowing interactive iteration, and show in a user study that the multiword anchors are easier and more effective for users.  Generally, I like this contribution a lot: it is a straightforward modification of an existing algorithm that actually produces a sizable benefit in an interactive setting. I appreciated the authors  efforts to evaluate their method on a variety of scales. While I think the technical contribution in itself is relatively small (a strategy to assemble pseudowords based on topic facets) the thoroughness of the evaluation merited having it be a full paper instead of a short paper. It would have been nice to see more ideas as to how to build these facets in the absence of convenient sources like category titles in 20 Newsgroups or when initializing a topic model for interactive learning.  One frustration I had with this paper is that I find evaluation on 20 Newsgroups to not be great for topic modeling: the documents are widely different lengths, preprocessing matters a lot, users have trouble making sense of many of the messages, and naive bag-of-words models beat topic models by a substantial margin. Classification tasks are useful shorthand for how well a topic model corresponds to meaningful distinctions in the text by topic; a task like classifying news articles by section or reviews by the class of the subject of the review might be more appropriate. It would also have been nice to see a use case that better appealed to a common expressed application of topic models, which is the exploration of a corpus.  There were a number of comparisons I think were missing, as the paper contains little reference to work since the original proposal of the anchor word model. In addition to comparing against standard Gram-Schmidt, it would have been good to see the method from Lee et. al. (2014),  Low-dimensional Embeddings for Interpretable Anchor-based Topic Inference d. I also would have liked to have seen references to Nguyen et. al. (2013),  Evaluating Regularized Anchor Words d and Nguyen et. al. (2015)  Is Your Anchor Going Up or Down? Fast and Accurate Supervised Topic Models d, both of which provide useful insights into the anchor selection process.  I had some smaller notes: - 164:  u2026entire dataset - 164-166: I m not quite sure what you mean here. I think you are claiming that it takes too long to do one pass? My assumption would have been you would use only a subset of the data to retrain the model instead of a full sweep, so it would be good to clarify what you mean. - 261&272: any reason you did not consider the and operator or element-wise max? They seem to correspond to the ideas of union and intersection from the or operator and element-wise min, and it wasn t clear to me why the ones you chose were better options. - 337: Usenet should be capitalized - 338-340: Why fewer than 100 (as that is a pretty aggressive boundary)? Also, did you remove headers, footers, and or quotes from the messages? - 436-440: I would have liked to see a bit more explanation of what this tells us about confusion. - 692: using tandem anchors  Overall, I think this paper is a meaningful contribution to interactive topic modeling that I would like to see available for people outside the machine learning community to investigate, classify, and test hypotheses about their corpora.  POST-RESPONSE: I appreciate the thoughtful responses of the authors to my questions. I would maintain that for some of the complimentary related work that it's useful to compare to non-interactive work, even if it does something different.