Modern models of event extraction for tasks like ACE are based on supervised learning of events from small hand-labeled data. However, hand-labeled training data is expensive to produce, in low coverage of event types, and limited in size, which makes supervised methods hard to extract large scale of events for knowledge base population. To solve the data labeling problem, we propose to automatically label training data for event extraction via world knowledge and linguistic knowledge, which can detect key arguments and trigger words for each event type and employ them to label events in texts automatically. The experimental results show that the quality of our large scale automatically labeled data is competitive with elaborately human-labeled data. And our automatically labeled data can incorporate with human-labeled data, then improve the performance of models learned from these data.
1 Introduction
Event Extraction (EE), a challenging task in Information Extraction, aims at detecting and typing events (Event Detection), and extracting arguments with different roles (Argument Identification) from natural-language texts. For example, in the sentence shown in Figure 1, an EE system is expected to identify an Attack event triggered by threw and extract the corresponding five augments with different roles: Yesterday (Role=Time), demonstrators (Role=Attacker), stones (Role=Instrument), soldiers (Role=Target), and Israeli (Role=Place).
To this end, so far most methods (Nguyen et al.,
2016; Chen et al., 2015; Li et al., 2014; Hong et al., 2011; Ji and Grishman, 2008) usually adopted supervised learning paradigm which relies on elaborate human-annotated data, such as ACE 20051, to train extractors. Although this paradigm was widely studied, existing approaches still suffer from high costs for manually labeling training data and low coverage of predefined event types. In ACE 2005, all 33 event types are manually predefined and the corresponding event information (including triggers, event types, arguments and their roles) are manually annotated only in 599 English documents since the annotation process is extremely expensive. As Figure 2 shown, nearly 60% of event types in ACE 2005 have less than 100 labeled samples and there are even three event types which have less than ten labeled samples. Moreover, those predefined 33 event types are in low coverage for Natural Language Processing (NLP) applications on large-scale data.
Therefore, for extracting large scale events, especially in open domain scenarios, how to automatically and efficiently generate sufficient training data is an important problem. This paper aims to automatically generate training data for EE, which involves labeling triggers, event types, arguments and their roles. Figure 1 shows an example of labeled sentence. Recent improvements of Distant Supervision (DS) have been proven to be effective to label training data for Relation Extraction (RE), which aims to predict semantic relations between pairs of entities, formulated as (entity1, relation, entity2). And DS for RE as-
1http://projects.ldc.upenn.edu/ace/
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
ACL 2017 Submission 350. Confidential Review Copy. DO NOT DISTRIBUTE.
0
200
400
600
800
1000
1200
1400
1600
Figure 2: Statistics of ACE 2005 English Data.
sumes that if two entities have a relationship in a known knowledge base, then all sentences that mention these two entities will express that relationship in some way (Mintz et al., 2009). However, when we use DS for RE to EE, we meet following challenges:
Triggers are not given out in existing knowledge bases. EE aims to detect an event instance of a specific type and extract their arguments and roles, formulated as (event instance, event type; role1, argument1; role2, argument2; ...; rolen, argumentn), which can be regarded as a kind of multiple or complicated relational data. In Figure 3, the right part shows an example of spouse of relation between Barack Obama and Michelle Obama, where two rectangles represent two entities and the edge connecting them represents their relation. DS for RE uses two entities to automatically label training data; In comparison, the left part in Figure 3 shows a marriage event of Barack Obama and Michelle Obama, where the dash circle represents the marriage event instance of Barack Obama and Michelle Obama, rectangles represent arguments of the event instance, and each edge connecting an argument and the event instance expresses the role of the argument. For example, Barack Obama plays a Spouse role in this marriage event instance. It seems that we could use an event instance and an argument to automatically generate training data for argument identification just like DS for RE. However, an event instance is a virtual node in existing knowledge bases and mentioned implicitly in texts. For example, in Freebase, the aforementioned marriage event instance is represented as m.02nqglv (see details in Section 2). Thus we cannot directly use an event instance and an argument, like m.02nqglv and Barack Obama, to label back in sentences. In ACE event extraction program, an event instance is represented as a trigger word, which is the main word that most clearly represents an event occurrence in sentences, like threw
MarriageMichelle Obama 10/03/1992 Trinity United Church of Christ Null Spouse Spouse location of ceremony time_from time_to Barack Obama Michelle ObamaSpouse_ofVS An example of marriage event An example of spouse_of relation MarriageMichelle Obama 10/03/1992 Trinity United Church of Christ Null Spouse Spouse location of ceremony time_from time_to Barack Obama Michelle ObamaSpouse_of An example of marriage event An example of spouse_of relation
MarriageMichelle Obama
10/03/1992
Trinity United Church of Christ
Null
Spouse Spouse
location of ceremony
time_from time_to
Barack Obama Michelle Obama Spouse_of
An example of marriage event An example of spouse_of relation
Michelle Obama
Figure 3: A comparison of events and relations.
in Figure 1. Following ACE, we can use trigger words to represent event instance, like married for people.marriage event instance. Unfortunately, triggers are not given out in existing knowledge bases.
To resolve the trigger missing problem mentioned above, we need to discover trigger words before employing distant supervision to automatically label event arguments. Following DS in RE, we could naturally assume that a sentence contains all arguments of an event in the knowledge base tend to express that event, and the verbs occur in these sentences tend to evoke this type of events. However, arguments for a specific event instance are usually mentioned in multiple sentences. Simply employing all arguments in the knowledge base to label back in sentences will generate few sentences as training samples. As shown in Table 1, only 0.02% of instances can find all argument mentions in one sentence.
Event Type EI# A# S# education.education 530,538 8 0 film.film crew gig 252,948 3 8 people.marriage 152,276 5 0
... ... ... ... military.military service 27,933 6 0
olympics.olympic medal honor 20,790 5 4 sum of the selected 21 events 3,870,492 100 798
Table 1: Statistics of events in Freebase. EI# denotes number of event instances in Freebase. A# denotes number of arguments for each event types, and S# indicates number of sentences contain all arguments of each event type in Wikipedia.
To solve above problems, we propose an approach to automatically generate labeled data for large scale EE by jointly using world knowledge (Freebase) and linguistic knowledge (FrameNet). At first, we put forward an approach to prioritize arguments and select key or representative arguments (see details in Section 3.1) for each event type by using Freebase; Secondly, we merely use key arguments to label events and figure out trigger words; Thirdly, an external linguistic knowledge resource, FrameNet, is employed to filter noisy trigger words and expand more triggers; Af-
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
ACL 2017 Submission 350. Confidential Review Copy. DO NOT DISTRIBUTE.
ter that, we propose a Soft Distant Supervision (SDS) for EE to automatically label training data, which assumes that any sentence containing all key arguments in Freebase and a corresponding trigger word is likely to express that event in some way, and arguments occurring in that sentence are likely to play the corresponding roles in that event. Finally, we evaluate the quality of the automatically labeled training data by both manual and automatic evaluations. In addition, we employ a CNNbased EE approach with multi-instance learning for the automatically labeled data as a baseline for further research on this data. In summary, the contributions of this paper are as follows:
• To our knowledge, it is the first work to automatically label data for large scale EE via world knowledge and linguistic knowledge. All the labeled data in this paper have been released and can be downloaded freely2. • We propose an approach to figure out key ar-
guments of an event by using Freebase, and use them to automatically detect events and corresponding trigger words. Moreover, we employ FrameNet to filter noisy triggers and expand more triggers. • The experimental results show that the qual-
ity of our large scale automatically labeled data is competitive with elaborately humanannotated data. Also, our automatically labeled data can augment traditional humanannotated data, which could significantly improve the extraction performance.
2 Background
In this paper, we respectively use Freebase as our world knowledge containing event instance and FrameNet as the linguistic knowledge containing trigger information. The articles in Wikipedia are used as unstructured texts to be labeled. To understand our method easily, we first introduce them as follows:
Freebase is a semantic knowledge base (Bollacker et al., 2008), which makes use of mediators (also called compound value types, CVTs) to merge multiple values into a single value. As shown in Figure 3, people.marriage is one type of CVTs. There are many instances of people.marriage and the marriage of Barack Obama and Michelle Obama is numbered as m.02nqglv. Spouse, from, to and location of ceremony are
2https://github.com/acl2017submission/event-data
roles of the people.marriage CVTs. Barack Obama, Michelle Obama, 10/3/1992 and Trinity United Church of Christ are the values of the instances. In this paper, we regard these CVTs as events, type of CVTs as event type, CVT instances as event instances, values in CVTs as arguments in events and roles of CVTs as the roles of arguments play in the event, respectively. According to the statistics of the Freebase released on 23th April, 2015, there are around 1885 CVTs and around 14 million CVTs instances. After filtering out useless and meaningless CVTs, such as CVTs about user profiles and website information, we select 21 types of CVTs with around 3.8 million instances for experiments, which mainly involves events about education, military, sports and so on.
FrameNet3 is a linguistic resource storing information about lexical and predicate argument semantics (Baker et al., 1998). FrameNet contains more than 1, 000 frames and 10, 000 Lexical Units (LUs). Each frame of FrameNet can be taken as a semantic frame of a type of events (Liu et al., 2016). Each frame has a set of lemmas with part of speech tags that can evoke the frame, which are called LUs. For example, appoint.v is a LU of Appointing frame in FrameNet, which can be mapped to people.appointment events in Freebase. And a LUs of the frame plays a similar role as the trigger of an event. Thus we use FrameNet to detect triggers in our automatically data labeling process.
Wikipedia4 that we used was released on January, 2016. All 6.3 million articles in it are used in our experiments. We use Wikipedia because it is relatively up-to-date, and much of the information in Freebase is derived from Wikipedia.
3 Method of Generating Training Data
Figure 4 describes the architecture of automatically labeling data, which primarily involves the following four components: (i) Key argument detection, which prioritizes arguments of each event type and selects key arguments for each type of event; (ii) Trigger word detection, which uses key arguments to label sentences that may express events preliminarily, and then detect triggers; (iii) Trigger word filtering and expansion, which uses FrameNet to filter noisy triggers and expand triggers; (iv) Automatically labeled data generation, which uses a SDS to label events in sentences.
3http://framenet.icsi.berkeley.edu 4https://www.wikipedia.org/
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
ACL 2017 Submission 350. Confidential Review Copy. DO NOT DISTRIBUTE.
Wikipedia
TrainingDS
Figure 4: The architecture of automatically labeling training data for large scale event extraction.
3.1 Key Argument Detection
This section illustrates how to detect key arguments for each event type via Freebase. Intuitively, arguments of a type of event play different roles. Some arguments play indispensable roles in an event, and serve as vital clues when distinguishing different events. For example, compared with arguments like time, location and so on, spouses are key arguments in a marriage event. We call these arguments as key arguments. We propose to use Key Rate (KR) to estimate the importance of an argument to a type of event, which is decided by two factors: Role Saliency and Event Relevance.
Role Saliency (RS) reflects the saliency of an argument to represent a specific event instance of a given event type. If we tend to use an argument to distinguish one event instance form other instances of a given event type, this argument will play a salient role in the given event type. We define RS as follows:
RSij = Count(Ai, ETj)
Count(ETj) (1)
where RSij is the role saliency of i-th argument to j-th event type, Count(Ai , ETj) is the number of Arguemnti occurring in all instances of eventTypej in Freebase and Count(ETj) is the number of instances of eventTypej in Freebase.
Event Relevance (ER) reflects the ability in which an argument can be used to discriminate different event types. If an argument occurs in every event type, the argument will have a low event relevance. We propose to compute ER as follows:
ERi = log Sum(ET )
1 + Count(ETCi) (2)
where ERi is the event relevance of i-th argument, Sum (ET ) is the number of all event types in knowledge base and Count(ETCi) is the number of event types containing i-th argument. Finally, KR is computed as follows:
KRij = RSij ∗ ERi (3) We compute KR for all arguments of each event type, and sort them according to KR. Then we choose top K arguments as key arguments.
3.2 Trigger Word Detection
After detecting key arguments for every event types, we use these key arguments to label sentences that may express events in Wikipedia. At first, we use Standford CoreNLP tool5 to converts the raw Wikipedia texts into a sequence of sentences, attaches NLP annotations (POS tag, NER tag). Finally, we select sentences contains all key arguments of an event instance in Freebase as sentences expressing corresponding events. Then we use these labeled sentences to detect triggers.
In a sentence, a verb tend to express an occurrence of an event. For example, in ACE 2005 English data, there are 60% of events triggered by verbs. As shown in Figure 1, threw is a trigger of Attack event. Intuitively, if a verb occurs more times than other verbs in the labeled sentences of one event type , the verb tends to trigger this type of event; and if a verb occurs in sentences of every event types, like is, the verb will have a low probability to trigger events. Thus we propose Trigger Candidate Frequency (TCF) and Trigger Event Type Frequency (TETF) to evaluate above two aspects. Finally we employ Trigger Rate (TR), which is the product of TCF and TETF to estimate the probability of a verb to be a trigger, which is formulated as follows:
TRij = TCFij ∗ TETFi (4)
TCFij = Count(Vi, ETSj)
Count(ETSj) (5)
TETFi = log Sum(ET )
1 + Count(ETIi) (6)
where TRij is the trigger rate of i-th verb to jth event type, Count(Vi, ETSj) is the number of sentences, which express j-th type of event and contain i-th verb, Count(ETSj) is the number of sentences expressing j-th event type, Count(ETIi) is the number of event types, which have the labeled sentences containing i-th verb. Finally, we choose verbs with high TR values as the trigger words for each event type.
3.3 Trigger Word Filtering and Expansion
We can obtain an initial verbal trigger lexicon by above trigger word detection. However, this initial trigger lexicon is noisy and merely contains verbal triggers. The nominal triggers like marriage are missing. Because the number of nouns in one sentence is usually larger than that of verbs, it is hard to use TR to find nominal triggers. Thus, we propose to use linguistic resource FrameNet to filter
5http://stanfordnlp.github.io/CoreNLP/
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
ACL 2017 Submission 350. Confidential Review Copy. DO NOT DISTRIBUTE.
noisy verbal triggers and expand nominal triggers. As the success of word embedding in capturing semantics of words (Turian et al., 2010), we employ word embedding to map the events in Freebase to frames in FrameNet. Specifically, we use the average word embedding of all words in i-th Freebase event type name ei and word embedding of k-th lexical units of j-th frame ej,k to compute the semantic similarity. Finally, we select the frame contains max similarity of ei and ej,k as the mapped frame, which can be formulated as follows:
frame(i) = argmax j (similarity(ei, ej,k)) (7) Then, we filter the verb, which is in initial verbal trigger word lexicon and not in the mapping frame. And we use all nouns in the mapped frame to expand trigger lexicon. 3.4 Automatically labeled data generation Finally, we propose a Soft Distant Supervision and use it to automatically generate training data, which assumes that any sentence containing all key arguments in Freebase and a corresponding trigger word is likely to express that event in some way, and arguments occurring in that sentence are likely to play the corresponding roles in that event.
4 Method of Event Extraction
In this paper, event extraction is formulated as a two-stage, multi-class classification task. The first stage is called Event Classification, which aims to predict whether the key argument candidates participate in a Freebase event. If the key arguments participate a Freebase event, the second stage is conducted, which aims to assign arguments to the event and identify their corresponding roles. We call this stage as argument classification. We employ two similar Dynamic Multi-pooling Convolutional Neural Networks with Multi-instance Learning (DMCNNs-MIL) for above two stages. The Dynamic Multi-pooling Convolutional Neural Networks (DMCNNs) is the best reported CNN-based model for event extraction (Chen et al., 2015) by using human-annotated training data. However, our automatically labeled data face a noise problem, which is a intrinsic problem of using DS to construct training data (Hoffmann et al., 2011; Surdeanu et al., 2012). In order to alleviate the wrong label problem, we use Multi-instance Learning (MIL) for two DMCNNs. Because the second stage is more complicated and limited in space, we take the MIL used in arguments classification as an example and describes as follows:
We define all of the parameters for the stage of argument classification to be trained in DMCNNs as θ. Suppose that there are T bags {M1,M2, ...,MT } and that the i-th bag contains qi instances (sentences) Mi = { m1i ,m 2 i , ...,m qi i } , the objective of multi-instance learning is to predict the labels of the unseen bags. In stage of argument classification, we take sentences containing the same argument candidate and triggers with a same event type as a bag and all instances in a bag are considered independently. Given an input instance mji , the network with the parameter θ outputs a vector O, where the r-th component Or corresponds to the score associated with argument role r. To obtain the conditional probability p(r|mji , θ), we apply a softmax operation over all argument role types:
p(r|mji , θ) = eor n∑ k=1 eok (8)
where, n is the number of roles. And the objective of multi-instance learning is to discriminate bags rather than instances. Thus, we define the objective function on the bags. Given all (T ) training bags (Mi, yi), we can define the objective function using cross-entropy at the bag level as follows:
J (θ) = T∑ i=1 log p(yi|mji , θ) (9)
where j is constrained as follows:
j∗ = argmax j p(r|mji , θ) 1 ≤ j ≤ qi (10)
To compute the network parameter θ, we maximize the log likelihood J (θ) through stochastic gradient descent over mini-batches with the Adadelta (Zeiler, 2012) update rule.
5 Experiments
In this section, we first manually evaluate our automatically labeled data. Then, we conduct automatic evaluations for our labeled data based on ACE corpus and analyze effects of different approaches to automatically label training data. Finally, we shows the performance of DMCNNs-MIL on our automatically labeled data.
5.1 Our Automatically Labeled Data
By using the proposed methods, a large set of labeled data could be generated automatically. Table 2 shows the statistics of the five largest automatically labeled events among selected 21 Freebase events. Two hyper parameters, the number of key arguments and the value of TR in our automatically data labeling, are set as 2 and 0.8, by
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
ACL 2017 Submission 350. Confidential Review Copy. DO NOT DISTRIBUTE.
Event Type FreebaseSize Sentences (KA) Sentences (KA+T) Examples of argument roles sorted by KR Examples of triggers
people.marriage 152,276 56,837 26,349 spouse, spouse, from, to, location marriage, marry, wed, wedding, couple,..., wife music.group membership 239,813 90,617 20,742 group, member, start, role, end musician, singer, sing, sang, sung, concert,..., play
education.education 530,538 26,966 11,849 student, institution, degree,..., minor educate, education, graduate, learn, study,..., student organization.leadership 43,610 5,429 3,416 organization, person, title,..., to CEO, charge, administer, govern, rule, boss,..., chair
olympics.olympic medal honor 20,790 4,056 2,605 medalist, olympics, event,..., country win, winner, tie, victor, gold, silver,..., bronze ... ... ... ... ... ...
sum of 21 selected events 3,870,492 421,602 72,611 argument1, argument2 ,..., argumentN trigger1, trigger2, trigger3, ... , triggerN
Table 2: The statistics of five largest automatically labeled events in selected 21 Freebase events, with their size of instances in Freebase, sentences labeled with key argument (KA) and KA + Triggers(T), examples of arguments roles sorted by KR and examples of triggers.
grid search respectively. When we merely use two key arguments to label data, we will obtain 421, 602 labeled sentences. However, these sentences miss labeling triggers. Thus, we leverage these rough labeled data and FrameNet to find triggers and use SDS to generate labeled data. Finally, 72, 611 labeled sentences are generated automatically. Compared with nearly 6, 000 human annotated labeled sentence in ACE, our method can automatically generate large scale labeled training data.
5.2 Manual Evaluations of Labeled Data
##001 He is the uncle of [Amal Clooney], [wife] of the actor [George Clooney]. Trigger: wife Event Type: Marriage MannalAnotate[Y/N]: Argument: Amal Clooney Role:Spouse MannalAnotate[Y/N]: Argument: George Clooney Role:Spouse MannalAnotate[Y/N]:
##002 She was [married] to the cinematographer [Theo Nischwitz] and was sometimes credited as [Gertrud Hinz-Nischwitz]. Trigger: married Event Type: Marriage MannalAnotate[Y/N]: Argument: Theo Nischwitz Role:Spouse MannalAnotate[Y/N]: Argument: Gertrud Hinz-Nischwitz Role:Spouse MannalAnotate[Y/N]:
Figure 5: Examples of manual evaluations.
We firstly manually evaluate the precision of our automatically generated labeled data. We randomly select 500 samples from our automatically labeled data. Each selected sample is a sentence with a highlighted trigger, labeled arguments and corresponding event type and argument roles. Figure 5 gives some samples. Annotators are asked to assign one of two labels to each sample. “Y”: the word highlighted in the given sentence indeed triggers an event of the corresponding type or the word indeed plays the corresponding role in that event. Otherwise “N” is labeled. It is very easy to annotate a sample for annotators, thus the annotated results are expected to be of high quality. Each sample is independently annotated by three annotators6 (including one of the authors and two of our colleagues who are familiar with event extraction task) and the final decision is made by voting.
We repeat above evaluation process on the final 72, 611 labeled data three times and the average
6The inter-agreement rate is 87.5%
Stage Average Precision Trigger Labeling 88.9
Argument Labeling 85.4
Table 3: Manual Evaluation Results
precision is shown in Table 3. Our automatically generated data can achieve a precision of 88.9 and 85.4 for trigger labeling and argument labeling respectively, which demonstrates that our automatically labeled data is of high quality.
5.3 Automatic Evaluations of Labeled Data
To prove the effectiveness of the proposed approach automatically, we add automatically generated labeled data into ACE dataset to expand the training sets and see whether the performance of the event extractor trained on such expanded training sets is improved. In our automatically labeled data, there are some event types that can correspond to those in ACE dataset. For example, our people.marriage events can be mapped to life.marry events in ACE2005 dataset. We mapped these types of events manually and we add them into ACE training corpus in two ways. (1) we delete the human annotated ACE data for these mapped event types in ACE dataset and add our automatically labeled data to remainder ACE training data. We call this Expanded Data (ED) as ED Only. (2) We directly add our automatically labeled data of mapped event types to ACE training data and we call this training data as ACE+ED. Then we use such data to train the same event extraction model (DMCNN) and evaluate them on the ACE testing data set. Following (Nguyen et al., 2016; Chen et al., 2015; Li et al., 2013), we used the same test set with 40 newswire articles and the same development set with 30 documents and the rest 529 documents are used for ACE training set. And we use the same evaluation metric P, R, F as ACE task defined. We select three baselines trained with ACE data. (1) Li’s structure, which is the best reported structured-based system (Li et al., 2013). (2) Chen’s DMCNN, which is the best reported CNN-based system (Chen et al., 2015). (3) Nguyen’s JRNN, which is the state-of-
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
ACL 2017 Submission 350. Confidential Review Copy. DO NOT DISTRIBUTE.
Methods TriggerIdentification(%) Trigger Identification + Classification(%) Argument Identification(%)
Argument Role(%)
P R F P R F P R F P R F Li’s structure trained with ACE 76.9 65.0 70.4 73.7 62.3 67.5 69.8 47.9 56.8 64.7 44.4 52.7 Chen’s DMCNN trained with ACE 80.4 67.7 73.5 75.6 63.6 69.1 68.8 51.9 59.1 62.2 46.9 53.5 Nguyen’s JRNN trained with ACE 68.5 75.7 71.9 66.0 73.0 69.3 61.4 64.2 62.8 54.2 56.7 55.4 DMCNN trained with ED Only 77.6 67.7 72.3 72.9 63.7 68.0 64.9 51.7 57.6 58.7 46.7 52.0 DMCNN trained with ACE+ED 79.7 69.6 74.3 75.7 66.0 70.5 71.4 56.9 63.3 62.8 50.1 55.7
Table 4: Overall performance on ACE blind test data
the-arts system (Nguyen et al., 2016). The results are shown in Table 4. Compared with all models, DMCNN trained with ACE+ED achieves the highest performance. This demonstrates that our automatically generated labeled data could expand human annotated training data effectively. Moreover, compared with Chen’s DMCNN trained with ACE, DMCNN trained with ED Only achieves a competitive performance. This demonstrates that our large scale automatically labeled data is competitive with elaborately humanannotated data.
5.4 Discussion Impact of Key Rate
In this section, we prove the effectiveness of KR to find key arguments and explore the impact of different numbers of key arguments to automatically generate data. We specifically select two methods as baselines for comparison with our KR method: ER and RS, which use the event relevance and role salience to sort arguments of each type of events respectively. Then we choose the same number of key arguments in all methods and use these key arguments to label data. After that we evaluate these methods by using above automatic evaluations based on ACE data. Results are shown in Table 5. ACE+KR achieve the best performance in both stages. This demonstrates the effectiveness of our KR methods.
Feature Trigger Argument F1 F1
ACE 69.1 53.5 ACE + RS 70.1 55.3 ACE + ER 69.5 54.2 ACE + KR 70.5 55.7
Table 5: Effects of ER, RS and KR
To explore the impact of different numbers of key arguments, we sort all arguments of each type of events according to KR value and select top k arguments as the key arguments. Examples are shown in Table 2. Then we automatically evaluate the performance by using automatic evaluations proposed above. Figure 6 shows the results, when we set k = 2, the method achieves a best
Impact of Trigger Rate and FrameNet In this section, we prove the effectiveness of TR and FrameNet to find triggers. We specifically select two methods as baselines: TCF and TETF. TCF, TETF and TR respectively use the trigger candidate frequency, trigger event type frequency and trigger rate to sort trigger candidates of each type of events. Then we generate initial trigger lexicon by using all trigger candidates with high TCF value, TETF value or TR value. We set these hyper parameters as 0.8, 0.9 and 0.8, respectively, which are determined by grid search from (0.5, 0.6, 0.7, 0.8, 0.9, 1.0). FrameNet was used to filter noisy verbal triggers and expand nominal triggers. Trigger examples generated by TR+Framenet are shown in Table 2. Then we evaluate the performance of these methods by using above automatic evaluations. Results are shown in Table 6, Compared with ACE+TCF and ACE+TETF, ACE+TR gains a higher performance in both stages. It demonstrates the effectiveness of our TR methods. When we use FrameNet to generate triggers, compared with ACE+TR, we get a 1.0 improvement on trigger classification and a 1.7 improvement on argument classification. Such improvements are higher than improvements gained by other methods (TCF, IEF, TR), which demon-
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
ACL 2017 Submission 350. Confidential Review Copy. DO NOT DISTRIBUTE.
strates the effectiveness of the usage of FrameNet.
Feature Trigger Argument F1 F1
ACE 69.1 53.5 ACE + TCF 69.3 53.8
ACE + TETF 69.2 53.7 ACE + TR 69.5 54.0 ACE + TR + FrameNet 70.5 55.7
Table 6: Effects of TCF, TETF,TR and FrameNet
5.5 Performance of DMCNN-MIL
Following previous work (Mintz et al., 2009) in distant supervised RE, we evaluate our method in two ways: held-out and manual evaluation.
Held-out Evaluation In the held-out evaluation, we hold out part of the Freebase event data during training, and compare newly discovered event instances against this heldout data. We use the following criteria to judge the correctness of each predicted event automatically: (1) An event is correct if its key arguments and event type match those of an event instance in Freebase; (2) An argument is correctly classified if its event type and argument role match those of any of the argument instance in the corresponding Freebase event. Figure 7 and Figure 8 show the precision-recall (P-R) curves for each method in the two stages of event extraction respectively. We can see that multi-instance learning is effective to alleviate the noise problem in our distant supervised event extraction.
Figure 7: P-R curves for event classification.
Human Evaluation Because the incomplete nature of Freebase, heldout evaluation suffers from false negatives problem. We also perform a manual evaluation to eliminate these problems. In the manual evaluation, we manually check the newly discovered event instances that are not in Freebase. Because the number of these event instances in the test data is unknown, we cannot calculate the recall in this case. Instead, we calculate the precision of the top n extracted event instances. The human evaluation results are presented in Table 7. We can see that DMCNNs-MIL achieves the best performance.
6 Related Work
Most of previous event extraction work focused on supervised learning paradigm and trained event extractors on human-annotated data which yield relatively high performance. (Ahn, 2006; Ji and Grishman, 2008; Hong et al., 2011; McClosky et al., 2011; Li et al., 2013, 2014; Chen et al., 2015; Nguyen and Grishman, 2015; Nguyen et al., 2016). However, these supervised methods depend on the quality of the training data and labeled training data is expensive to produce. Unsupervised methods can extract large numbers of events without using labeled data (Chambers and Jurafsky, 2011; Cheung et al., 2013; Huang et al., 2016). But extracted events may not be easy to be mapped to events for a particular knowledge base.
Distant supervision have been used in relation extraction for automatically labeling training data (Mintz et al., 2009; Hinton et al., 2012; Krause et al., 2012; Krishnamurthy and Mitchell, 2012; Berant et al., 2013; Surdeanu et al., 2012; Zeng et al., 2015). But DS for RE cannot directly use for EE. For the reasons that an event is more complicated than a relation and the task of EE is more difficult than RE. The best reported supervised RE and EE system got a F1-score of 88.0% (Wang et al., 2016) and 55.4% (Nguyen et al., 2016) respectively. Reschke et al. (2014) extended the distant supervision approach to fill slots in plane crash. However, the method can only extract arguments of one plane crash type and need flight number strings as input. In other words, the approach cannot extract whole event with different types automatically.
7 Conclusion and Future Work
In this paper, we present an approach to automatically label training data for EE. The experimental results show the quality of our large scale automatically labeled data is competitive with elaborately human-annotated data. Also, we provide a DMCNN-MIL model for this data as a baseline for further research. In the future, we will use the proposed automatically data labeling method to more event types and explore more models to extract events by using automatically labeled data.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
ACL 2017 Submission 350. Confidential Review Copy. DO NOT DISTRIBUTE.

- Strengths:  Improves over the state-of-the-art. Method might be applicable for other domains.  - Weaknesses:  Not much novelty in method.  Not quite clear if data set is general enough for other domains.  - General Discussion:  This paper describes a rule-based method for generating additional weakly labeled data for event extraction.  The method has three main stages.  First, it uses Freebase to find important slot fillers for matching sentences in Wikipedia (using all slot fillers is too stringent resulting in too few matches).  Next, it uses FrameNet to to improve reliability of labeling trigger verbs and to find nominal triggers.  Lastly, it uses a multi-instance learning to deal with the noisily generated training data.  What I like about this paper is that it improves over the state-of-the-art on a non-trival benchmark.  The rules involved don't seem too obfuscated, so I think it might be useful for the practitioner who is interested to improve IE systems for other domains.  On the other hand, some some manual effort is still needed, for example for mapping Freebase event types to ACE event types (as written in Section 5.3 line 578).  This also makes it difficult for future work to calibrate apple-to-apple against this paper.              Apart from this, the method also doesn't seem too novel.  Other comments:  - I'm also concern with the generalizability of this method to other   domains.  Section 2 line 262 says that 21 event types are selected   from Freebase.  How are they selected?  What is the coverage on the 33 event types in the ACE data.  - The paper is generally well-written although I have some   suggestions for improvement.              Section 3.1 line 316 uses  arguments liked time, location... .  If you mean roles or arguments, or maybe you want to use actual realizations of time and location as examples.  There are minor typos, for e.g. line 357 is missing a  that , but this is not a major concern I have for this paper.


Volatility Prediction using Financial Disclosures Sentiments with Word Embedding-based IR Models
Volatility prediction—an essential concept in financial markets—has recently been addressed using sentiment analysis methods. We investigate the sentiment of annual disclosures of companies in stock markets to forecast volatility. We specifically explore the use of recent Information Retrieval (IR) term weighting models that are effectively extended by related terms using word embeddings. In parallel to textual information, factual market data have been widely used as the mainstream approach to forecast market risk. We therefore study different fusion methods to combine text and market data resources. Our word embedding-based approach significantly outperforms state-ofthe-art methods. In addition, we investigate the characteristics of the reports of the companies in different financial sectors.
1 Introduction
Financial volatility is an essential indicator of instability and risk of a company, sector or economy. Volatility forecasting has gained considerable attention during the last three decades. In addition to using historic stock prices, new methods in this domain use sentiment analysis to exploit various text resources, such as financial reports (Kogan et al., 2009; Wang et al., 2013; Tsai and Wang, 2014; Nopp and Hanbury, 2015), news (Kazemian et al., 2014; Ding et al., 2015), message boards (Nguyen and Shirai, 2015), and earning calls (Wang and Hua, 2014).
An interesting resource of textual information are the companies’ annual disclosures, known as 10-K filing reports. They contain comprehensive information about the companies’ business as well as risk factors. Specifically, section Item 1A - Risk
Factors of the reports contains information about the most significant risks for the company. These reports are however long, redundant, and written in a style that makes them complex to process. Dyer et al. (2016) notes that: “10-K reports are getting more redundant and complex [...] (it) requires a reader to have 21.6 years of formal education to fully comprehend”. Dyer et al. also analyse the topics discussed in the reports and observe a constant increase over the years in both the length of the documents as well as the number of topics. They claim that the increase in length is not the result of economic factors but is due to verboseness and redundancy in the reports. They suggest that only the risk factors topic appears to be useful and informative to investors. Their analysis motivates us to study the effectiveness of the Risk Factors section for volatility prediction.
Our research builds on previous studies on volatility prediction and information analysis of 10-K reports using sentiment analysis (Kogan et al., 2009; Tsai and Wang, 2014; Wang et al., 2013; Nopp and Hanbury, 2015; Li, 2010; Campbell et al., 2014), in the sense that since the reports are long (average length of 5000 words), different approaches are required, compared with studies of sentiment analysis on short-texts. Such previous studies on 10-K reports have mostly used the data before 2008 and there is little work on the analysis of the informativeness and effectiveness of the recent reports with regards to volatility prediction. We will indeed show that the content of the reports changes significantly not only before and after 2008, but rather in a cycle of 3-4 years.
In terms of use of the textual content for volatility prediction, this paper shows that state-ofthe-art Information Retrieval (IR) term weighting models, which benefit from word embedding information, have a significantly positive impact on prediction accuracy. The most recent study on
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
the topic (Tsai and Wang, 2014) used related terms obtained by word embeddings to expand the lexicon of sentiment terms. In contrast, similar to Rekabsaz et al. (2016b), we define the weight of each lexicon term by extending it to the similar terms in the document. The significant improvement of this approach for document retrieval by capturing the importance of the terms motivates us to apply it on sentiment analysis. We extensively evaluate various state-of-the-art sentiment analysis methods to investigate the effectiveness of our approach.
In addition to text, factual market data (i.e. historical prices) provide valuable resources for volatility prediction e.g. in the framework of GARCH models (Engle, 1982). An emerging question is how to approach the combination of the textual and factual market information. We propose various methods for this issue and show the performance and characteristics of each.
The financial system covers a wide variety of industries, from daily-consumption products to space mission technologies. It is intuitive to consider that the factors of instability and uncertainty are different between the various sectors while similar inside them. We therefore also analyse the sentiment of the reports of each sector separately and study their particular characteristics.
The present study shows the value of information in the 10-K reports for volatility prediction. Our proposed approach to sentiment analysis significantly outperforms state-of-the-art methods (Kogan et al., 2009; Tsai and Wang, 2014; Wang et al., 2013). We also show that performance can be further improved by effectively combining textual and factual market information. In addition, we shed light on the effects of tailoring the analysis to each sector: despite the reasonable expectation that domain-specific training would lead to improvements, we show that our general model generalizes well and outperforms sector-specific trained models.
The remainder of the paper is organized as follows: in the next section, we review the state-ofthe-art and related studies. Section 3 formulates the problem, followed by a detailed explanation of our approach in Section 4. We explain the dataset and settings of the experiments in Section 5, followed by the full description of the experiments in Section 6. We conclude the work in Section 7.
2 Related Work
Market prediction has been attracting much attention in recent years in the natural language processing community. Kazemian et al. (2014) use sentiment analysis for predicting stock price movements in a simulated security trading system using news data, showing the advantages of the method against simple trading strategies. Ding et al. (2015) address a similar objective while using deep learning to extract and learn events in the news. Xie et al. (2013) introduce a semantic treebased model to represent news data for predicting stock price movement. Luss et al. (2015) also exploit news in combination with return prices to predict intra-day price movements. They use the Multi Kernel Learning (MKL) algorithm for combining the two features. The combination shows improvement in final prediction in comparison to using each of the features alone. Motivated by this study, we investigate the performance of the MKL algorithm as one of the methods to combine the textual with non-textual information. Other data resources, such as stocks’ message boards, are used by Nguyen and Shirai (2015) to study topic modelling for aspect-based sentiment analysis. Wang and Hua (2014) investigate the sentiment of the transcript of earning calls for volatility prediction using the Gaussian Copula regression model.
While the mentioned studies use short-length texts (sentence or paragraph level), approaching long texts (document level) for market prediction is mainly based on n-gram bag of words methods. Nopp and Hanbury (2015) study the sentiment of banks’ annual reports to assess banking systems risk factors using a finance-specific lexicon, provided by Loughran and McDonald (2011), in both unsupervised and supervised manner.
More directly related to the informativeness of the 10-K reports for volatility prediction, Kogan et al. (2009) use a linear Support Vector Machine (SVM) algorithm on the reports published between 1996–2006. Wang et al. (2013) improve upon this by using the Loughran and McDonald (2011) lexicon, observing improvement in the prediction. Later, Tsai and Wang (2014) apply the same method as Wang et al. (2013) while additionally using word embedding to expand the financial lexicon. We reproduce all the methods in these studies, and show the advantage of our sentiment analysis approach.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
3 Problem Formulation
In this section, we formulate the volatility forecasting problem and the prediction objectives of our experiments. Similar to previous studies (Christiansen et al., 2012; Kogan et al., 2009; Tsai and Wang, 2014), volatility is defined as the natural log of the standard deviation of (adjusted) return prices in a window of τ days. This definition is referred to as standard volatility (Li and Hong, 2011) or realized volatility (Liu and TSE, 2013), defined as follows:
v[s,s+τ ] = ln √∑s+τt=s (rt − r̄)2 τ  (1) where rt is the return price and r̄ the mean of return prices. The return price is calculated by rt = ln(Pt)−ln(Pt−1), where Pt is the (adjusted) closing price of a given stock at the trading date t.
Given an arbitrary report i, we define a prediction label yki as the volatility of the stock of the reporting company in the kth quarter-sized window starting from the issue date of the report si:
yki = v[si+64(k−1),si+64k] (2)
Every quarter is considered as per convention, 64 working days, while the full year is assumed to have 256 working days.
We use 8 learners for labels y1 to y8. For brevity, unless otherwise mentioned, we report the volatility of the first year by calculating the mean of the first four quartiles after the publication of each report.
4 Methodology
We first describe our text sentiment analysis methods, followed by the features obtained from factual market data, and finally explain the methods to combine textual and market feature sets.
4.1 Sentiment Analysis
Similar to previous studies (Nopp and Hanbury, 2015; Wang et al., 2013), we extract the keyword set from a finance-specific lexicon (Loughran and McDonald, 2011) using the positive, negative, and uncertain groups, stemmed using the Porter stemmer. We refer to this keyword set as Lex. Tsai and Wang (2014) expanded this set by adding the top 20 related terms to each term to the original set. The related terms are obtained using the Word2Vec (Mikolov et al., 2013) model, built on
the corpus of all the reports, with Cosine similarity. We also use this expanded set in our experiments and refer to it as LexExt.
The following word weighting schemes are commonly used in Information Retrieval and we consider them as well in our study: TC : log(1 + tcdi(t))
TF : log(1+tcdi (t))
‖di‖
TFIDF : log(1+tcdi (t)) ‖di‖ log(1 + |di| df(t))
BM25 : (k+1)tfdi (t)
k+tfdi (t) , tfdi(t) =
tcdi (t)
(1−b)+b |di| avgdl
where tcdi(t) is the number of occurrences of keyword t in report i, ‖di‖ denotes the Euclidean norm of the keyword weights of the report, |di| is the length of the report (number of the words in the report), avgdl is the average document length, and finally k and b are parameters. For them, we use the settings used in previous studies (Rekabsaz et al., 2016b) i.e. k = 1.2 and b = 0.65.
In addition to the standard weighting schemes, we use state-of-the-art weighting methods in Information Retrieval (Rekabsaz et al., 2016b) which benefit directly from word embedding models: They exploit similarity values between words provided by the word embedding model into the weighting schemes by extending the weight of each lexicon keyword with its similar words:
t̂cdi(t) = tcdi(t) + ∑
t′∈R(t)
sim(t, t′)tcdi(t ′) (3)
where R(t) is the list of similar words to the keyword t, and sim(t, t′) is the Cosine similarity value between the vector representations of the words t and t′. As previously suggested by Rekabsaz et al. (2016a, 2017), we use the Cosine similarity function with threshold 0.70 for selecting the set R(t) of similar words.
We define the extended versions of the standard weighting schemes as T̂C, T̂F, T̂FIDF, and B̂M25 by replacing tcdi(t) with t̂cdi(t) in each of the schemes.
The feature vector generated by the weights of the Lex or LexExt lexicons is highly sparse, as the number of dimensions is larger than the number of data-points. We therefore reduce the dimensions by applying Principle Component Analysis (PCA). Our initial experiments show 400 dimen-
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
sion as the optimum by trying on a range of dimensions from 50 to 1000.
Given the final feature vector x with l dimensions, we apply SVM as a well-known method for training both regression and classification methods. Support Vector Regression (Drucker et al., 1997) formulates the training as the following optimization problem:
min w∈IRl
1 2 ‖w‖2+C N N∑ i=1 max(0, ‖yi − f(xi;w)‖ − )
(4) Similar to previous studies (Tsai and Wang, 2014; Kogan et al., 2009), we set C = 1.0 and = 0.1. To solve the above problem, the function f can be re-parametrized in terms of a kernel function K with weights αi:
f(xi;w) = N∑ i=1 αiK(xi, x) (5)
The kernel can be considered as a (similarity) function between the feature vector of the document and vectors of all the other documents. Our initial experiments showed better performance of the Radial Basis Function (RBF) kernel in comparison to linear and cosine kernels and is therefore used in this paper.
4.2 Market Features
In addition to textual features, we define three features using the factual market data and historical prices—referred to as market features—as follows: Current Volatility is calculated on the window of one quartile before the issue date of the report: v[si−64,si]. GARCH (Bollerslev, 1986) is a common econometric time-series model used for predicting stock price volatility. Due to lack of space, the implementation details are moved to supplementary materials. Sector is the sector that the corresponding company of the report belongs to, namely energy (ene), basic industries (ind), finance (fin), technology (tech), miscellaneous (misc), consumer nondurables (n-dur), consumer durables (dur), capital goods (capt), consumer services (serv), public utilities (pub), and health care (hlth)1. The feature is converted to numerical representation using onehot encoding.
1We follow NASDAQ categorization of sectors.
4.3 Feature Fusion
To combine the text and market feature sets, the first approach, used also in previous studies ((Kogan et al., 2009; Wang et al., 2013)) is simply joining all the features in one feature space. In the context of multi-model learning, the method is referred to as early fusion.
In contrast, late fusion approaches first learn a model on each feature set and then use/learn a meta model to combine their results. As our second approach, we use stacking (Wolpert, 1992), a special case of late fusion. In stacking, we first split the training set into two parts (70%-30% portions). Using the first portion, we train separate machine learning models for each of the text and market feature sets. Next, we predict labels of the second portion with the trained models and finally train another model to capture the combinations between the outputs of the base models. In our experiments, the final model is always trained with SVM with RBF kernel.
Stacking is computationally inexpensive. However, due to the split of the training set, the base models or the meta model may suffer from lack of training data. A potential approach to learn both the feature sets in one model is the MKL method.
The MKL algorithm (also called intermediate fusion (Noble et al., 2004)) extends the kernel of the SVM model by learning (simultaneous to the parameter learning) an optimum combination of several kernels. The MKL problem as formulated in Lanckriet et al. (2004) add the following criterion to Eq. 5 for kernel learning:
K∗ = ∑ i diKi where ∑ i di = 1, di ≥ 0 (6)
where Ki is a predefined kernel. Gönen and Alpaydın (2011) mention two uses of MKL: learning the optimum kernel in SVM, and combining multiple modalities (feature sets) via each kernel.
However, the optimization can be computationally challenging. We use the mklaren method (Stražar and Curk, 2016) which has linear complexity in the number of data instances and kernels. It has been shown to outperform recent multi kernel approximation approaches. We use RBF kernels for both the text and market feature sets.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
5 Experiment Setup
In this section, we first describe the data, followed by introducing the baselines. We report the parameters applied in various algorithms and describe the evaluation metrics.
Dataset We download the reports of companies of the U.S. stock markets from 2006 to 2015 from the U.S. Securities and Exchange Commission (SEC) website2. We remove HTML tags and extract the text parts. We extract the Risk Factors section using term matching heuristics. Finally, the texts are stemmed using the Porter stemmer. We calculate the volatility values (Eq 1) and the volatility of the GARCH model based on the stock prices, collected from the Yahoo website. We filter the volatility values greater/smaller than the mean plus/minus three times the standard deviation of all the volatility values3.
Baselines GARCH: although the GARCH model is of market factual information, we use it as a baseline to compare the effectiveness of text-based methods with mainstream approaches.
Market: uses all the market features. For both the GARCH and Market baselines, we use an SVM learner with RBF kernel.
Wang et al. (2013): they use the Lex keyword set with TC weighting scheme and the SVM method. They combine the textual features with current volatility using the early fusion method.
Tsai et al. (2014): similar to Wang et al. (2013), while they use the LexExt keyword set.
Evaluation Metrics As a common metric in volatility prediction, we use the r2 metric (square of the correlation coefficient) for evaluation:
r2 =  ∑ni=1(ŷi − ¯̂y)(yi − ȳ)√∑n i=1(ŷi − ¯̂y)2 √∑n i=1(yi − ȳ)2 2 (7) where ŷi is the predicted value, yi denotes the labels and ȳ, their mean. The r2 metric indicates the proportion of variance in the labels explained by the prediction. The measure is close to 1 when the predicted values can explain a large proportion of the variability in the labels and 0 when it fails to explain the labels’ variabilities. An alternative metric, used in previous studies (Wang et al.,
2https://www.sec.gov 3The complete dataset is available in anonymousurl
20 06 20 07 20 08 20 09 20 10 20 11 20 12 20 13 20 14 20 15
2015 2014 2013 2012 2011 2010 2009 2008 2007 2006
0.8
0.4
0.0
0.4
0.8
(a)
20 06 20 07 20 08 20 09 20 10 20 11 20 12 20 13 20 14
0.0
0.1
0.2
0.3
0.4
0.5
r 2
(b)
Figure 1: (a) Cosine similarity between the centroid vectors of the years. (b) Volatility prediction performance when using reports from the specified year to 2015
2013; Tsai and Wang, 2014; Kogan et al., 2009) is Mean Squared Error MSE = ∑ i(ŷi − yi)2/n. However, especially when comparing models, applied on different test sets (e.g. performance of first quartile with second quartile), r2 has better interpretability since it is independent of the scale of y. We use r2 in all the experiments while the MSE measure is reported only when the models are evaluated on the same test set.
6 Experiments and Results
In this section, first we analyse the contents of the reports, followed by studying our sentiment analysis methods for volatility prediction. Finally, we investigate the effect of sentiment analysis of the reports in different industry sectors.
6.1 Content Analysis of 10-K Reports
Let us start our experiment with observing changes in the feature vectors of the reports over the years. To compare them, we use the state-ofthe-art sentiment analysis method, introduced by Tsai and Wang (2014). We first represent the feature vector of each year by calculating the centroid (element-wise mean) of the feature vectors of all reports published that year and then calculate the Cosine similarity of each pair of centroid vectors, for the years 2006–2015.
Figure 1a shows the similarity heat-map for each pair of the years. We observe a high similarity between three ranges of years: 2006–2008, 2009–2011, and 2012–2015. These considerable differences between the centroid reports in years across these three groups hints at probable issues when using the data of the older years for the more recent ones.
To validate this, we apply 5-fold cross valida-
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Table 1: Performance of sentiment analysis methods for the first year.
Component Method Text Text+Market (r2) (MSE) (r2) (MSE)
Weighting Schema (+Stacking)
B̂M25 0.439 0.132 0.527 0.111 BM25 0.433 0.136 0.523 0.114 T̂C 0.427 0.136 0.517 0.115 TC 0.425 0.137 0.521 0.114 ̂TFIDF 0.301 0.166 0.502 0.118 TFIDF 0.264 0.189 0.497 0.119 T̂F 0.218 0.190 0.495 0.120 TF 0.233 0.200 0.495 0.120
Feature Fusion (+B̂M25)
Stacking - - 0.527 0.111 MKL - - 0.488 0.126 Early Fusion - - 0.473 0.125
tion, first on all the data (2006–2015), and then on smaller sets by dropping the oldest year i.e. the next subsets use the reports 2007–2015, 2008– 2015 and so forth. The results of the r2 measure are shown in Figure 1b. We observe that by dropping the oldest years one by one (from left to right in the figure), the performance starts improving. We argue that this improvement is due to the reduction of noise in data, noise caused by conceptual drifts in the reports as also mentioned by Dyer et al. (2016). In fact, although in machine learning in general using more data results in better generalization of the model and therefore better prediction, the reports of the older years introduce noise.
As shown, the most coherent and largest data consists of the subset of the reports published between 2012 to 2015. This subset is also the most recent cluster and presumably more similar to the future reports. Therefore, in the following, we only use this subset, which consists of 3892 reports, belonging to 1323 companies.
6.2 Volatility Prediction
Given the dataset of the 2012–2015 reports, we try all combinations of different term weighting schemes using the LexExt keyword set. All weighting schemes are then combined with the market features with the introduced fusion methods. The prediction is done with 5-fold cross validation. The averages of the results of the first four quartiles (first year) are reported in Table 1. To make showing the results tractable, we use the best fusion (stacking) for the weighting schemes and the best scheme (B̂M25) for fusions.
Regarding the weighting schemes, B̂M25, BM25, and T̂C show the best results. In general, the extended schemes (with hat) improve upon their normal forms. For the feature fusion meth-
Table 2: Performance of the methods using 5-fold cross validation.
Method (r2) (MSE) GARCH 0.280 0.170
Text Wang (2013) 0.345 0.154 Tsai (2014) 0.395 0.142 Our method 0.439 0.132 Market 0.485 0.122 Text+Market Wang (2013) 0.499 0.118 Tsai (2014) 0.484 0.122 Our method 0.527 0.111
ods, stacking outperforms the other approaches in both evaluation measures. MKL however has better performance than early fusion while it has the highest computational complexity among the methods. Based on these results, as our best performing approach in the remainder of the paper, we use B̂M25 (with LexExt set), reduced to 400 dimensions and stacking as the fusion method. Table 2 summarizes the results of our best performing method compared with previously existing methods. Our method outperforms all state-ofthe-art methods both when using textual features only as well as a combination of textual and market features.
Let us now take a closer look on the changes in the performance of the prediction in time. The results of 5-fold cross validation for both tasks on the dataset of the reports, published between 2012–2015 are shown in Figure 2a. The X-axes show eight quartiles after the publication date of the report. For comparison, the GARCH and only market features are depicted with dashed lines.
As shown, the performance of the GARCH method as well as that using only market features (Market) decrease faster in the later quartiles since the historical prices used for prediction become less relevant as time goes by. Using only text features (Text), we see a roughly similar performance between the first four quartiles (first year), while the performance, in general, slightly decreases in the second year. By combining the textual and market features (Text+Market), we see a consistent improvement in comparison to each of them alone. In comparison to using only market features, the combination of the features shows more stable results in the later quartiles. These results support the informativeness of the 10-K reports to more effectively foreseen volatility in long-term
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 0.0
0.1
0.2
0.3
0.4
0.5
0.6
r 2 Text+Market
Text Market GARCH
(a) CV 2013 2014 2015
0.0
0.1
0.2
0.3
0.4
0.5
0.6
r 2
Text Text+Market
(b)
en e in d fin te ch m is c ndu r ca pt du r se rv pu b hl th
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 r 2
Text Text+Market
(c)
Figure 2: (a) Performance of our approach on 8 quartiles using the Text and Text+Market feature sets. The dashed lines show the market-based baselines. (b) Performance of volatility prediction of each year given the past data. The hashed areas show corresponding baselines. (c) Performance per sector. Abbreviations are defined in Section 4.2
windows. While the above experiments are based on cross-validation, for the sake of completeness it is noteworthy to consider the scenarios of realworld applications where the future prediction is based on past data. We therefore design three experiments by considering the reports published in 2013, 2014, and 2015 as test set and the reports published before each year as training set (only 2012, 2012–2013, and 2012–2014 respectively). The results of predicting the reports of each year together with the cross validation scenario (CV) are shown in Figure 2b. While the performance becomes slightly worse in the target years 2013 and 2015, in general the combination of textual and market features can explain approximately half of volatility in the financial system.
6.3 Sectors
Corporations in the same sector share not only similar products or services but also risks and instability factors. Considering the sentiment of the financial system as a homogeneous body may neglect the specific factors of each sector. We therefore set out to investigate the existence and nature of these differences.
We start by observing the prediction performance on different sectors: We use our method from the previous section, but split the test set across sectors and plot the results in Figure 2c. The hashed areas indicate the GARCH and Market baselines for the Text and Text+Market feature sets, respectively. We observe considerable differences between the performance of the sectors, especially when using only sentiment analysis methods (i.e. only text features).
Table 3: Number of reports per sectors
ene ind hlth fin tech pub 187 160 305 847 408 217
n-dur dur capt serv misc 151 115 255 639 153
Given these differences and also the probable similarities between the risk factors of the reports in the same sector, a question immediately arises: can training different models for different sectors improve the performance of prediction?
To answer it, for each sector, we train a model using only the subset of the reports in that sector and use 5-fold validation to observe performance. We refer to these models as sector-specific in contrast to the general model, trained on all the data. Figures 3a and 3b compare their results: we can see that the sector-specific bars are lower than the general model ones. This is to some extent surprising, as one would expect that domainspecific training would improve the performance of sentiment analysis in text. However, we need to consider the size of the training set. By training on each sector we have reduced the size of our training sets to those reported in Table 3. To verify the effect of the size of training data, we train a sector-agnostic model for each sector. Each sector-agnostic model is trained by random sampling of a training set of the same size as the set available for its sector from all the reports, but evaluated–similar to sector-specific models–on the test set of the sector. Figures 3a and 3b also plot the results of the sector-agnostic models.
The large performance differences between
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
en e in d fin te ch m is c ndu
r ca pt du r se rv pu b hl th
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 r 2
Sector-agnostic Sector-specific General model
(a) Text
en e in d fin te ch m is c ndu
r ca pt du r se rv pu b hl th
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 r 2
Sector-agnostic Sector-specific General model
(b) Text+Market
Figure 3: Results when retraining on sector-specific subsets versus the general model and versus subsets of the same size but sector-agnostic. The hashed area in (a) indicates the GARCH and in (b) the Market baseline.
sector-agnostic and -specific show the existence of particular risk factors in each sector and their importance. Results also confirm the hypothesis that the data for training in each sector is simply too small, and as additional data is accumulated, we can further improve on the results by training on different sectors independently.
We continue by examining some examples of essential terms in sectors. To address this, we have to train a linear regression method on all the reports of each sector, without using any dimensionality reduction. Linear regression without dimensionality reduction has the benefit of interpretability: the coefficient of each feature (i.e. term in the lexicon) can be seen as its importance with regards to volatility prediction. After training, we observe that some keywords e.g. crisis, or delist constantly have high coefficient values in the sector-specific as well as general model. However, some keywords are particularly weighted high in specificsector models.
For instance, the keyword fire has a high coefficient in the energy sector, but very low in the others. The reason is due to the problem of ambiguity i.e. in the energy sector, fire is widely used to refer to explosion e.g. ‘fire and explosion hazards’ while in the lexicon, it is stemmed from firing and fired: the act of dismissing from a job. This later sense of word is however weighted as a low risk-sensitive keyword in the other sectors. Such an ambiguity can indeed be mitigated by sectorspecific models since the variety of the words’ senses are more restricted inside each sector. Another example is an interesting observation on the word beneficial. The word is introduced as a positive sentiment in the lexicon while it gains highly
negative sentiments in some sectors (health care, and basic industries). Investigating in the reports, we observe the broad use of the expression ‘beneficial owner’ which is normally followed by riskfull sentences since the beneficial owners can potentially influence shareholders’ decision power.
7 Conclusion
In this work, we studied the sentiment of recent 10-K annual disclosures of companies in stock markets for forecasting volatility. Our bag-ofwords sentiment analysis approach benefits from state-of-the-art models in information retrieval which use word embeddings to extend the weight of the terms to the similar terms in the document. Additionally, we explored fusion methods to combine the text features with factual market features, achieved from historical prices i.e. GARCH prediction model, and current volatility. In both cases, our approach outperforms state-ofthe-art volatility prediction methods with 10-K reports and demonstrates the effectiveness of sentiment analysis in long-term volatility forecasting.
In addition, we studied the characteristics of each individual sector with regard to risk-sensitive terms. Our analysis shows that reports in same sectors considerably share particular risk and instability factors. However, despite expectations, training different models on different sectors does not improve performance compared to the general model. We traced this to the size of the available data in each sector, and show that there are still benefits in considering sectors, which could be further explored in the future as more data becomes available.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths:  - Weaknesses:  - General Discussion:  This paper investigates sentiment signals in  companies  annual 10-K filing reports to forecast volatility.   The authors evaluate information retrieval term weighting models which are seeded with a finance-oriented sentiment lexicon and expanded with word embeddings. PCA is used to reduce dimensionality before Support Vector Regression is applied for similarity estimation.  In addition to text-based features, the authors also use non-text-based market features (e.g. sector information and volatility estimates).  Multiple fusion methods to combine text features with market features are evaluated.  COMMENTS  It would be interesting to include two more experimental conditions, namely 1) a simple trigram SVM which does not use any prior sentiment lexica, and 2) features that reflect delta-IDFs scores for individual features. As an additional baseline, it would be good to see binary features.  This paper could corroborate your references:  https:  pdfs.semanticscholar.org 57d6 29615c19caa7ae6e0ef2163eebe3b272e65a.pdf