Volatility Prediction using Financial Disclosures Sentiments with Word Embedding-based IR Models
Volatility prediction—an essential concept in financial markets—has recently been addressed using sentiment analysis methods. We investigate the sentiment of annual disclosures of companies in stock markets to forecast volatility. We specifically explore the use of recent Information Retrieval (IR) term weighting models that are effectively extended by related terms using word embeddings. In parallel to textual information, factual market data have been widely used as the mainstream approach to forecast market risk. We therefore study different fusion methods to combine text and market data resources. Our word embedding-based approach significantly outperforms state-ofthe-art methods. In addition, we investigate the characteristics of the reports of the companies in different financial sectors.
1 Introduction
Financial volatility is an essential indicator of instability and risk of a company, sector or economy. Volatility forecasting has gained considerable attention during the last three decades. In addition to using historic stock prices, new methods in this domain use sentiment analysis to exploit various text resources, such as financial reports (Kogan et al., 2009; Wang et al., 2013; Tsai and Wang, 2014; Nopp and Hanbury, 2015), news (Kazemian et al., 2014; Ding et al., 2015), message boards (Nguyen and Shirai, 2015), and earning calls (Wang and Hua, 2014).
An interesting resource of textual information are the companies’ annual disclosures, known as 10-K filing reports. They contain comprehensive information about the companies’ business as well as risk factors. Specifically, section Item 1A - Risk
Factors of the reports contains information about the most significant risks for the company. These reports are however long, redundant, and written in a style that makes them complex to process. Dyer et al. (2016) notes that: “10-K reports are getting more redundant and complex [...] (it) requires a reader to have 21.6 years of formal education to fully comprehend”. Dyer et al. also analyse the topics discussed in the reports and observe a constant increase over the years in both the length of the documents as well as the number of topics. They claim that the increase in length is not the result of economic factors but is due to verboseness and redundancy in the reports. They suggest that only the risk factors topic appears to be useful and informative to investors. Their analysis motivates us to study the effectiveness of the Risk Factors section for volatility prediction.
Our research builds on previous studies on volatility prediction and information analysis of 10-K reports using sentiment analysis (Kogan et al., 2009; Tsai and Wang, 2014; Wang et al., 2013; Nopp and Hanbury, 2015; Li, 2010; Campbell et al., 2014), in the sense that since the reports are long (average length of 5000 words), different approaches are required, compared with studies of sentiment analysis on short-texts. Such previous studies on 10-K reports have mostly used the data before 2008 and there is little work on the analysis of the informativeness and effectiveness of the recent reports with regards to volatility prediction. We will indeed show that the content of the reports changes significantly not only before and after 2008, but rather in a cycle of 3-4 years.
In terms of use of the textual content for volatility prediction, this paper shows that state-ofthe-art Information Retrieval (IR) term weighting models, which benefit from word embedding information, have a significantly positive impact on prediction accuracy. The most recent study on
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
the topic (Tsai and Wang, 2014) used related terms obtained by word embeddings to expand the lexicon of sentiment terms. In contrast, similar to Rekabsaz et al. (2016b), we define the weight of each lexicon term by extending it to the similar terms in the document. The significant improvement of this approach for document retrieval by capturing the importance of the terms motivates us to apply it on sentiment analysis. We extensively evaluate various state-of-the-art sentiment analysis methods to investigate the effectiveness of our approach.
In addition to text, factual market data (i.e. historical prices) provide valuable resources for volatility prediction e.g. in the framework of GARCH models (Engle, 1982). An emerging question is how to approach the combination of the textual and factual market information. We propose various methods for this issue and show the performance and characteristics of each.
The financial system covers a wide variety of industries, from daily-consumption products to space mission technologies. It is intuitive to consider that the factors of instability and uncertainty are different between the various sectors while similar inside them. We therefore also analyse the sentiment of the reports of each sector separately and study their particular characteristics.
The present study shows the value of information in the 10-K reports for volatility prediction. Our proposed approach to sentiment analysis significantly outperforms state-of-the-art methods (Kogan et al., 2009; Tsai and Wang, 2014; Wang et al., 2013). We also show that performance can be further improved by effectively combining textual and factual market information. In addition, we shed light on the effects of tailoring the analysis to each sector: despite the reasonable expectation that domain-specific training would lead to improvements, we show that our general model generalizes well and outperforms sector-specific trained models.
The remainder of the paper is organized as follows: in the next section, we review the state-ofthe-art and related studies. Section 3 formulates the problem, followed by a detailed explanation of our approach in Section 4. We explain the dataset and settings of the experiments in Section 5, followed by the full description of the experiments in Section 6. We conclude the work in Section 7.
2 Related Work
Market prediction has been attracting much attention in recent years in the natural language processing community. Kazemian et al. (2014) use sentiment analysis for predicting stock price movements in a simulated security trading system using news data, showing the advantages of the method against simple trading strategies. Ding et al. (2015) address a similar objective while using deep learning to extract and learn events in the news. Xie et al. (2013) introduce a semantic treebased model to represent news data for predicting stock price movement. Luss et al. (2015) also exploit news in combination with return prices to predict intra-day price movements. They use the Multi Kernel Learning (MKL) algorithm for combining the two features. The combination shows improvement in final prediction in comparison to using each of the features alone. Motivated by this study, we investigate the performance of the MKL algorithm as one of the methods to combine the textual with non-textual information. Other data resources, such as stocks’ message boards, are used by Nguyen and Shirai (2015) to study topic modelling for aspect-based sentiment analysis. Wang and Hua (2014) investigate the sentiment of the transcript of earning calls for volatility prediction using the Gaussian Copula regression model.
While the mentioned studies use short-length texts (sentence or paragraph level), approaching long texts (document level) for market prediction is mainly based on n-gram bag of words methods. Nopp and Hanbury (2015) study the sentiment of banks’ annual reports to assess banking systems risk factors using a finance-specific lexicon, provided by Loughran and McDonald (2011), in both unsupervised and supervised manner.
More directly related to the informativeness of the 10-K reports for volatility prediction, Kogan et al. (2009) use a linear Support Vector Machine (SVM) algorithm on the reports published between 1996–2006. Wang et al. (2013) improve upon this by using the Loughran and McDonald (2011) lexicon, observing improvement in the prediction. Later, Tsai and Wang (2014) apply the same method as Wang et al. (2013) while additionally using word embedding to expand the financial lexicon. We reproduce all the methods in these studies, and show the advantage of our sentiment analysis approach.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
3 Problem Formulation
In this section, we formulate the volatility forecasting problem and the prediction objectives of our experiments. Similar to previous studies (Christiansen et al., 2012; Kogan et al., 2009; Tsai and Wang, 2014), volatility is defined as the natural log of the standard deviation of (adjusted) return prices in a window of τ days. This definition is referred to as standard volatility (Li and Hong, 2011) or realized volatility (Liu and TSE, 2013), defined as follows:
v[s,s+τ ] = ln √∑s+τt=s (rt − r̄)2 τ  (1) where rt is the return price and r̄ the mean of return prices. The return price is calculated by rt = ln(Pt)−ln(Pt−1), where Pt is the (adjusted) closing price of a given stock at the trading date t.
Given an arbitrary report i, we define a prediction label yki as the volatility of the stock of the reporting company in the kth quarter-sized window starting from the issue date of the report si:
yki = v[si+64(k−1),si+64k] (2)
Every quarter is considered as per convention, 64 working days, while the full year is assumed to have 256 working days.
We use 8 learners for labels y1 to y8. For brevity, unless otherwise mentioned, we report the volatility of the first year by calculating the mean of the first four quartiles after the publication of each report.
4 Methodology
We first describe our text sentiment analysis methods, followed by the features obtained from factual market data, and finally explain the methods to combine textual and market feature sets.
4.1 Sentiment Analysis
Similar to previous studies (Nopp and Hanbury, 2015; Wang et al., 2013), we extract the keyword set from a finance-specific lexicon (Loughran and McDonald, 2011) using the positive, negative, and uncertain groups, stemmed using the Porter stemmer. We refer to this keyword set as Lex. Tsai and Wang (2014) expanded this set by adding the top 20 related terms to each term to the original set. The related terms are obtained using the Word2Vec (Mikolov et al., 2013) model, built on
the corpus of all the reports, with Cosine similarity. We also use this expanded set in our experiments and refer to it as LexExt.
The following word weighting schemes are commonly used in Information Retrieval and we consider them as well in our study: TC : log(1 + tcdi(t))
TF : log(1+tcdi (t))
‖di‖
TFIDF : log(1+tcdi (t)) ‖di‖ log(1 + |di| df(t))
BM25 : (k+1)tfdi (t)
k+tfdi (t) , tfdi(t) =
tcdi (t)
(1−b)+b |di| avgdl
where tcdi(t) is the number of occurrences of keyword t in report i, ‖di‖ denotes the Euclidean norm of the keyword weights of the report, |di| is the length of the report (number of the words in the report), avgdl is the average document length, and finally k and b are parameters. For them, we use the settings used in previous studies (Rekabsaz et al., 2016b) i.e. k = 1.2 and b = 0.65.
In addition to the standard weighting schemes, we use state-of-the-art weighting methods in Information Retrieval (Rekabsaz et al., 2016b) which benefit directly from word embedding models: They exploit similarity values between words provided by the word embedding model into the weighting schemes by extending the weight of each lexicon keyword with its similar words:
t̂cdi(t) = tcdi(t) + ∑
t′∈R(t)
sim(t, t′)tcdi(t ′) (3)
where R(t) is the list of similar words to the keyword t, and sim(t, t′) is the Cosine similarity value between the vector representations of the words t and t′. As previously suggested by Rekabsaz et al. (2016a, 2017), we use the Cosine similarity function with threshold 0.70 for selecting the set R(t) of similar words.
We define the extended versions of the standard weighting schemes as T̂C, T̂F, T̂FIDF, and B̂M25 by replacing tcdi(t) with t̂cdi(t) in each of the schemes.
The feature vector generated by the weights of the Lex or LexExt lexicons is highly sparse, as the number of dimensions is larger than the number of data-points. We therefore reduce the dimensions by applying Principle Component Analysis (PCA). Our initial experiments show 400 dimen-
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
sion as the optimum by trying on a range of dimensions from 50 to 1000.
Given the final feature vector x with l dimensions, we apply SVM as a well-known method for training both regression and classification methods. Support Vector Regression (Drucker et al., 1997) formulates the training as the following optimization problem:
min w∈IRl
1 2 ‖w‖2+C N N∑ i=1 max(0, ‖yi − f(xi;w)‖ − )
(4) Similar to previous studies (Tsai and Wang, 2014; Kogan et al., 2009), we set C = 1.0 and = 0.1. To solve the above problem, the function f can be re-parametrized in terms of a kernel function K with weights αi:
f(xi;w) = N∑ i=1 αiK(xi, x) (5)
The kernel can be considered as a (similarity) function between the feature vector of the document and vectors of all the other documents. Our initial experiments showed better performance of the Radial Basis Function (RBF) kernel in comparison to linear and cosine kernels and is therefore used in this paper.
4.2 Market Features
In addition to textual features, we define three features using the factual market data and historical prices—referred to as market features—as follows: Current Volatility is calculated on the window of one quartile before the issue date of the report: v[si−64,si]. GARCH (Bollerslev, 1986) is a common econometric time-series model used for predicting stock price volatility. Due to lack of space, the implementation details are moved to supplementary materials. Sector is the sector that the corresponding company of the report belongs to, namely energy (ene), basic industries (ind), finance (fin), technology (tech), miscellaneous (misc), consumer nondurables (n-dur), consumer durables (dur), capital goods (capt), consumer services (serv), public utilities (pub), and health care (hlth)1. The feature is converted to numerical representation using onehot encoding.
1We follow NASDAQ categorization of sectors.
4.3 Feature Fusion
To combine the text and market feature sets, the first approach, used also in previous studies ((Kogan et al., 2009; Wang et al., 2013)) is simply joining all the features in one feature space. In the context of multi-model learning, the method is referred to as early fusion.
In contrast, late fusion approaches first learn a model on each feature set and then use/learn a meta model to combine their results. As our second approach, we use stacking (Wolpert, 1992), a special case of late fusion. In stacking, we first split the training set into two parts (70%-30% portions). Using the first portion, we train separate machine learning models for each of the text and market feature sets. Next, we predict labels of the second portion with the trained models and finally train another model to capture the combinations between the outputs of the base models. In our experiments, the final model is always trained with SVM with RBF kernel.
Stacking is computationally inexpensive. However, due to the split of the training set, the base models or the meta model may suffer from lack of training data. A potential approach to learn both the feature sets in one model is the MKL method.
The MKL algorithm (also called intermediate fusion (Noble et al., 2004)) extends the kernel of the SVM model by learning (simultaneous to the parameter learning) an optimum combination of several kernels. The MKL problem as formulated in Lanckriet et al. (2004) add the following criterion to Eq. 5 for kernel learning:
K∗ = ∑ i diKi where ∑ i di = 1, di ≥ 0 (6)
where Ki is a predefined kernel. Gönen and Alpaydın (2011) mention two uses of MKL: learning the optimum kernel in SVM, and combining multiple modalities (feature sets) via each kernel.
However, the optimization can be computationally challenging. We use the mklaren method (Stražar and Curk, 2016) which has linear complexity in the number of data instances and kernels. It has been shown to outperform recent multi kernel approximation approaches. We use RBF kernels for both the text and market feature sets.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
5 Experiment Setup
In this section, we first describe the data, followed by introducing the baselines. We report the parameters applied in various algorithms and describe the evaluation metrics.
Dataset We download the reports of companies of the U.S. stock markets from 2006 to 2015 from the U.S. Securities and Exchange Commission (SEC) website2. We remove HTML tags and extract the text parts. We extract the Risk Factors section using term matching heuristics. Finally, the texts are stemmed using the Porter stemmer. We calculate the volatility values (Eq 1) and the volatility of the GARCH model based on the stock prices, collected from the Yahoo website. We filter the volatility values greater/smaller than the mean plus/minus three times the standard deviation of all the volatility values3.
Baselines GARCH: although the GARCH model is of market factual information, we use it as a baseline to compare the effectiveness of text-based methods with mainstream approaches.
Market: uses all the market features. For both the GARCH and Market baselines, we use an SVM learner with RBF kernel.
Wang et al. (2013): they use the Lex keyword set with TC weighting scheme and the SVM method. They combine the textual features with current volatility using the early fusion method.
Tsai et al. (2014): similar to Wang et al. (2013), while they use the LexExt keyword set.
Evaluation Metrics As a common metric in volatility prediction, we use the r2 metric (square of the correlation coefficient) for evaluation:
r2 =  ∑ni=1(ŷi − ¯̂y)(yi − ȳ)√∑n i=1(ŷi − ¯̂y)2 √∑n i=1(yi − ȳ)2 2 (7) where ŷi is the predicted value, yi denotes the labels and ȳ, their mean. The r2 metric indicates the proportion of variance in the labels explained by the prediction. The measure is close to 1 when the predicted values can explain a large proportion of the variability in the labels and 0 when it fails to explain the labels’ variabilities. An alternative metric, used in previous studies (Wang et al.,
2https://www.sec.gov 3The complete dataset is available in anonymousurl
20 06 20 07 20 08 20 09 20 10 20 11 20 12 20 13 20 14 20 15
2015 2014 2013 2012 2011 2010 2009 2008 2007 2006
0.8
0.4
0.0
0.4
0.8
(a)
20 06 20 07 20 08 20 09 20 10 20 11 20 12 20 13 20 14
0.0
0.1
0.2
0.3
0.4
0.5
r 2
(b)
Figure 1: (a) Cosine similarity between the centroid vectors of the years. (b) Volatility prediction performance when using reports from the specified year to 2015
2013; Tsai and Wang, 2014; Kogan et al., 2009) is Mean Squared Error MSE = ∑ i(ŷi − yi)2/n. However, especially when comparing models, applied on different test sets (e.g. performance of first quartile with second quartile), r2 has better interpretability since it is independent of the scale of y. We use r2 in all the experiments while the MSE measure is reported only when the models are evaluated on the same test set.
6 Experiments and Results
In this section, first we analyse the contents of the reports, followed by studying our sentiment analysis methods for volatility prediction. Finally, we investigate the effect of sentiment analysis of the reports in different industry sectors.
6.1 Content Analysis of 10-K Reports
Let us start our experiment with observing changes in the feature vectors of the reports over the years. To compare them, we use the state-ofthe-art sentiment analysis method, introduced by Tsai and Wang (2014). We first represent the feature vector of each year by calculating the centroid (element-wise mean) of the feature vectors of all reports published that year and then calculate the Cosine similarity of each pair of centroid vectors, for the years 2006–2015.
Figure 1a shows the similarity heat-map for each pair of the years. We observe a high similarity between three ranges of years: 2006–2008, 2009–2011, and 2012–2015. These considerable differences between the centroid reports in years across these three groups hints at probable issues when using the data of the older years for the more recent ones.
To validate this, we apply 5-fold cross valida-
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Table 1: Performance of sentiment analysis methods for the first year.
Component Method Text Text+Market (r2) (MSE) (r2) (MSE)
Weighting Schema (+Stacking)
B̂M25 0.439 0.132 0.527 0.111 BM25 0.433 0.136 0.523 0.114 T̂C 0.427 0.136 0.517 0.115 TC 0.425 0.137 0.521 0.114 ̂TFIDF 0.301 0.166 0.502 0.118 TFIDF 0.264 0.189 0.497 0.119 T̂F 0.218 0.190 0.495 0.120 TF 0.233 0.200 0.495 0.120
Feature Fusion (+B̂M25)
Stacking - - 0.527 0.111 MKL - - 0.488 0.126 Early Fusion - - 0.473 0.125
tion, first on all the data (2006–2015), and then on smaller sets by dropping the oldest year i.e. the next subsets use the reports 2007–2015, 2008– 2015 and so forth. The results of the r2 measure are shown in Figure 1b. We observe that by dropping the oldest years one by one (from left to right in the figure), the performance starts improving. We argue that this improvement is due to the reduction of noise in data, noise caused by conceptual drifts in the reports as also mentioned by Dyer et al. (2016). In fact, although in machine learning in general using more data results in better generalization of the model and therefore better prediction, the reports of the older years introduce noise.
As shown, the most coherent and largest data consists of the subset of the reports published between 2012 to 2015. This subset is also the most recent cluster and presumably more similar to the future reports. Therefore, in the following, we only use this subset, which consists of 3892 reports, belonging to 1323 companies.
6.2 Volatility Prediction
Given the dataset of the 2012–2015 reports, we try all combinations of different term weighting schemes using the LexExt keyword set. All weighting schemes are then combined with the market features with the introduced fusion methods. The prediction is done with 5-fold cross validation. The averages of the results of the first four quartiles (first year) are reported in Table 1. To make showing the results tractable, we use the best fusion (stacking) for the weighting schemes and the best scheme (B̂M25) for fusions.
Regarding the weighting schemes, B̂M25, BM25, and T̂C show the best results. In general, the extended schemes (with hat) improve upon their normal forms. For the feature fusion meth-
Table 2: Performance of the methods using 5-fold cross validation.
Method (r2) (MSE) GARCH 0.280 0.170
Text Wang (2013) 0.345 0.154 Tsai (2014) 0.395 0.142 Our method 0.439 0.132 Market 0.485 0.122 Text+Market Wang (2013) 0.499 0.118 Tsai (2014) 0.484 0.122 Our method 0.527 0.111
ods, stacking outperforms the other approaches in both evaluation measures. MKL however has better performance than early fusion while it has the highest computational complexity among the methods. Based on these results, as our best performing approach in the remainder of the paper, we use B̂M25 (with LexExt set), reduced to 400 dimensions and stacking as the fusion method. Table 2 summarizes the results of our best performing method compared with previously existing methods. Our method outperforms all state-ofthe-art methods both when using textual features only as well as a combination of textual and market features.
Let us now take a closer look on the changes in the performance of the prediction in time. The results of 5-fold cross validation for both tasks on the dataset of the reports, published between 2012–2015 are shown in Figure 2a. The X-axes show eight quartiles after the publication date of the report. For comparison, the GARCH and only market features are depicted with dashed lines.
As shown, the performance of the GARCH method as well as that using only market features (Market) decrease faster in the later quartiles since the historical prices used for prediction become less relevant as time goes by. Using only text features (Text), we see a roughly similar performance between the first four quartiles (first year), while the performance, in general, slightly decreases in the second year. By combining the textual and market features (Text+Market), we see a consistent improvement in comparison to each of them alone. In comparison to using only market features, the combination of the features shows more stable results in the later quartiles. These results support the informativeness of the 10-K reports to more effectively foreseen volatility in long-term
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 0.0
0.1
0.2
0.3
0.4
0.5
0.6
r 2 Text+Market
Text Market GARCH
(a) CV 2013 2014 2015
0.0
0.1
0.2
0.3
0.4
0.5
0.6
r 2
Text Text+Market
(b)
en e in d fin te ch m is c ndu r ca pt du r se rv pu b hl th
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 r 2
Text Text+Market
(c)
Figure 2: (a) Performance of our approach on 8 quartiles using the Text and Text+Market feature sets. The dashed lines show the market-based baselines. (b) Performance of volatility prediction of each year given the past data. The hashed areas show corresponding baselines. (c) Performance per sector. Abbreviations are defined in Section 4.2
windows. While the above experiments are based on cross-validation, for the sake of completeness it is noteworthy to consider the scenarios of realworld applications where the future prediction is based on past data. We therefore design three experiments by considering the reports published in 2013, 2014, and 2015 as test set and the reports published before each year as training set (only 2012, 2012–2013, and 2012–2014 respectively). The results of predicting the reports of each year together with the cross validation scenario (CV) are shown in Figure 2b. While the performance becomes slightly worse in the target years 2013 and 2015, in general the combination of textual and market features can explain approximately half of volatility in the financial system.
6.3 Sectors
Corporations in the same sector share not only similar products or services but also risks and instability factors. Considering the sentiment of the financial system as a homogeneous body may neglect the specific factors of each sector. We therefore set out to investigate the existence and nature of these differences.
We start by observing the prediction performance on different sectors: We use our method from the previous section, but split the test set across sectors and plot the results in Figure 2c. The hashed areas indicate the GARCH and Market baselines for the Text and Text+Market feature sets, respectively. We observe considerable differences between the performance of the sectors, especially when using only sentiment analysis methods (i.e. only text features).
Table 3: Number of reports per sectors
ene ind hlth fin tech pub 187 160 305 847 408 217
n-dur dur capt serv misc 151 115 255 639 153
Given these differences and also the probable similarities between the risk factors of the reports in the same sector, a question immediately arises: can training different models for different sectors improve the performance of prediction?
To answer it, for each sector, we train a model using only the subset of the reports in that sector and use 5-fold validation to observe performance. We refer to these models as sector-specific in contrast to the general model, trained on all the data. Figures 3a and 3b compare their results: we can see that the sector-specific bars are lower than the general model ones. This is to some extent surprising, as one would expect that domainspecific training would improve the performance of sentiment analysis in text. However, we need to consider the size of the training set. By training on each sector we have reduced the size of our training sets to those reported in Table 3. To verify the effect of the size of training data, we train a sector-agnostic model for each sector. Each sector-agnostic model is trained by random sampling of a training set of the same size as the set available for its sector from all the reports, but evaluated–similar to sector-specific models–on the test set of the sector. Figures 3a and 3b also plot the results of the sector-agnostic models.
The large performance differences between
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
en e in d fin te ch m is c ndu
r ca pt du r se rv pu b hl th
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 r 2
Sector-agnostic Sector-specific General model
(a) Text
en e in d fin te ch m is c ndu
r ca pt du r se rv pu b hl th
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 r 2
Sector-agnostic Sector-specific General model
(b) Text+Market
Figure 3: Results when retraining on sector-specific subsets versus the general model and versus subsets of the same size but sector-agnostic. The hashed area in (a) indicates the GARCH and in (b) the Market baseline.
sector-agnostic and -specific show the existence of particular risk factors in each sector and their importance. Results also confirm the hypothesis that the data for training in each sector is simply too small, and as additional data is accumulated, we can further improve on the results by training on different sectors independently.
We continue by examining some examples of essential terms in sectors. To address this, we have to train a linear regression method on all the reports of each sector, without using any dimensionality reduction. Linear regression without dimensionality reduction has the benefit of interpretability: the coefficient of each feature (i.e. term in the lexicon) can be seen as its importance with regards to volatility prediction. After training, we observe that some keywords e.g. crisis, or delist constantly have high coefficient values in the sector-specific as well as general model. However, some keywords are particularly weighted high in specificsector models.
For instance, the keyword fire has a high coefficient in the energy sector, but very low in the others. The reason is due to the problem of ambiguity i.e. in the energy sector, fire is widely used to refer to explosion e.g. ‘fire and explosion hazards’ while in the lexicon, it is stemmed from firing and fired: the act of dismissing from a job. This later sense of word is however weighted as a low risk-sensitive keyword in the other sectors. Such an ambiguity can indeed be mitigated by sectorspecific models since the variety of the words’ senses are more restricted inside each sector. Another example is an interesting observation on the word beneficial. The word is introduced as a positive sentiment in the lexicon while it gains highly
negative sentiments in some sectors (health care, and basic industries). Investigating in the reports, we observe the broad use of the expression ‘beneficial owner’ which is normally followed by riskfull sentences since the beneficial owners can potentially influence shareholders’ decision power.
7 Conclusion
In this work, we studied the sentiment of recent 10-K annual disclosures of companies in stock markets for forecasting volatility. Our bag-ofwords sentiment analysis approach benefits from state-of-the-art models in information retrieval which use word embeddings to extend the weight of the terms to the similar terms in the document. Additionally, we explored fusion methods to combine the text features with factual market features, achieved from historical prices i.e. GARCH prediction model, and current volatility. In both cases, our approach outperforms state-ofthe-art volatility prediction methods with 10-K reports and demonstrates the effectiveness of sentiment analysis in long-term volatility forecasting.
In addition, we studied the characteristics of each individual sector with regard to risk-sensitive terms. Our analysis shows that reports in same sectors considerably share particular risk and instability factors. However, despite expectations, training different models on different sectors does not improve performance compared to the general model. We traced this to the size of the available data in each sector, and show that there are still benefits in considering sectors, which could be further explored in the future as more data becomes available.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
- Strengths:  - Weaknesses:  - General Discussion:  This paper investigates sentiment signals in  companies  annual 10-K filing reports to forecast volatility.   The authors evaluate information retrieval term weighting models which are seeded with a finance-oriented sentiment lexicon and expanded with word embeddings. PCA is used to reduce dimensionality before Support Vector Regression is applied for similarity estimation.  In addition to text-based features, the authors also use non-text-based market features (e.g. sector information and volatility estimates).  Multiple fusion methods to combine text features with market features are evaluated.  COMMENTS  It would be interesting to include two more experimental conditions, namely 1) a simple trigram SVM which does not use any prior sentiment lexica, and 2) features that reflect delta-IDFs scores for individual features. As an additional baseline, it would be good to see binary features.  This paper could corroborate your references:  https:  pdfs.semanticscholar.org 57d6 29615c19caa7ae6e0ef2163eebe3b272e65a.pdf