## Enhancing Financial Market Predictions: Causality-Driven Feature Selection

Wenhao Liang, Zhengyang Li, and Weitong Chen ⋆

School of Computer and Mathematical Sciences Adelaide University, Adelaide, Australia

Abstract. This paper introduces FinSen dataset that revolutionizes financial market analysis by integrating economic and financial news articles from 197 countries with stock market data. The dataset's extensive coverage spans 15 years from 2007 to 2023 with temporal information, offering a rich, global perspective 160,000 records on financial market news. Our study leverages causally validated sentiment scores and LSTM models to enhance market forecast accuracy and reliability. Utilizing the FinSen dataset, we introduce an innovative Focal Calibration Loss, reducing Expected Calibration Error (ECE) to 3.34% with the DAN 3 model. This is not only improves prediction accuracy but also aligns probabilistic forecasts closely with real outcomes, crucial for financial sector where predicted probability is paramount. Our approach demonstrates the effectiveness of combining sentiment analysis with precise calibration techniques for trustworthy financial forecasting where the cost of misinterpretation can be high.

Keywords: Financial News Dataset · Time-Series · Calibration

## 1 Introduction

Forecasting stock market volatility [15,42,48] presents a paramount yet formidable challenge within the financial sector, attributed to the market's non-linear and inherently volatile nature. The capacity to predict market trends not only underpins investment strategies but also informs economic forecasting and risk management practices. The complexity of achieving accurate predictions is exacerbated by the market's sensitivity to temporal factors, with prices reflecting the latest events and sentiments [3,44]. This sensitivity leads to rapid and often unpredictable fluctuations, influenced by a confluence of factors including economic indicators, global events, and investor behaviors.

Traditional models [4,12,14,17,32], which predominantly rely on historical numeric data, encounter limitations in capturing the multifaceted dynamics of market volatility. Moreover, a notable gap exists in public datasets [25,31,39,43,53], particularly in their lack of precise temporal information, complicating the pursuit of unknown factors-referred to as alpha or beta that significantly impact

⋆ Corresponding Author. Email: Weitong.Chen@Adelaide.edu.au

financial market volatility. The Efficient Market Hypothesis (EMH) [13] is one of the key theories in behavioral finance, which set the stage for considering alternative indicators like sentiment in market analysis. This context in Table 1 raises the pivotal research question: Is the contribution of sentiment from news events a reliable predictor of market volatility?

Table 1. High-Tech Companies News and Movement (Mov.) from June 2022 to June 2023. Sentiments are represented by positive and negative.

| Date       | Symbol Mov.(%) News Events   |   Symbol Mov.(%) News Events | Symbol Mov.(%) News Events                        | Sent.   |
|------------|------------------------------|------------------------------|---------------------------------------------------|---------|
| 2022/7/27  | MSFT                         |                         6.68 | Microsoft Cloud strength drives fourth quarter.   | Pos.    |
| 2022/11/30 | MSFT                         |                         6.16 | ChatGPT has launched on November 30, 2022.        | Pos.    |
| 2023/1/20  | GOOG                         |                        -5.72 | Eliminating about 12000 jobs, or 6% of workforce. | Neg.    |
| 2022/11/10 | META                         |                       -10.25 | Mark Zuckerberg's Meta cuts 11000 jobs.           | Neg.    |
| 2022/10/28 | AAPL                         |                         7.56 | Titan's sales and profits exceeded expectations.  | Pos.    |

Antweiler and Frank's [2] seminal study analyzed the influence of internet stock message boards on market volatility using traditional statistical methods, which struggled to capture market dynamics' non-linearity. Conversely, Neural Networks (NNs) have shown superior capability in detecting the non-linear nature of stock market volatility [15,48]. Furthermore, the incorporation of sentiment analysis from unstructured financial news sources allows neural networks (NNs) to comprehend more nuanced market sentiments, enhancing their predictive accuracy in financial markets. Subsequent research, such as Li et al.'s [28] examination of the effects of news sentiment on stock prices, illustrates the benefits of merging NNs with sentiment analysis to refine prediction accuracy. Neural networks excel at identifying patterns and correlations within datasets, yet they often overlook the temporal sensitivity inherent in stock market data. Contemporarily, deep learning techniques, including LSTMs [22,34] and Transformerbased models like BERT [20], which have been refined by incorporating sentiment analysis for superior context and nuance understanding. However, they do not inherently understand the causality behind these correlations that make NNs model applied with less interpretability. In addition, modern NNs are poorly calibrated even though they have higher accuracy [19,36]. In critical decision-making fields such as financial investment [30], a poor calibrated model can potentially lead to lower confidence on prediction, making an unreliable investment in realworld scenarios. We consider a truth-worthy model is much more significant than other SOTA model in the context of financial prediction.

To close the gaps above, our work publicly presents a novel financial dataset, FinSen, notable for its temporal attributes. Subsequently, utilizing the FinSen dataset, we establish a demonstrable causal relationship between market sentiment scores and volatility within the S&amp;P 500. Thirdly, capitalizing on the established causality, we enrich LSTM models by integrating causal-validated sentiment scores, yielding impressive results and interpretability to financial prediction model. Lastly, we propose a novel loss function, Focal Calibration Loss,

to further enhance trustworthiness and dramatically improves the Expected Calibration Error (ECE) to 3.34% upon the SOTA calibration methods, signifying a close alignment of probabilistic predictions with actual market outcomes.

## 2 Related Works

## 2.1 Related Sentiment Datasets on Stock Market Prediction

The predictive modeling of stock prices has been a focal point of many studies, leveraging a wide array of data characteristics. Among these, sentiment datasets derived from financial news have been identified as particularly valuable for capturing the market's mood and its potential impact on stock volatility. The Financial PhraseBank [31] contains sentences from financial news categorized into positive, negative, and neutral sentiments, specifically designed for financial context. Reuters-21578 News Dataset [25] is a foundational benchmark for text classification tasks, originating from the Reuters newswire service in the year 1987. Comprising a total of 21,578 documents, but it is no longer available online for research purposes due to copyright issues. StockTwits Dataset [5,21,37,38] includes messages from the StockTwits platform, tagged with sentiments by users, providing a crowd-sourced sentiment analysis on stocks . TweetFinSent [39] is dataset of stock sentiments on Twitter, designed for sentiment analysis in the stock market domain, with a focus on "meme stocks.", which is distinctive due to its expert-annotated nature. PsychSignal Dataset [43] gathers data from multiple sources, including Twitter, StockTwits, and various of market sentiments across different social media and online platforms. AG News [53] is commonly used in text classification tasks and consists of news articles from the AG's corpus of news articles on the web, categorized into four main topics: World, Sports, Business, and Science. While these datasets ignore the influences of financial indicator and assume that the stock market price can be easily predicted by stock price and the news itself. We argued that financial indicator and news are main contributors.

Table 2. Comparison of Financial Sentiment Datasets

| Dataset       | Scope                    | Size   | Annotation       | Temperol   |
|---------------|--------------------------|--------|------------------|------------|
| PhraseBank    | Finance News             | 4840   | Expert           | N          |
| Reuters-21578 | Finance News             | 21,578 | User             | 1Y, 1987   |
| StockTwits    | Stock Discussions        | 6.4M   | User             | 10Y        |
| TweetFinSent  | Meme Stocks              | 2,113  | Expert           | N          |
| PsychSignal   | Market News              | Varied | Algorithm & User | N          |
| AG News       | News                     | 1M     | User             | N          |
| FinSen (Ours) | Market News & Indicators | 160K   | FinBERT          | 15Y        |

## 2.2 Related Stock Market Forecasting Methods

The fusion of sentiment analysis and machine learning techniques has emerged as a promising avenue in the realm of stock market prediction. While the LSTM model with an attention mechanism [26] has proven its superiority in prediction accuracy, it falls short in providing reliable confidence estimates. Although previous works have shown the effectiveness of various techniques on financial prediction, they all ignore model interpretability and trustworthiness, risking the failing rate of investment in practical market. The KGEB-CNN technique[11], utilized data from the S&amp;P 500 and Reuters, achieving an accuracy of 66.93%. Similarly, Vargas et al.[47] used the SI-RCNN technique with data from the S&amp;P 500 and Reuters, yielding an accuracy of 62.03%. Chen et al.[7] implemented the BiLSTM-CRF technique on data from TOPIX and Reuters, resulting in an accuracy of 65.7%. Deng et al.[8] applied the KDTCN technique to DJIA and Reddit WorldNews data, achieving the highest accuracy of 71.8%. Additionally, Jin et al.[22] used the EMD-LSTM technique with data from Apple and StockTwits, reaching an accuracy of 70.01%. Li et al.[27] employed the LSTM technique on data from the Hong Kong Exchange and FINET, with an accuracy of 60.10%. Finally, Li et al.[26] also used the LSTM technique, but with data from CSI300 and Eastmoney Forum, achieving the highest reported accuracy of 87.86%.

## 2.3 Nerual Network Model Calibration Techniques

Initial efforts in model calibration, such as Platt Scaling [41] and Isotonic Regression [52], primarily focused on adjusting the outputs of already trained models. These techniques, while pioneering, often fell short in directly influencing the model's internal decision-making process, propelling the development of more flexible approaches. Later modification on loss function, Focal Loss [29] potentially leads to overconfident or underconfident predictions because it modifies the standard cross-entropy loss to focus more on hard, misclassified examples, without directly addressing how accurate the predicted probabilities are in reflecting the true likelihood of an event [16,45]. Various network calibration techniques are measured by sophisticated metrics that gauge calibration quality, such as the Expected Calibration Error (ECE) [19] and Brier Score [6].

## 3 Data and Methodology

The target variable is the daily percentage change in the Standard &amp; Poor's 500 Index (S&amp;P 500), which serves as an indicator of the US stock market's health. We utilize the volatility derived from its closing price to explore the causal relationship with sentiment scores from our FinSen(US) dataset, as illustrated in Fig. 2. Upon establishing causal evidence, we incorporate this causally validated feature into our proposed LSTM model for prediction and calibration experiments as shown in Fig. 1.

Fig. 1. Framework of LSTM Volatility Prediction and DAN 3 Text Classification. X 4 is sentiment scores generated from FinSen dataset by FinBERT as the input of both models after causally validated with volatility y .

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01005v1-figure-cc0cdfb3-199e-4680-b40e-ce5ae01961f5.jpg)

## 3.1 FinSen Data Collection and Preprocessing

Numerical Data The S&amp;P 500 OHLCV data, including open, high, low, close, and volume, was used to extract price momentum and volatility. This data was sourced from the Yahoo Finance API [51] from January 2020 to June 2023. Our study's supervised target are the S&amp;P 500 index momentum [49], volatility [1,33], and log volatility [1] which are defined in Appendix 1. Textual Data Data collection was conducted using advanced web scraping techniques to collect news headlines and contents from reputable financial news website, Trading Economics, source of economic indicators and financial news [46]. The Trading Economics platform offers access to over 20 million indicators from 196 countries, including historical and current data on exchange rates, stocks, indexes, bonds, and commodity prices. Our FinSen dataset, comprising over 160,000 records collected over a span of 15 years, was curated to ensure a broad representation of global economic conditions with temporal information. Sentiment analysis is conducted and annotated on the preprocessed financial news articles using FinBERT [9], a finetuned BERT model specifically designed for financial text. This step transforms qualitative news content into quantitative sentiment scores, facilitating the integration of sentiment data. The model and its tokenizer are accessed from the Hugging Face Transformers library [50] under the identifier "ProsusAI/finbert". Table 3, on a few selected days, the sentiment scores we obtained from FinBERT.

- 1. Sentiment Scoring of News: The encoded text is fed into FinBERT, which outputs logits representing the sentiment classification. This score, denoted as S i , ranges possibilities among -1 (negative), 0 (neutral) and +1 (positive), representing the sentiment of the i th news.

- 2. Daily Sentiment Aggregation: To obtain a daily sentiment score, we aggregated the sentiment scores of all news for each day. The daily sentiment score SAgg d = 1 N d ∑ N d i =1 (( - ∗ 1 P neg ) + (0 ∗ P neu ) + (1 ∗ P pos )) , for day d is calculated as the average of sentiment scores of all news made on that day, where N d is the total number of news on day d .

Table 3. FinSen Dataset: The Aggregated Sentiment Scores by FinBERT

| Date      | News Content                                                                                                                                                                   | Sent.   | FinBERT                             |   SAgg d |
|-----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------|-------------------------------------|----------|
| 2023/2/14 | Coca-Cola earnings meet market ex- pectations at 0.45 USD.                                                                                                                     | Pos.    | [0.448771, 0.0296302, 0.5215982]    |   0.6597 |
| 2023/3/10 | The unemployment rate in the US edged up to 3.6 percent in February 2023, up from a 50-year low of 3.4 per- cent seen in January and above market expectations of 3.4 percent. | Neg.    | [0.9373965, 0.04212248, 0.02048101] |   0.2637 |
| 2023/4/25 | Microsoft decreased to a 4-week low of 278.39.                                                                                                                                 | Neu.    | [0.01425993, 0.9658618, 0.01987824] |   0.2142 |
| 2023/5/03 | Apple increased to a 35-week high of 170.47.                                                                                                                                   | Neg.    | [0.9287658, 0.02683192, 0.04440225] |   0.2293 |

Fig. 2. Granger Cause Test: X is Aggregated Sentiment Scores and y is Market Volatility. Yellow dashed lines highlight prominent peaks and valleys of alignment, selected subjectively to illustrate instances of notable alignment between features. 2

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01005v1-figure-4b449aba-1911-4100-b2ce-4135c3538666.jpg)

2 It is important to note that the stock market is influenced by various factors beyond investor sentiments, including economic indicators, geopolitical events, and market speculation[3]. While sentiment scores offer valuable insights, they do not solely dictate market fluctuations, and market reactions to news can be immediate or delayed. The presence of unaccounted variables can lead to instances where alignment between sentiment scores and market volatility is not perfect.

## 3.2 Proposed Methods

Integrating Causal-Validated Sentiment Scores into LSTM for Volatility Prediction To explore the potential of market sentiment as a predictive factor for S&amp;P 500 volatility, we conducted a Granger causality test [18] and treated SAgg d as X . This statistical method determines whether one time series can forecast another, effectively testing for causality between two variables. Specifically, we assessed whether sentiment scores ( X ) derived from financial news could predict the S&amp;P 500's volatility ( y ) by equation (1). If the coefficients λ i are statistically significant, X Granger-causes y .

$$y _ { t } & = \alpha + \sum _ { \substack { i = 1 \\ \_ \dots \_ \dots \_ \_ } } ^ { k } \beta _ { i } Y _ { t - i } + \sum _ { i = 1 } ^ { k } \lambda _ { i } X _ { t - i } + \epsilon _ { t } & & ( 1 )$$

$$X _ { t } = \{ X | X \, \text{Granger-causes} \, y \}.$$

Given the established Granger causality between market sentiment scores ( X ) and the S&amp;P 500's volatility ( y ), we integrated sentiment analysis into an LSTM model to predict the volatility. The LSTM model uses the traditional input features along with sentiment scores, represented by the feature vector [ V , X t t ] , where V t denotes the 'Volatility' feature and X t represents the sentiment scores at time step t . The LSTM model is based on the standard LSTM structure defined as h t = LSTM ( h t -1 , [ V , X t t ]) , where h t is the hidden state at time t , and LSTM represents the LSTM function that updates the hidden state based on the previous hidden state h t -1 and the input features [ V , X t t ] .

Embedding Causal-Validated Feature into DAN 3 for Text Classification Our approach implemented 3 feed-forward layers of Deep Averaging Network with Batch Normalization (DAN 3) [19], which is structured in Fig. 1 and to process textual data by first converting token indices to dense embeddings, followed by averaging these embeddings to form a fixed-size vector representation of each text instance. This representation is then processed through a series of dense layers interleaved with batch normalization, culminating in an output layer that maps to the number of output classes.

Focal Calibration Loss Our Focal Calibration Loss is designed to address two main challenges for text classifier during training: class imbalance and model calibration. The intuition behind its design is rooted in enhancing model performance while ensuring the predicted probabilities are aligned with the true outcomes. The Focal Calibration Loss (Eq. 4) combines with two components, Focal Loss [29] and Calibration Error (Eq. 3), incorporating a regularization parameter λ to balance the trade-off between focusing on hard examples and ensuring proper calibration of the model's predictions:

$$\text{FL} ( p _ { t } ) = - ( 1 - p _ { t } ) ^ { \gamma } \log ( p _ { t } ), \ \text{CalibError} = \frac { 1 } { N } \sum _ { i = 1 } ^ { N } \sum _ { c = 1 } ^ { C } | p _ { i, c } - y _ { i, c } | \quad \ \ ( 3 )$$

$$FocalCalibrationLoss = FL ( p _ { t } ) + \lambda \cdot \text{CalibError}$$

In Focal Loss term, p t is the model's estimated probability for the class label t . γ is the focusing parameter that adjusts the rate at which easy examples are

down-weighted. In Calibration Error term, formulated as the mean absolute error between the predicted probabilities and the one-hot encoded ground truth labels across all classes. N C , is the number of examples and classes, p i,c is the predicted probability of class c for the i th sample, y i,c is the one-hot encoding of the true class label for the i th example. The theoretical underpinning of using MAE for Calibration Error comes from the concept of expected calibration error (ECE) [19], which is a popular metric for assessing calibration by grouping predictions into bins and comparing the average predicted probability to the actual outcome frequency within each bin. While ECE provides a binned estimate of calibration, MAE offers a continuous, instance-level calibration measure.

## 4 Experiments and Results

## 4.1 Establishing Causality: Sentiment Scores on Market Volatility

$$\Delta S P 5 0 0 _ { t } = \begin{cases} a _ { 0 } + \sum _ { i = 1 } ^ { k } a _ { i } ( \Delta S P 5 0 0 ) _ { t - i } + \sum _ { i = 1 } ^ { k } b _ { i } ( X _ { t - i } ) + \epsilon _ { t } \\ c _ { 0 } + \sum _ { i = 1 } ^ { k } c _ { i } ( \Delta S P 5 0 0 ) _ { t - i } + \sum _ { i = 1 } ^ { k } d _ { i } ( X _ { t - i } ) + \epsilon _ { t } \\ \dots - \dots - \dots \end{cases} \quad ( 5 )$$

$$F = \frac { ( S S R _ { R } - S S R _ { U } ) / p } { S S R _ { U } / ( T - k - 1 ) }.$$

Our analysis begins with evaluating the stationarity of the percent changes in both the volatility of S&amp;P 500 and the news sentiment scores. We applied Augmented Dickey-Fuller (ADF) Test [10] that they are stationary and suitable for Granger causality testing. To determine the causal relationship, we employ a regression model that predicts the percentage change in the S&amp;P 500 , denoted as ∆ SP500, based on sentiment scores from financial news articles. The model is structured to include the sentiment scores lagged up to 30 days, represented as X t -i , where i ranges from 1 to k , with k being the maximum lag of 30 days.

The null hypothesis posits that the coefficients b i = 0 for all i . We apply an F-test by equation(6) to the residuals from the restricted model (excluding sentiment scores) and the unrestricted model (including sentiment scores). If the F-value is greater than the critical value at a chosen level of significance (0.05), we reject the null hypothesis, suggesting that sentiment scores "Granger cause" the S&amp;P 500 volatility. The null hypothesis for each test is that the lagged values of the independent variable do not belong in the regression. Based on the results from these tests, we formulated four possibilities representing potential causal relationships between sentiment scores and ∆ SP500:

̸

- 1. Sentiment scores "Granger-cause" ∆ SP500 if they improve the prediction of volatility, and volatility do not improve the prediction of sentiment scores ( b i = 0 and d i = 0 ).

̸

- 2. ∆ SP500 "Granger-causes" sentiment scores if volatility improve the prediction of sentiment scores, and sentiment scores do not improve the prediction of volatility ( b i = 0 and d i = 0 ).

̸

̸

- 3. Amutual relationship exists if both sentiment scores "Granger-cause" ∆ SP500, and ∆ SP500 "Granger-causes" sentiment scores ( b i = 0 and d i = 0 ).

- 4. Independence is indicated when no causal relationships are found ( b i = 0 and d i = 0 ).

The results of the Granger causality tests for Eq.(5) are summarized in Fig. 4. We tested whether ∆ SP500 predicts sentiment score, and whether sentiment score predicts ∆ SP500. Separate regressions were run for all values of k (1 to 30), where k represents the maximum lag length in the regression. The F-statistics are significant at the 1%, 5%, and 10% levels for lagged day 1, 3, 7, 14 and 30, respectively. This suggests that sentiment score does "Granger cause" ∆ SP500. However, none of the F-statistics are sufficient to reject the null hypothesis, indicating that ∆ SP500 does not "Granger cause" sentiment score. The results suggest that sentiment scores do "Granger cause" volatility.

## 4.2 Enhancing LSTM with Causal-Validated Sentiment Scores

The experimental setup and procedure, depicted in Fig. 3, commences with the collection of news articles and historical stock data. Sentiment is first annotated at the sentence level within each news article. This process involves utilizing FinBERT pretrained model to assign a sentiment score to each sentence. These scores are then aggregated to compute an overall sentiment score daily.

With market volatility and aggregated sentiment scores prepared, we then put them into our proposed LSTM model. The model accepts input through an initial layer configured to handle sequences of data, with the input shape specified as timesteps and dim, and the timesteps is set to 1 to process data in single timestep increments, and dim is from 1 expanded to 2 to accommodate the inclusion of both volatility and sentiment scores as features. Sequentially, three LSTM layers are employed, each comprising 100 neurons. We've applied 2 dropout layers with a rate of 0.5 to prevent overfitting. The model is compiled using the Adam optimizer and mean squared error (MSE) as loss function. Training is conducted with 100 epochs, with a batch size of 32. After model trained, the test dataset proceeds to generate predictions for volatility and compared with baseline model without sentiment score.

Fig. 3. The Proposed LSTM Experimental Flow Chart

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01005v1-figure-c44c1b5d-1d5c-4b16-9063-2a47aafa7717.jpg)

The volatility prediction result over a period from March to July 2023 by LSTM model show in Fig. 5. The prediction with sentiment score is better than

that one without and blue one is more closer approximation to the ground true value. Lag 14 exhibits better performance with lower error metrics (MAE, MSE, RMSE) and the highest R 2 value 94.84% among all lags. This implies that the model is most effective and accurate in capturing the relationship at this lag. The prediction result shows the evidence that the causal-validated sentiment score, as reflected in financial news, can help predict stock market volatility.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01005v1-figure-e2a07508-818d-43d2-936e-5909c1d228fe.jpg)

Fig. 4. Granger Causality Test Result. Filtered p -values &lt; 0.05 on lags 1, 3, 7, 14, 30.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01005v1-figure-7d529302-eb50-467e-8fe0-e8c8ece3079c.jpg)

Fig. 5. Volatility Prediction Comparison by LSTM Model. Red dash line is the prediction w/o sentiment score, while the blue dash line with it which is more closer with actual volatility (black line).

## 4.3 Calibration: Embedding Enhanced DAN 3 Model

We used the 20 Newsgroups dataset [23], Financial PhraseBank [31], AG News [53] and FinSen(Ours) to train global-pooling CNN network [24] and DAN 3 [19]. The datasets are preprocessed to load texts and their corresponding labels, tokenize the texts using a basic English tokenizer, and build a vocabulary from the tokenized texts. In the implementation, the GloVe (Global Vectors for Word Representation) [40] embeddings are utilized within DAN 3 text classification model. This GloVe embeddings used are from the 6B tokens dataset, which contains embeddings for 400k words, each represented in a 100-dimensional space. To accommodate varying lengths of text data, the texts are vectorized and padded to a uniform length, ensuring that each input to the model has the same size. This vectorization process transforms the tokenized texts into sequences of indices, with each index representing a word in the vocabulary. Padding is applied to shorter texts to reach the predefined maximum sequence length. Our models are trained on these datasets using cross entropy (CE), AdaFocal [16], DualFocal [45] and Focal Calibration Loss (Ours) with 20 epochs and measured by Expected Calibration Error (ECE) [19] and reliability diagram [35].

Fig. 6. FinSen Reliability Diagram of Different Methods w/o Temperature Scaling, γ setting for AdaFocal range from 1 to 20; γ for DFL is 2; γ for FCL is 2 and λ is 0.1.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01005v1-figure-98785e7b-effe-43cb-b81a-94fb25198004.jpg)

Our FinSen dataset trained on DAN 3 model with Focal Calibration Loss shows an improvement in reliability diagram, which is competitive with other datasets in Fig. 6. With the DAN 3 model, FinSen achieves one of the lowest ECEs across all datasets and models, indicating that predictions are wellcalibrated. In terms of ECE, the improvement for the FinSen dataset is notable, especially considering the significant reduction in classification error, which is only 0.13% in global pooling CNN.

Table 4. Error metrics (%) on test set

| Datasets   | Model              | Error Type         | CE          | AFL         | DFL        | FCL(Ours)   |
|------------|--------------------|--------------------|-------------|-------------|------------|-------------|
| FinSen     | Global Pooling CNN | Classification ECE | 4.73 6.46   | 2.80 6.95   | 0.95 5.55  | 0.13 0.76   |
| FinSen     |                    | MCE                | 47.70       | 53.90       | 76.21      | 42.51       |
| FinSen     | DAN 3              | Classification ECE | 39.00 36.97 | 33.92 13.94 | 15.32 4.77 | 4.43 3.34   |
| FinSen     |                    |                    |             | 30.06       |            |             |
| FinSen     |                    | MCE                | 49.58       |             | 66.67      | 33.29       |

## 5 Conclusion

By proving a causal relationship between sentiment scores and market volatility and integrating these causal-validated feature into LSTM models, our model achieves more nuanced and reliable market forecasts. Our FinSen dataset demonstrates relative better calibration on text classification task among others. The dramatic improvement in ECE to 3.34% with the DAN 3 model on our Focal Calibration Loss suggests that not only has the model achieved a high level of accuracy, but its probability predictions are also well-aligned with actual outcomes, making it highly reliable for decision-making processes.

## Appendix

## 1 Momentum, Volatility and Log Volatility

This appendix provides the detailed mathematical formulations used in our analysis to calculate the momentum and volatility of financial assets. Momentum t =

P t -P t -n represents the calculation of momentum at time t , where P t is the price of the asset at time t , and P t -n is the price of the asset n periods before t . Equation Volatility = √ 1 N -1 ∑ N i =1 ( R i -¯ ) R 2 defines the volatility of an asset, calculated as the standard deviation of its returns. R i = ( P t P t -n -1 ) × 100% represents the return of the asset at time i , and ¯ R is the average return over N periods. Equation Log Volatility = ln( Volatility ) shows the natural logarithm of the volatility, which log volatility used in certain financial models.

## 2 Evaluation Metrics &amp; Other Calibration Experiments

Mean Absolute Error (MAE) = 1 n ∑ n i =1 | y i - ˆ y i Mean Squared Error (MSE) = 1 ∑ n ( y i - ˆ ) y i 2

| n i =1 √ ∑ ∣ ∣ 100% i =1 ( y i - ¯) y ) ) M m =1 | B | n | m =1

Root Mean Squared Error (RMSE) = 1 n n i =1 ( y i - ˆ ) y i 2

Mean Absolute Percentage Error (MAPE) = 1 n ∑ n i =1 ∣ ∣ y i - ˆ y i y i ∣ ∣ ×

Coefficient of Determination ( R 2 ) = 1 - ∑ n i =1 ( y i - ˆ ) y i 2 ∑ n 2

Akaike Information Criterion (AIC) = 2 k - 2 ln( L

Bayesian Information Criterion (BIC) = ln( n k ) - 2 ln( L

Expected Calibration Error (ECE) = ∑ m | acc ( B m ) - conf ( B m )

Maximum Calibration Error (MCE) = max M | acc ( B m ) - conf ( B m ) |

Table 5. Error metrics (%) on test set

| Datasets      | Model              | Error Type     |    CE |   AFL |   DFL |   FCL(Ours) |
|---------------|--------------------|----------------|-------|-------|-------|-------------|
|               |                    | Classification | 24.25 | 24.8  | 25.3  |       21.85 |
| 20 Newsgroups | Global Pooling CNN | ECE            |  6.82 |  6.24 |  5.55 |       11.13 |
|               |                    | MCE            | 17.28 | 19.65 | 20.29 |       30.07 |
|               |                    | Classification | 83    | 84.25 | 80    |       77.65 |
| 20 Newsgroups | DAN 3              | ECE            | 31.69 | 53.9  | 28.14 |       29.97 |
|               |                    | MCE            | 50.99 | 76.27 | 42.14 |       52.9  |
|               |                    | Classification | 73.6  | 72.93 | 72.31 |       71.75 |
| AG News       | Global Pooling CNN | ECE            | 45.57 | 38.81 | 41.16 |       29.04 |
|               |                    | MCE            | 67.92 | 67.07 | 73.7  |       61.22 |
|               |                    | Classification | 71.68 | 71.13 | 73.82 |       74.71 |
| AG News       | DAN 3              | ECE            | 51.32 |  4.25 | 35.15 |       37.85 |
|               |                    | MCE            | 66.17 | 63.02 | 45.08 |       45.67 |
|               |                    | Classification | 10.6  | 18.98 | 13.25 |       15.67 |
| PhraseBank    | Global Pooling CNN | ECE            |  3.53 | 21.81 |  8.22 |       12.08 |
|               |                    | MCE            | 21.7  | 89.22 | 18.63 |       60.53 |
|               |                    | Classification | 22.96 | 24.06 | 18.76 |       12.8  |
| PhraseBank    | DAN 3              | ECE            |  3.94 |  4.15 | 11.31 |        4.74 |
|               |                    | MCE            | 61.32 | 19.3  | 32.43 |       56.49 |

## References

- 1. Agustini, W.F., Affianti, I.R., Putri, E.R.: Stock price prediction using geometric brownian motion. In: Journal of Physics: Conference Series. vol. 974, p. 012047. IOP Publishing (2018). https://doi.org/10.1088/1742-6596/974/1/012047

- 2. Antweiler, W., Frank, M.Z.: Is all that talk just noise? the information content of internet stock message boards. The Journal of finance 59 (3), 1259-1294 (2004)
- 4. Bao, W., Yue, J., Rao, Y.: A deep learning framework for financial time series using stacked autoencoders and long-short term memory. PloS one 12 (7), e0180944 (2017)
- 3. Baker, M., Wurgler, J.: Investor sentiment in the stock market. Journal of Economic Perspectives 21 (2), 129-152 (June 2007). https://doi.org/10.1257/jep.21.2. 129 , https://www.aeaweb.org/articles?id=10.1257/jep.21.2.129
- 5. Batra, R., Daudpota, S.M.: Integrating stocktwits with sentiment analysis for better prediction of stock price movement. In: 2018 international conference on computing, mathematics and engineering technologies (ICoMET). pp. 1-5. IEEE (2018)
- 7. Chen, D., Zou, Y., Harimoto, K., Bao, R., Ren, X., Sun, X.: Incorporating finegrained events in stock movement prediction. arXiv preprint arXiv:1910.05078 (2019)
- 6. Brier, G.W.: Verification of forecasts expressed in terms of probability. Monthly weather review 78 (1), 1-3 (1950)
- 8. Deng, S., Zhang, N., Zhang, W., Chen, J., Pan, J.Z., Chen, H.: Knowledge-driven stock trend prediction and explanation via temporal convolutional network. In: Companion Proceedings of The 2019 World Wide Web Conference. pp. 678-685 (2019)
- 10. Dickey, D.A., Fuller, W.A.: Distribution of the estimators for autoregressive time series with a unit root. Journal of the American Statistical Association 74 (366), 427-431 (1979)
- 9. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
- 11. Ding, X., Zhang, Y., Liu, T., Duan, J.: Knowledge-driven event embedding for stock prediction. In: Proceedings of coling 2016, the 26th international conference on computational linguistics: Technical papers. pp. 2133-2142 (2016)
- 13. Fama, E.F.: Efficient capital markets: A review of theory and empirical work. The journal of Finance 25 (2), 383-417 (1970)
- 12. Engle, R.F., Sokalska, M.E.: Forecasting intraday volatility in the us equity market: Multiplicative component garch. Journal of Financial Econometrics 10 , 54-83 (2012)
- 14. Franses, P.H., Van Dijk, D.: Forecasting stock market volatility using (non-linear) garch models. Journal of forecasting 15 (3), 229-235 (1996)
- 16. Ghosh, A., Schaaf, T., Gormley, M.: Adafocal: Calibration-aware adaptive focal loss. Advances in Neural Information Processing Systems 35 , 1583-1595 (2022)
- 15. Ge, W., Lalbakhsh, P., Isai, L., Lenskiy, A., Suominen, H.: Neural network-based financial volatility forecasting: A systematic review. ACM Computing Surveys (CSUR) 55 (1), 1-30 (2022)
- 17. Gong, X.L., Liu, X.H., Xiong, X., Zhuang, X.T.: Forecasting stock volatility process using improved least square support vector machine approach. Soft computing 23 , 11867-11881 (2019)
- 19. Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: International conference on machine learning. pp. 1321-1330. PMLR (2017)
- 18. Granger, C.W.J.: Investigating causal relations by econometric models and crossspectral methods. Econometrica 37 (3), 424-438 (1969)

- 20. Hiew, J.Z.G., Huang, X., Mou, H., Li, D., Wu, Q., Xu, Y.: Bert-based financial sentiment index and lstm-based stock return predictability. arXiv preprint arXiv:1906.09024 (2019)
- 22. Jin, Z., Yang, Y., Liu, Y.: Stock closing price prediction based on sentiment analysis and lstm. Neural Computing and Applications 32 , 9713-9729 (2020)
- 21. Jaggi, M., Mandal, P., Narang, S., Naseem, U., Khushi, M.: Text mining of stocktwits data for predicting stock prices. Applied System Innovation 4 (1), 13 (2021)
- 23. Joachims, T., et al.: A probabilistic analysis of the rocchio algorithm with tfidf for text categorization. In: ICML. vol. 97, pp. 143-151. Citeseer (1997)
- 25. Lewis, D.: Reuters-21578 Text Categorization Collection. UCI Machine Learning Repository (1997), DOI: https://doi.org/10.24432/C52G6M
- 24. Kim, Y.: Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882 (2014)
- 26. Li, J., Bu, H., Wu, J.: Sentiment-aware stock market prediction: A deep learning method. In: 2017 international conference on service systems and service management. pp. 1-6. IEEE (2017)
- 28. Li, X., Xie, H., Chen, L., Wang, J., Deng, X.: News impact on stock price return via sentiment analysis. Knowledge-Based Systems 69 , 14-23 (2014)
- 27. Li, X., Wu, P., Wang, W.: Incorporating stock prices and news sentiments for stock market prediction: A case of hong kong. Information Processing &amp; Management 57 (5), 102212 (2020)
- 29. Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision. pp. 2980-2988 (2017)
- 31. Malo, P., Sinha, A., Korhonen, P., Wallenius, J., Takala, P.: Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology 65 (4), 782-796 (2014)
- 30. Liu, S., Borovykh, A., Grzelak, L.A., Oosterlee, C.W.: A neural network-based framework for financial model calibration. Journal of Mathematics in Industry 9 , 1-28 (2019)
- 32. Marcucci, J.: Forecasting stock market volatility with regime-switching garch models. Studies in Nonlinear Dynamics &amp; Econometrics 9 (4) (2005)
- 34. Mohan, S., Mullapudi, S., Sammeta, S., Vijayvergia, P., Anastasiu, D.C.: Stock price prediction using news sentiment analysis. In: 2019 IEEE fifth international conference on big data computing service and applications (BigDataService). pp. 205-208. IEEE (2019)
- 33. Markowitz, H.: Portfolio selection. The Journal of Finance 7 (1), 77-91 (1952)
- 35. Murphy, A.H., Winkler, R.L.: Reliability of subjective probability forecasts of precipitation and temperature. Journal of the Royal Statistical Society Series C: Applied Statistics 26 (1), 41-47 (1977)
- 37. Oliveira, N., Cortez, P., Areal, N.: On the predictability of stock market behavior using stocktwits sentiment and posting volume. In: Progress in Artificial Intelligence: 16th Portuguese Conference on Artificial Intelligence, EPIA 2013, Angra do Heroísmo, Azores, Portugal, September 9-12, 2013. Proceedings 16. pp. 355-365. Springer (2013)
- 36. Niculescu-Mizil, A., Caruana, R.: Predicting good probabilities with supervised learning. In: Proceedings of the 22nd international conference on Machine learning. pp. 625-632 (2005)
- 38. Oliveira, N., Cortez, P., Areal, N.: Automatic creation of stock market lexicons for sentiment analysis using stocktwits data. In: Proceedings of the 18th international database engineering &amp; applications symposium. pp. 115-123 (2014)

- 39. Pei, Y., Mbakwe, A., Gupta, A., Alamir, S., Lin, H., Liu, X., Shah, S.: Tweetfinsent: A dataset of stock sentiments on twitter. In: Proceedings of the Fourth Workshop on Financial Technology and Natural Language Processing (FinNLP). pp. 37-47 (2022)
- 41. Platt, J., et al.: Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers 10 (3), 61-74 (1999)
- 40. Pennington, J., Socher, R., Manning, C.D.: Glove: Global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). pp. 1532-1543 (2014)
- 42. Poon, S.H., Granger, C.W.J.: Forecasting volatility in financial markets: A review. Journal of economic literature 41 (2), 478-539 (2003)
- 44. Solaiman, S., Rahman, B.O., Arefin, M.F., Ahmed, C.F., Leung, C.K., Madill, E.W.: Unveiling market sentiments: A comprehensive analysis of stock market responses to diverse news events using data mining techniques. In: 2024 18th International Conference on Ubiquitous Information Management and Communication (IMCOM). pp. 1-8. IEEE (2024)
- 43. PsychSignal: https://www.PsychSignal.com
- 45. Tao, L., Dong, M., Xu, C.: Dual focal loss for calibration. arXiv preprint arXiv:2305.13665 (2023)
- 47. Vargas, M.R., De Lima, B.S., Evsukoff, A.G.: Deep learning for stock market prediction from financial news articles. In: 2017 IEEE international conference on computational intelligence and virtual environments for measurement systems and applications (CIVEMSA). pp. 60-65. IEEE (2017)
- 46. TradingEconomics: https://www.tradingeconomics.com
- 48. Vui, C.S., Soon, G.K., On, C.K., Alfred, R., Anthony, P.: A review of stock market prediction with artificial neural network (ann). In: 2013 IEEE international conference on control system, computing and engineering. pp. 477-482. IEEE (2013)
- 50. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al.: Transformers: State-of-the-art natural language processing. In: Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations. pp. 38-45 (2020)
- 49. Wilder, W.: New Concepts In Technical Trading Systems. Wilder's own publication (1978), introduction of the momentum concept in stock market analysis
- 51. Yahoo!Finance'sAPI: https://pypi.org/project/yfinance/
- 53. Zhang, X., Zhao, J., LeCun, Y.: Character-level convolutional networks for text classification. Advances in neural information processing systems 28 (2015)
- 52. Zadrozny, B., Elkan, C.: Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In: Icml. vol. 1, pp. 609-616 (2001)
Wenhao Liang Zhengyang Li Weitong Chen
Enhancing Financial Market Predictions: Causality-Driven Feature Selection
This paper introduces the FinSen dataset that revolutionizes financial market
analysis by integrating economic and financial news articles from 197 countries
with stock market data. The dataset's extensive coverage spans 15 years from
2007 to 2023 with temporal information, offering a rich, global perspective
with 160,000 records on financial market news. Our study leverages causally
validated sentiment scores and LSTM models to enhance market forecast accuracy
and reliability. Utilizing the FinSen dataset, we introduce an innovative Focal
Calibration Loss, reducing Expected Calibration Error (ECE) to 3.34 percent
with the DAN 3 model. This not only improves prediction accuracy but also
aligns probabilistic forecasts closely with real outcomes, crucial for the
financial sector where predicted probability is paramount. Our approach
demonstrates the effectiveness of combining sentiment analysis with precise
calibration techniques for trustworthy financial forecasting where the cost of
misinterpretation can be high. Finsen Data can be found at [this github
URL](https://github.com/EagleAdelaide/FinSen_Dataset.git).