Volatility prediction is complex due to the stock market's stochastic nature. Existing research focuses on the textual elements of financial disclosures like earnings calls transcripts to forecast stock volatility and risk, but ignores the rich acoustic features in the company executives' speech. Recently, new multimodal approaches that leverage the verbal and vocal cues of speakers in financial disclosures significantly outperform previous state-of-the-art approaches demonstrating the ben-efits of multimodality and speech. However, the financial realm is still plagued with a severe underrepresentation of various communities spanning diverse demographics, gender, and native speech. While multimodal models are better risk forecasters, it is imperative to also investigate the potential bias that these models may learn from the speech signals of company executives. In this work, we present the first study to discover the gender bias in multimodal volatility prediction due to gender-sensitive audio features and fewer female executives in earnings calls of one of the world's biggest stock indexes, the S&P 500 index. We quantitatively analyze bias as error disparity and investigate the sources of this bias. Our results suggest that multimodal neural financial models accentuate gender-based stereotypes. 1 1 Introduction Earnings calls are publicly available, quarterly conference calls where CEOs discuss their company's performance and future prospects with outside analysts and investors (Qin and Yang, 2019; Sawhney et al., 2020b). They consist of two sections: a prepared delivery of performance statistics, analysis and future expectations, and a spontaneous question-answer session to seek additional information not disclosed before (Keith and Stent, 2019). Researchers have studied the Post Earnings Announcement Drift (PEAD) to observe that statements made by upper management affect the way information is digested and acted upon impacting short-term price movements (Ball and Brown, 1968; Bernard and Thomas, 1989; Yang et al., 2020). Audio features contextualize text and connotate speaker's emotional and psychological state (Fish et al., 2017; Jiang and Pell, 2017; Burgoon et al., 2015; Bachorowski, 1999). Hence, when used with textual features, audio features significantly determine the effect of earning calls on the stock market (Qin and Yang, 2019; Yang et al., 2020). Past research has shown that audio features such as speakers' pitch, intensity, etc. vary greatly across genders (Mendoza et al., 1996; Burris et al., 2014; Latinus and Taylor, 2012). Moreover, female executives are highly underrepresented in these earnings calls(Agarwal, 2019; Investments, 2017). The variation in audio features is amplified by deep learning models due to a dearth of female training examples and is manifested as a gender bias. The system learns unneeded correlations between stock volatility and sensitive attributes like gender, accent, etc. It further perpetuates gender-based stereotypes and generalizations like female executives are less con-fident than male executives (Lonkani, 2019), men are assessed as more charismatic than female executives under identical conditions (Novk-Tt et al., 2017), and nurses are female and doctors are male (Saunders and Byrne, 2020). Biased models further perpetuate stereotypes that can harm underrepresented communities, specifically in the financial and corporate world. Novk-Tt et al. (2017) even show that female speakers have to deliver better acoustic-melodic performance to seem as charismatic as men. Taking a step towards fair risk forecasting models, we analyze gender bias by studying the error disparity in the state-of-the-art for multimodal volatility prediction, MDRM (Qin and Yang, 2019). Bias in Finance Public financial data is impacting virtually every aspect of investment decision making (Peric et al., 2016; Brynjolfsson et al., 2011). Prior research shows that NLP methods leveraging social media (Sawhney et al., 2020a), news (Du and Tanaka-Ishii, 2020), and earning calls (Wang and Hua, 2014) can accurately forecast financial risk. Companies and investors use statistical and neural models on multimodal financial data to forecast volatility (Cornett and Saunders, 2003; Trippi and Turban, 1992) and minimize risk. These models although effective, may be tainted by bias due to individual and societal differences, often unintended (Mehrabi et al., 2019). For example, models trained on the audio features extracted from CEO's speech in earnings calls (Qin and Yang, 2019), may be prone to bias given the underrepresentation of several demographics across race, gender, native language, etc. in the financial realm. Bias in AI Bias is prevalent in AI based neural models owing to the lack of diversity in training data (Torralba and Efros, 2011; Tommasi et al., 2017). The design and utilization of AI models trained on gender imbalanced data, pose potential deprivation of opportunities to underrepresented groups such as females(Niethammer, 2020; Dastin, 2018). With over 75% of AI professionals being men, male experiences also dominate algorithmic creation (Forum, 2018). In terms of natural language representation, embeddings such as word2vec and GloVe, trained on news articles may inherit gender stereotypes (Packer et al., 2018; Bolukbasi et al., 2016; Park et al., 2018). Recent studies also show the presence of bias in speech emotion recognition (Li et al., 2019). Bias in AI and Finance With the advent of AI and Big Data, companies are intelligently using data to measure performance (Newman, 2020). But seldom do enterprises check on the imbalance in gathered data. Women still represent fewer than 20% positions in the financial-services C-suite (Chin et al., 2018) and only 5% of Fortune-500 CEOs are women (Suresh and Guttag, 2019). Studies show that models trained on gender imbalanced data reduce the chances for women to get capital investments or loans (Grdeniz et al., 2020). Apart from that, using feature representations in-Figure 1: Model architecture used for training the multimodal audio-text model for evaluating the gender specific performance inspired by (Qin and Yang, 2019) trinsic to different genders can inculcate semantic gender bias (Li et al., 2019; Suresh and Guttag, 2019). Professional studies have found that men tend to self-reference using I', me' and mine' whereas women tend to reference the team, like we', our' and us' (Investments, 2017). Although there is great progress in mitigating bias in text, understanding its presence in multimodal speech based analysis, particularly in real world scenarios like corporate earnings calls analysis remain an understudied yet promising research direction. Another study found that despite having identical credibility, female CEOs are perceived as less capable to attract growth capital (Bigelow et al., 2014). Stock volatility Following Kogan et al. (2009); Sawhney et al. (2020c), for a given stock, with a close price of p i on trading day i , we calculate the average log volatility over n days following the day of the earnings call as: Volatility Prediction Consider each earnings call E , with aligned audio recordings A and text transcripts T . The earnings calls are divided into separate distributions based on the gender of the speaker to analyse the effect of gender on the model performance. Building upon the work of Qin and Yang (2019); Yang et al. (2020) our main focus is to learn a function f ( E { T,A } )  v [0 , ] , over   { 3 , 7 , 15 , 30 } days to evaluate the bias for different time periods. Earnings Call Data We use the dataset 2 created by Qin and Yang (2019) comprising 559 public earnings calls audio recordings with their transcripts for 277 companies in the S&P 500 index spanning over a year of earnings calls. The details of the dataset splits for training have been given in Table 1. For the identification of gender bias in the earnings calls acoustics, we first map the speakers from all the earnings calls to their self reported gender. For this we perform web scrapping from Reuters 3 (pronouns), Crunchbase 4 where the genders are self-declared and the available genders from the Wikidata API. The genders extracted correspond only to male and female, 11.8% of the speakers are female and 88.2% are male which motivates us to estimate the error disparity in model performance. Evaluating Gender Bias We use performance error disparity  G = MSE f  MSE m where f and m stand for female and male respectively (Saunders and Byrne, 2020). A higher  G is indicative of bias is in favour of the male distribution. Model Architecture and Training We use the state-of-the-art, Multimodal Deep Regression Model (MDRM) Qin and Yang (2019), as shown in Figure 1. MDRM takes utterancde level audio A and text T embeddings and models them through two contextual BiLSTM layers followed by late multimodal fusion. The fused text-audio features are fed to another BiLSTM followed by two fully-connected layers. MDRM is trained end-to-end by optimizing the mean square error (MSE) between the predicted and true stock volatility. Training Setup For textual features we use Fin-BERT embeddings 5 (Araci, 2019) with default 2 https://github.com/GeminiLn/ EarningsCall_Dataset 3 https://www.thomsonreuters.com/en/ profiles.html 4 https://www.crunchbase.com/discover/ people 5 https://github.com/ProsusAI/finBERT G = MSEF  MSEM   = 3  = 7  = 15  = 30 MDRM(A) 0.38 0.16 0.26 0.18 MDRM(T) 0.33 0.12 0.20 0.16 MDRM(AT) 0.30 0.11 0.28 0.14 Table 2: Modality specific  G i.e. the difference between the MSE for female and male distributions for 3, 7, 15 and 30 days over 5 runs. parameters and for audio cues, we extract 26 dimensional vectors with Praat (Boersma and Van Heuven, 2001) extracted by Qin and Yang (2019), spanning Shimmer, Jitter, Pitch, Intensity, etc. We report the complete list in Table 4. The maximum number of audio clips in any call is 520 . Hence, we zero-pad the calls that have less than 520 clips. The model is trained on TPU version 3.8 for 20 epochs using a learning rate of 0.001. The hyperparameters are tuned on the validation set defined by Qin and Yang (2019) following the same preprocessing. We perform 5 end-to-end runs with early stopping over the validation loss to arrive at the decision of training for 20 epochs. Bias in Multimodal Volatility Prediction For evaluating gender bias in MDRM, we analyze the error disparity quantified by  G for the individual text and audio modalities and their combination for  = 3,7,15,30 days. We tabulate the error disparity 1 2 3 4 5 6 7 8 0 . in terms of  G across modalities in Table 2 and performance in Table 3. We observe that for all modalities the error for male distribution is consistently less than that of female distribution for both shortand long-term durations. Although the audio modality improve model performance significantly, it has the highest amount of bias as audio features for males and females vary significantly. Further, the skewed distribution of speakers' gender in the earnings calls amplifies this error disparity. Over amplification refers to bias that occurs in a system during model fitting. The model learns imperfect generalizations between the attributes and the final labels and amplifies them while predicting on the test set. In our case, since female examples are very less in comparison to the male counterparts, the model discriminates between male and female examples by inferring insufficient information beyond its source base rate as shown in Table 2. To study this effect we train the model for different training sample ratios as per gender to observe performance variation in Figure 2. We note that as the male:female training ratio increases, the test loss is amplified the most in audio modality followed by audio+text and text. Test MSE male decreases in comparison to increase in MSE female . MSE female increases as the percentage of female examples in the train set decreases as the generalised notions of this underrepresented community are learnt and the incorrect inferences do not harm the overall performance much. Since the difference in test loss for male and female is significantly less when the number of samples across genders Audio features P value Bonferroni Pitch Analysis Mean Fundamental Frequency (F0)  Stdev Fundamental Frequency (F0)  Number of pulses  * Number of periods  * Degree of voice breaks  Maximum Pitch  Minimum Pitch  Voiced Frames  Voiced to Unvoiced Ratio  Voiced to Total Ratio  Intensity Analysis Mean Intensity  SD Energy  Maximum Intensity  Minimum Intensity  Voice Analysis Local Jitter  * Local Absolute Jitter  * Relative Average Perturbation Jitter  * Period Perturbation Quotient-5 Jitter  * ddp Jitter  * Local Shimmer  * Local dB Shimmer  * apq3 Shimmer  * apq5 Shimmer  * apq11 Shimmer  * dda Shimmer  * Harmonicity Analysis Harmonic to Noise Ratio  Table 4: Comparison of the audio features for male and female speaker distributions. is equal. Through this observation, we note that performance for female examples can be improved by augmentation techniques or cross domain adaptation, which we leave for future work. Semantic Bias occurs in embeddings and representations of audio and textual data which learn unwanted stereotypes. For our case semantic bias occurs as the audio features are significantly different for male and female distributions. We analyze each audio feature for both distributions in Table 4. We find that 13 out of 26 features have a statistically significant difference under the two-tailed T-test (  = 0 . 05 ) after applying Bonferroni correction (Weisstein, 2004), a multiple comparison correction when multiple statistical tests are being performed. These differences in audio features of executives' speech can amplify the error disparity, as models may associate certain gender specific features such as Voice analysis-based features like Shimmer and Jitter. Degradation in the performance of speech models could be due to discernible noise and indiscernible sources like demographic bias: age, gender, dialect, culture, etc (Meyer et al., 2020; Hashimoto et al., 2018; Tatman and Kasten, 2017). Studies also show that AI can deploy biases against black people in criminal sentencing (Angwin et al., 2016; Tatman and Kasten, 2017). Although we only account for the gender bias in our study, we acknowledge that there could exist other kinds of bias due to age, accent, culture, ethnic and regional disparities in audio cues, as the publicly available earnings calls majorly have companies belonging to the US. Moreover, only publicly available earnings calls have been used limiting the scope of the data. This also limits the availability of genders in the data to only male and female. In the future, we hope to increase the amount of data to expand our study to more categories and types of sensitive attributes. Earnings calls provide company insights from executives proving to be high risk-reward opportunities for investors. Recent multimodal approaches that utilize these acoustic and textual features to predict the financial risk achieve state-of-the-art performance, but overlook the gender bias associated with speech. We analyze the gender bias in volatility prediction of earnings calls due to gender sensitive audio features and underrepresentation of women in executive positions. We observe that the while adding speech features improves performance, it also perpetuates gender bias, as the audio modality has the highest error disparity. We further probe into the sources of bias, and analyze audio feature variations across gender, and perform experiments with varying training data distributions. Our study presents the first analysis of its kind to analyze gender bias in multimodal financial forecasting to bridge the gap between fairness in AI, neural financial forecasting and multimodality.
abstain abstain abstain abstain abstain objective objective result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method other other other other other other other other other other other other other other other abstain other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result result abstain abstain abstain other abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain method result method objective