\section{Introduction}

It is typically shown to be beneficial for acoustic modeling to make full use of the future contextual information.  In the literature, there are variety of methods to realize this idea for different model architectures. For feed-forward neural network (FFNN), this context is usually provided by splicing a fixed set of future frames in the input representation\cite{DongYu}. \iffalse It also exists other approaches relating modifying FFNN model structures.\fi The authors in \cite{FSMN1,FSMN2,FSMN3} proposed a model called feedforward sequential memory networks (FSMN), which is a standard FFNN equipped with some learnable memory blocks in the hidden layers to encode the long context information into a fixed-size representation. The time delay neural network (TDNN) \cite{SPL2,SPL17} is another FFNN architecture which has been shown to be effective in modeling long range dependencies through temporal convolution over context.

As for unidirectional recurrent neural network (RNN), this is usually accomplished using a delayed prediction of the output labels\cite{ASRU6}. However, this method only provides quite limited modeling power of future context\cite{SPL}. While for bidirectional RNN, this is accomplished by processing the data in the backward direction using a separate RNN layer \cite{SPL4,SPL5,SPL6}. Although the bidirectional versions have been shown to outperform the unidirectional ones with a large margin \cite{SPL7,SPL8}, the latency of bidirectional models is significantly larger, making them unsuitable for online speech recognition. To overcome this limitation, chunk based training and decoding schemes such as context-sensitive-chunk (CSC) \cite{SPL10, Chen2016A} and latency-controlled (LC) BLSTM \cite{SPL7, SPL12} have been investigated. However, the model latency is still quite high, \iffalse since in all these online variants, inference is restricted to chunk-level increments to amortize the computation cost of backward RNN.\fi  for example, the decoding latency of LC-BLSTM in \cite{SPL12} is about 600 ms. To overcome the shortcomings of the chunk-based methods, Peddinti \emph{et al.} \cite{SPL} proposed the use of temporal convolution, in the form of TDNN layers, for modeling the future temporal context while affording inference with frame-level increments. The proposed model is called TDNN-LSTM, and is designed by interleaving of temporal convolution (TDNN layers) with unidirectional long short-term memory (LSTM)  \cite{ASRU1,ASRU2,ASRU3,ASRU4} layers. This model was shown to outperform bidirectional LSTM in two automatic speech recognition (ASR) tasks, while enabling online decoding with a maximum latency of 200 ms \cite{SPL}. However, TDNN-LSTM's ability to model the future context comes from the TDNN part, whereas the LSTM itself is incapable of utilizing the future information effectively.

Recently, in \cite{my}, we proposed a RNN model called minimal gated recurrent unit with input projection (mGRUIP), in which the inserted input projection forms a bottleneck and a context module called \emph{temporal convolution}, is specifically designed on it to model the future context. This model, mGRUIP with context module (mGRUIP-Ctx, for short in this paper), is able of utilizing the future context effectively and directly, meanwhile with quite low model latency and computation cost.

\iffalse This work continues to improve the performance of mGRUIP (with context module) by a two-fold revision:\fi
In this work, we continue to improve the proposed model mGRUIP-Ctx. The revision is two-fold: First, we investigate how to use batch normalization (BN) \cite{BN-Google} in this model. \iffalse Experiments in \cite{my}\cite{my} have shown that on a 1400 hour Mandarin ASR task, mGRUIP-Ctx gives 3$\sim$5\% relative CER reduction over TDNN-LSTM. However, \iffalse after that\fi after that, we find that when the training data is augmented 6 times using speed perturbation (x3) [Japan23] and noise/reverberation perturbation (x2)[Japan24], mGRUIP-Ctx's gain over TDNN-LSTM disappears (as shown in section). In addition, mGRUIP-Ctx's training log reveals that the training convergence is not very stable.\fi
The starting point is that we found the proposed mGRUIP-Ctx model performs not quite well if the training data contains a lot of noisy speech, for example, if the original clean data is perturbed with noise and reverberation. This prompts us to use batch normalization to improve the convergence of optimization process. In our previous work\cite{my}, batch normalization is only used on the cell ReLU activation to deal with numerical instabilities originating from the unbounded ReLU functions. In this work we find it is also beneficial to apply BN to the update gate as well. In the literature, batch normalization has been applied to RNN in different ways. In \cite{lightRNN49}, the authors suggest to apply it to input-to-hidden (ItoH) connections only, whereas in \cite{lightRNN50} the normalization step is extended to hidden-to-hidden (HtoH) transitions\iffalse, using separate statistics for each time-step\fi. Our finding in this work is slightly different with \cite{lightRNN49} and \cite{lightRNN50}. It is shown that the best method to apply BN in mGRUIP-Ctx is a \emph{hybrid} way, that is, applying BN to both ItoH and HtoH connections for the cell ReLU activation, while for the update gate, applying BN to ItoH only. \iffalse Experimental results on Switchboard task and 1400 hour (x6) internal Mandarin ASR task clearly demonstrate that doing so speeds up optimization and improves generalization.\fi Experimental results on several ASR tasks clearly demonstrate that doing so can speed up optimization and improve generalization, especially when the training data is augmented with perturbation.

The second revision is to enlarge the model context. In our previous work\cite{my}, the context module in mGRUIP-Ctx is restricted to just modeling the future context, leaving the history to be modeled by the RNN structure.  In this work, we release this restriction and allow the context module to model the future and history information simultaneously. \iffalse We empirically show that it's beneficial to the performance.\fi It is empirically shown that this method is beneficial to the performance. Besides that, we also find that enlarging the order of future context can future improve the performance\iffalse, with a cost of  increasing the model latency\fi. It should be noted that this context extension brings quite limited additional parameters, thanks to the small dimensionality of the input projection.

With these two revisions, mGRUIP-Ctx's performance is improved significantly. On a 8400 hours Mandarin ASR task (1400 hours original data with 6-fold augmentation), the revised mGRUIP-Ctx provides 6\% to 10\% relative CER reduction over the previous one in \cite{my}, demonstrating the effectiveness of the revision methods. Compared to LSTM, the relative improvement is 18\% to 37\%, and the gain over TDNN-LSTM is about 5\% to 12\%. This model even outperforms a very strong baseline, BLSTM, with much less parameters and only 290 ms decoding latency. The proposed model's superiority is further verified on a much larger Mandarin ASR task, which contains 60K hours of speech in total (10K hours original data with 6-fold augmentation). On this task, the gain of mGRUIP-Ctx over LSTM and TDNN-LSTM is 11\% to 20\% and 3\% to 11\%, respectively. Compared with a much stronger baseline, TDNN-BLSTM, mGRUIP-Ctx just shows 2\% to 6\% relative loss, with advantages of much less parameters and much lower latency.

\section {Prerequisites}
\subsection {mGRU}
mGRU, short for minimal gated recurrent unit, is a revised version of GRU model. It is proposed by \cite{Bengio,LightRNN} and contains two modifications: removing the reset gate and replacing the hyperbolic tangent function with ReLU activation. The model is defined by the following equations (the layer index $l$ has been omitted for simplicity):
{\setlength\abovedisplayskip{3pt}
    \setlength\belowdisplayskip{3pt}
\begin{eqnarray}
  && \bm{\mathrm{z}}_t = \sigma(\bm{\mathrm{W}}_z\bm{\mathrm{x}}_t + \bm{\mathrm{U}}_z\bm{\mathrm{h}}_{t-1} + \bm{\mathrm{b}}_z) \\
  && \bm{\mathrm{\widetilde{h}}}_t = ReLU(BN(\bm{\mathrm{W}}_h\bm{\mathrm{x}}_t )+ BN(\bm{\mathrm{U}}_h\bm{\mathrm{h}}_{t-1})) \\
  && \bm{\mathrm{h}}_t = \bm{\mathrm{z}}_t*\bm{\mathrm{h}}_{t-1} + (1-\bm{\mathrm{z}}_t)*\bm{\mathrm{\widetilde{h}}}_t
\end{eqnarray}}
In particular, $\bm{\mathrm{z}}_t$ is a vector corresponding to the update gate, of which the activation is element-wise logistic sigmoid functions $\sigma(\cdot)$. $\bm{\mathrm{h}}_t$ represents the output state vector for the current frame $t$, and $\bm{\mathrm{\widetilde{h}}}_t$ is the candidate state obtained with a ReLU function. The parameters of the model are $\bm{\mathrm{W}}_z$, $\bm{\mathrm{W}}_h$ (the ItoH connections), $\bm{\mathrm{U}}_z$, $\bm{\mathrm{U}}_h$ (the HtoH weights), and the bias vector $\bm{\mathrm{b}}_z$.

\subsection {mGRUIP}
mGRUIP is obtained by inserting a linear input projection layer into mGRU, leading to the following update equations:
{\setlength\abovedisplayskip{3pt}
    \setlength\belowdisplayskip{3pt}
\begin{eqnarray}

  && \bm{\mathrm{v}}_t = \bm{\mathrm{W}}_{v1}\bm{\mathrm{x}}_t + \bm{\mathrm{W}}_{v2}\bm{\mathrm{h}}_{t-1} \\
  && \bm{\mathrm{z}}_t = \sigma(\bm{\mathrm{W}}_z\bm{\mathrm{v}}_t + \bm{\mathrm{b}}_z) \\
  && \bm{\mathrm{\widetilde{h}}}_t = ReLU(BN(\bm{\mathrm{W}}_h\bm{\mathrm{v}}_t)) \\
  && \bm{\mathrm{h}}_t = \bm{\mathrm{z}}_t*\bm{\mathrm{h}}_{t-1} + (1-\bm{\mathrm{z}}_t)*\bm{\mathrm{\widetilde{h}}}_t
\end{eqnarray}}
where the current input vector $\bm{\mathrm{x}}_t$ and the previous output state vector $\bm{\mathrm{h}}_{t-1}$, are compressed into a lower dimensional space by weight matrix $\bm{\mathrm{W}}_{v1}$ and $\bm{\mathrm{W}}_{v2}$ respectively and added together to get a projected vector $\bm{\mathrm{v}}_t$. Then the update gate activation $\bm{\mathrm{z}}_t$ and the candidate state vector $\bm{\mathrm{\widetilde{h}}}_t$ are calculated based on it.

\subsection {mGRUIP-Ctx}
The input projection forms a bottleneck in mGRUIP, on which we design a context module, \emph{temporal convolution}, to effectively model the future context\cite{my}. For the $l$ th layer ($l>1$), equation (4) now becomes:
{\setlength\abovedisplayskip{3pt}
    \setlength\belowdisplayskip{3pt}
\begin{eqnarray}

  && \bm{\mathrm{v}}_t^l = \bm{\mathrm{W}}_{v1}^l\bm{\mathrm{\widetilde{x}}}_t^l + \bm{\mathrm{W}}_{v2}^l\bm{\mathrm{h}}_{t-1}^l
\end{eqnarray}
where $\bm{\mathrm{\widetilde{x}}}_t$ is the concatenation of the current input vector and the output state vector of preceding layer from serval future frames:
\begin{eqnarray}
  && \bm{\mathrm{\widetilde{x}}}_t = \left[\bm{\mathrm{x}}_t^l; \bm{\mathrm{h}}_{t+s\times i}^{l-1};\cdots;\bm{\mathrm{h}}_{t+s\times K}^{l-1}\right]
\end{eqnarray}}
In particular, $\bm{\mathrm{x}}_t^l$ is the input vector of layer $l$ ($\bm{\mathrm{x}}_t^l$ is actually $\bm{\mathrm{h}}_t^{l-1}$ since $l>1$), and $\bm{\mathrm{h}}_{t+s\times i}^{l-1}$ is the output state vector of layer $l-1$ on the $(t+s\times i)$th frame. $s\ge1$ is the step stride, $K\ge1$ is order of future context, and $1\le i \le K$ is loop index.

\section {Proposed Revisions}
\subsection {Applying Batch Normalization}
Our first refinement is to figure out the best way to apply BN in the proposed models. There are two possible locations to apply BN in the structure: the update gate and the cell activation. \iffalse Section 2 reviews the realization form defined in \cite{my}, where no BN is applied for the update gate while BN is applied to both ItoH and HtoH for the cell activation. In this part, we will try different BN configurations for the three models. \fi
\subsubsection {BN for the update gate}
Three different BN methods for update gate are experimented:
{\setlength\abovedisplayskip{3pt}
    \setlength\belowdisplayskip{1pt}
\begin{itemize}
  \setlength{\itemsep}{0pt}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \item $\textbf{\emph{No BN}}$ \\

  Just as defined by equation (1) and (5) in Section 2 for model mGRU and mGRUIP(-Ctx), respectively.
  \item $\textbf{\emph{BN on ItoH only}}$ \\
  For mGRU, equation (1) now becomes:
  \begin{eqnarray}
    \bm{\mathrm{z}}_t = \sigma(BN(\bm{\mathrm{W}}_z\bm{\mathrm{x}}_t)+ \bm{\mathrm{U}}_z\bm{\mathrm{h}}_{t-1})
  \end {eqnarray}
  For mGRUIP(-Ctx), equation (5) now becomes:
  \begin {eqnarray}
    \bm{\mathrm{z}}_t = \sigma(BN(\bm{\mathrm{W}}_z\bm{\mathrm{v}}_{1t})+\bm{\mathrm{W}}_z\bm{\mathrm{v}}_{2t})
  \end{eqnarray}
  where $\bm{\mathrm{v}}_{1t}$ and $\bm{\mathrm{v}}_{2t}$ are the projected vector from $\bm{\mathrm{x}}_t$ (or $ \bm{\mathrm{\widetilde{x}}}_t $ for mGRUIP-Ctx) and $\bm{\mathrm{h}}_{t-1}$, respectively.
  \item $\textbf{\emph{BN on ItoH and HtoH}}$ \\
  For mGRU, equation (1) now becomes:
  \begin{eqnarray}
    \bm{\mathrm{z}}_t = \sigma(BN(\bm{\mathrm{W}}_z\bm{\mathrm{x}}_t)+ BN(\bm{\mathrm{U}}_z\bm{\mathrm{h}}_{t-1}))
  \end {eqnarray}
  For mGRUIP(-Ctx), equation (5) now becomes:
  \begin {eqnarray}
    \bm{\mathrm{z}}_t = \sigma(BN(\bm{\mathrm{W}}_z\bm{\mathrm{v}}_t))
  \end{eqnarray}
\end{itemize}}

\subsubsection {BN for the cell activation}
For the cell activation, \emph{No BN} can probably cause numerical instabilities for ReLU, thus only two methods are tried: \iffalse \emph{BN on ItoH only} and \emph{BN on ItoH and HtoH}.\fi
{\setlength\abovedisplayskip{3pt}
    \setlength\belowdisplayskip{1pt}
\begin{itemize}
  \setlength{\itemsep}{0pt}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \item $\textbf{\emph{BN on ItoH only}}$ \\
  We first try this method for mGRU model. Equation (2) now becomes:
  \begin{eqnarray}
    \bm{\mathrm{\widetilde{h}}}_t = ReLU(BN(\bm{\mathrm{W}}_h\bm{\mathrm{x}}_t )+ \bm{\mathrm{U}}_h\bm{\mathrm{h}}_{t-1})
  \end{eqnarray}
  It degrades the performance significantly (shown in Table 1), thus no further trial is needed for mGRUIP(-Ctx).
  \item $\textbf{\emph{BN on ItoH and HtoH}}$ \\
  Just as defined by equation (2) and (6) in Section 2 for model mGRU and mGRUIP(-Ctx), respectively.

\end{itemize}}

\subsection {Enlarging Model Context}
Our second revision is to enlarge the model context. The context module is used to model not only the future context but also the history information. Equation (9) now becomes:
{\setlength\abovedisplayskip{3pt}
    \setlength\belowdisplayskip{3pt}
\begin{eqnarray}
  \bm{\mathrm{\widetilde{x}}}_t &= \left[\bm{\mathrm{x}}_t^l; \bm{\mathrm{h}}_{t-s_1\times i}^{l-1};\cdots;\bm{\mathrm{h}}_{t-s_1\times K_1}^{l-1}; \right. \\
  \nonumber & \qquad \ \ \, \left. \bm{\mathrm{h}}_{t+s_2\times j}^{l-1};\cdots;\bm{\mathrm{h}}_{t+s_2\times K_2}^{l-1}\right]
\end{eqnarray}}
where $s_1\ge1$ and $s_2\ge1$ is the step stride for the history and future context, respectively. $K_1\ge1$ is the order of history and $K_2\ge1$ is the order of future context.

{\setlength{\belowcaptionskip}{-0.2cm}
\setlength{\abovecaptionskip}{0.2cm}
\begin{table}[ht]\footnotesize
  \caption{Comparison of BN methods with mGRU and mGRUIP on Switchboard task.}
  \label{tab1}
  \centering
  \begin {tabular}{|cc|cc|ccc|}
  \toprule
   \multicolumn{2}{|c|} {Gate, BN on} & \multicolumn{2}{c|} {Cell, BN on} & \multicolumn{3}{c|}{WER (\%)} \\
   \midrule
   ItoH & HtoH & ItoH & HtoH & SWB & CHM & Total \\
   \midrule
   \multicolumn{7}{c}{mGRU} \\
  \midrule
   N & N & Y & Y & 10.2 & 20.6 & 15.5 \\
   Y & Y & Y & Y & \multicolumn{3}{c|}{diverge} \\

   Y & N & Y & Y & \textbf{10.1} & \textbf{19.9} & \textbf{15.1} \\
   Y & N & Y & N & 11.1 & 22.1 & 16.7 \\
  \midrule
  \multicolumn{7}{c}{mGRUIP} \\
  \midrule
  N & N & Y & Y & 9.8 & 19.0 & 14.5 \\
   Y & Y & Y & Y & 19.8 & 27.1 & 23.5 \\
   Y & N & Y & Y & \textbf{9.5} & \textbf{18.6} & \textbf{14.2} \\
  \bottomrule
  \end{tabular}
\end{table}}

\section {Experimental Settings}
This section will introduce the three ASR tasks used in this work. All the models in this paper are trained with LF-MMI objective function computed on 33Hz outputs \cite{SPL16}.

\subsection {Switchboard ASR Task}
The training data is 309-hour Switchboard-I data. Evaluation is performed in terms of WER on the full Switchboard Hub5'00 test set, consisting of two subsets: Switchboard (SWB) and CallHome (CHE). The experimental setup follows \cite{SPL16}.\iffalse and is described here briefly. The GMM-HMM system, which is used to generate the alignments and the numerator lattice for LF-MMI training; and the language model (LM) which is used to for decoding, are similar to those described in \cite{Chain2}. We use the speed-perturbation technique \cite{Chain16} for 3-fold data augmentation, and iVectors to perform instantaneous adaptation of the neural network \cite{Chain17}.\fi WER results are reported after 4-gram LM rescoring of lattices generated using a trigram LM. Please refer to \cite{SPL16} for more details.

\subsection {Medium-Scale Mandarin ASR Task}
The second task is an internal medium-scale mandarin ASR task, which actually has two sub-tasks. The first one contains 1400 hours original mobile recording data. The second one is with 6 times data augmentation using speed perturbation (x3) \cite{Japan23} and noise/reverberation perturbation (x2) \cite{Japan24}, resulting in 8400 hours training data in total.

\subsection {Large-Scale Mandarin ASR Task}
The large-scale mandarin ASR task contains 10K hours original data and is augmented 6 times, resulting in 60K hours training data in total. The performance of mandarin ASR tasks are evaluated on five public-available test sets, including three clean and two noisy ones. The clean sets are:
\begin{itemize}
\setlength{\itemsep}{0pt}
\setlength{\parsep}{0pt}
\setlength{\parskip}{0pt}
\item AiShell\_dev and AiShell\_test: the development and test set of the released corpus AiShell-1\cite{Aishell}, containing 14326 and 7176 utterances, respectively.
\item THCHS-30\_Clean: the clean test set of THCHS-30 database\cite{thchs30}, containing 2496 utterances.
\end{itemize}

The two noisy test sets are THCHS-30\_Car and THCHS-30\_Cafe, which are the corrupted version of THCHS-30\_Clean by car and cafeteria noise, respectively. The noise level is 0db.
\iffalse \begin{itemize}
\setlength{\itemsep}{0pt}
\setlength{\parsep}{0pt}
\setlength{\parskip}{0pt}
\item THCHS-30\_Car and THCHS-30\_Cafe: the corrupted version of THCHS-30\_Clean by car and cafeteria noise, respectively. The noise level is 0db.
\end{itemize}
\fi

  \setlength{\abovecaptionskip}{0.1cm}
  \setlength{\belowcaptionskip}{-0.5cm}

\begin{figure}
\flushleft
{\centering\includegraphics [width=8cm]{en_4170-B_064608-064704_new_r.eps}
\caption{Average activation of update gate for three mGRUIP models trained with different BN methods.}
\label{1}}
\end{figure} }

\section {Experimental Results}
\subsection {Applying Batch Normalization}
\subsubsection {Switchboard ASR Task}
On this task, we compare different BN methods for two models: mGRU and mGRUIP, and the results are shown in Table 1. \iffalse (top half part for mGRU and the rest for mGRUIP)\fi Both of them contain 5 layers, and each layer consists 1024 cells. The input projection layer for mGRUIP has 512 units.

Several observations can be obtained from Table 1. Firstly, comparing the first two lines of each model's results, we can see that applying BN to both ItoH and HtoH for update gate is quite harmful. The training of mGRU model even diverges.
We think the reason is as below. The update gate is controlled by a sigmoid function, and is expected to learn to open or close at the right time. \emph{BN on ItoH and HtoH} will keep the input away from the saturated regime of this nonlinearity. This may be helpful when sigmoid serves as a hidden node activation. However, when it's used to control a gate, this will be harmful since it will make the gate half-closed or half-opened, which is opposite to the responsibility of the gate control. To verify this,
we investigate the evolution of update activation when performing recognition. A speech segment is chosen from the test set (the utterance id is \emph{en\_4170-B\_064608-064704} with the text content \emph{That is quite a difference}), and the features are sent to the three mGRUIP models trained with different BN methods. The average activation of the update gate over cells (the 5th layer) for each frame is shown in Figure 1. It's very clear that applying BN to both ItoH and HtoH for the update gate will cause the gate activation fluctuating around 0.5, meaning the gate unable to effectively control the dependency between the history $\bm{\mathrm{h}}_{t-1}$ and candidate state $\bm{\mathrm{\widetilde{h}}}_t$. Another interesting finding is that, the average gate activations of \emph{No BN} and \emph{BN on ItoH} models are almost always greater than 0.5. We think it's because the speech signal is a sequence that evolves rather slowly\iffalse (30ms in this work)\fi, in which the past history can virtually always be helpful. \iffalse We plan to do more investigations on this in the future.\fi

{\setlength{\belowcaptionskip}{-0.2cm}
\setlength{\abovecaptionskip}{0.2cm}
\begin{table*}[h]\footnotesize
  \caption{Performance comparison of mGRUIP-Ctx with various settings of context module on 1400h Mandarin ASR task.}
  \label{tab3}
  \centering
  \begin {tabular}{c|c c c c |c|c|c|c|c|c}
  \toprule
   \multirow{3}*{Model} & \multicolumn {4} {c|} {\multirow{2}*{Layerwise Context Setting}} & \multirow{2}*{Latency} & \multicolumn {5} {c} {CER} \\
   \cline {7-11}

   & & & & & & \multicolumn {2} {c|} {AiShell} & \multicolumn {3} {c} {THCHS-30} \\
   & 2 & 3 & 4 & 5 & (ms) & dev & test & Clean & Car & Cafe \\
   \midrule
   mGRUIP-Ctx-A & {0; $1 \times 1$} & {0; $1 \times 3$} & {0; $1 \times 3$} & {0; $1 \times 3$} & 170 & 4.61 & 5.56 & 10.34 & 10.67 & 41.89 \\
   mGRUIP-Ctx-B & {$1 \times 6$; $1 \times 1$} & {$1 \times 6$; $1 \times 3$} & {$1 \times 6$; $1 \times 3$} & {$1 \times 6$; $2 \times 3$} & 200 & 4.49 & \textbf{5.46} & 10.11 & 10.46 & 39.79 \\
   mGRUIP-Ctx-C & {$2 \times 6$; $1 \times 1$} & {$2 \times 6$; $1 \times 3$} & {$2 \times 6$; $1 \times 3$} & {$2 \times 6$; $2 \times 3$} & 200 & 4.49 & 5.50 & 10.08 & 10.45 & \textbf{38.41} \\
   mGRUIP-Ctx-D & {$1 \times 6$; $1 \times 1$} & {$1 \times 6$; $1 \times 3$} & {$1 \times 6$; $1 \times 6$} & {$1 \times 6$; $2 \times 6$} & 290 & \textbf{4.47} & 5.47 & \textbf{9.95} & \textbf{10.36} & 38.76 \\
  \bottomrule
  \end{tabular}
\end{table*}}

The second observation from Table 1 is that we should apply BN to both ItoH and HtoH for the cell ReLU activation (comparing the last two lines of mGRU's results). It's possibly because applying BN this way will provide the best numerical stabilities for ReLU. Finally, we can conclude from Table \ref{tab1} that, the best method to apply BN in mGRU related models is a hybrid way, that is, applying BN to just ItoH for update gate, and for cell ReLU activation, applying BN to both ItoH and HtoH.

{\setlength{\belowcaptionskip}{-0.2cm}
  \setlength{\abovecaptionskip}{0.2cm}
  \begin{table}[ht]\footnotesiz
  \caption{Performance of different models on Medium-Scale Mandarin ASR task.}
  \label{tab2}
  \centering
  \begin {tabular}{|c|c|c|c|c|c|c|}
  \toprule
   \multirow{3}*{Model} & \multirow{2}*{Gate} & \multicolumn {5} {c|} {CER} \\
   \cline {3-7}
   & & \multicolumn {2} {c|} {AiShell} & \multicolumn {3} {c|} {THCHS-30} \\
   & BN & dev & test & Clean & Car & Cafe \\
   \midrule
   \multicolumn{7}{c}{1400 Hours Original Data} \\
   \midrule
   TDNN-LSTM                          & -          & 4.81 & 5.98 & 10.97 & 11.38 & 44.20 \\
   \cmidrule {1-2} \cmidrule {3-7}
   \multirow{2}*{mGRUIP-Ctx}& No\cite{my}      & 4.66 & 5.71 & 10.38 & 10.77 & \textbf{40.26} \\
                                                    & ItoH   &  \textbf{4.61} & \textbf{5.56} & \textbf{10.34} & \textbf{10.67} & 41.89 \\
   \midrule
   \multicolumn{7}{c}{8400 Hours Augmented Data} \\
   \midrule
   TDNN-LSTM                          & -          & 4.50 & 5.42 & 10.11 & 10.30 & 23.86 \\
   \cmidrule {1-2}\cmidrule {3-7}
   \multirow{2}*{mGRUIP-Ctx}& No      & 4.45 & 5.42 & 10.24 & 10.34 & 23.43 \\
                                                    & ItoH   &  \textbf{4.33} & \textbf{5.26} & \textbf{10.00} & \textbf{10.02} & \textbf{22.35} \\
 \bottomrule
  \end{tabular}
\end{table}}

\subsubsection {Medium-Scale Mandarin ASR Task}
According to the findings in Section 5.1.1, we apply the optimal BN methods to the model mGRUIP-Ctx on the medium-scale mandarin ASR task, and the results are shown in Table \ref{tab2}. \iffalse The top half part is the results for the model trained with 1400 hours original data, and the bottom part gives the model performance with data augmentation (8400 hours). The terms \emph{No BN} and \emph{BN} in Table 2 are specific to the update gate, meaning applying no BN  and applying BN to ItoH, respectively.\fi The baseline model is TDNN-LSTM \cite{SPL} which has interleaving 7 TDNN and 3 LSTM layers. The model mGRUIP-Ctx contains 5 hidden layers, each one has 2560 cells and a 256-dimensional input projection layer. The settings of context order $K$ and step stride $s$ for each layer is same as our previous work \cite{my}.

According to Table \ref{tab2}, with 1400 hours original training data, mGRUIP-Ctx with no BN on update gate gives 3\% to 5\% relative gain over TDNN-LSTM. However, this gain disappears after the training data is augmented with perturbation. It can be attributed to the different difficulties of these two sub-tasks, of which model learning with perturbed data is harder than with original data. After BN is correctly applied, mGRUIP-Ctx performs better than TDNN-LSTM on the harder sub-task, demonstrating the effectiveness of the BN method. Moreover, the training curve reveals that the optimization process becomes much stable (not shown in this paper).

\subsection {Enlarging Model Context}
\subsubsection {Medium-Scale Mandarin ASR Task}
Our second revising method is to enlarge the model context. We compare the performance of mGRUIP-Ctx with different settings of context order $K_1$, $K_2$ and step stride $s_1$, $s_2$ (\iffalse following equation (15)\fi BN methods are correctly used). For fast experiments, all the models are trained with 1400 hours original data, and the results are reported in Table \ref{tab3}. The setting of context order and stride for each layer is represented with a format of $\{K_1\times s_1$;$K_2 \times s_2\}$.

From Table \ref{tab3}, comparing  mGRUIP-Ctx-A and  mGRUIP-Ctx-B, it can be seen that allowing context module to model the history information as well is quite beneficial to the performance (mGRUIP-Ctx-A is just the model used in Section 5.1.2). In addition, increasing the order of future context can further improve the performance, with a cost of additional model latency ( mGRUIP-Ctx-D vs.  mGRUIP-Ctx-B).

Next, the best model mGRUIP-Ctx-D is compared with several baselines, including BLSTM, TDNN-LSTM and LSTM, on the 8400 hours data-augmented task, and the results are presented in Table 4. BLSTM and LSTM both contain 5 hidden layers. Each layer of LSTM and each directional sub-layer of BLSTM, have 1024 cells and 512 linear projection units. TDNN-LSTM is the model used in Section 5.1.2.

{\setlength{\belowcaptionskip}{-0.2cm}
\setlength{\abovecaptionskip}{0.2cm}
\begin{table}[ht]\footnotesiz
  \caption{Performance of different models on 8400h Mandarin ASR task.}
  \label{tab4}
  \centering
  \begin {tabular}{|c|c|c|c|c|c|c|}
  \toprule
   \multirow{3}*{Model} & \multirow{2}*{\scriptsize{Latency}} & \multicolumn {5} {c|} {CER} \\
   \cline {3-7}
   & & \multicolumn {2} {c|} {AiShell} & \multicolumn {3} {c|} {THCHS-30} \\
   & (ms) & dev & test & Clean & Car & Cafe \\
   \midrule
   BLSTM           & 2020 & 4.24 & 5.05 & 9.79 & 9.87 & 22.82 \\
   TDNN-LSTM & 210   & 4.50 & 5.42 & 10.11 & 10.3 & 23.86 \\
   LSTM             & 70     &  5.07 & 6.23 & 12.06 & 11.82 & 33.67 \\
   \midrule
   \scriptsize{mGRUIP-Ctx-D}  & 290 & \textbf{4.13} & \textbf{5.03} & \textbf{9.56} & \textbf{9.71} & \textbf{20.94} \\
   \midrule
   CERR (\%) &   -   & 18.5 & 19.3              & 20.7 & 17.9 & 37.8 \\
 \bottomrule
  \end{tabular}
\end{table}}

The \emph{CERR} line in Table 4 is the relative gain of mGRUIP-Ctx-D over LSTM, ranging from 18\% to 38\% for different test sets, indicating that the proposed model performs significantly better than LSTM. Compared with TDNN-LSTM, mGRUIP-Ctx-D gives 5\% to 12\% relative CER reduction with 12.4M less parameters (22.4M vs. 34.8M) and 80 ms additional latency (290ms vs. 210ms). mGRUIP-Ctx-D even outperforms BLSTM with much lower model latency (290ms vs. 2020ms) and much less model parameters (22.4M vs. 55.3M). Compared to the performance of mGRUIP-Ctx before revision (the next-to-last line of Table \ref{tab2}), mGRUIP-Ctx-D shows 6\% to 10\% relative gain, which demonstrates the effectiveness of the two proposed revising methods.

\subsubsection {Large-Scale Mandarin ASR Task}
Finally, mGRUIP-Ctx-D's superiority is verified on a much larger mandarin ASR task, which contains 60K hours speech data. We train a TDNN-BLSTM model as one baseline, which is stronger than BLSTM and contains 3 TDNN layers and 5 BLSTM layers. All of the other models have the same structure as Section 5.2.1. Results are shown in Table 5.

{\setlength{\belowcaptionskip}{-0.2cm}
 \setlength{\abovecaptionskip}{0.2cm}
\begin{table}[ht]\footnotesiz
  \caption{Performance of different models on 60K hour Mandarin ASR task.}
  \label{tab5}
  \centering
  \begin {tabular}{|c|c|c|c|c|c|}
  \toprule
   \multirow{3}*{Model} &  \multicolumn {5} {c|} {CER} \\
   \cline {2-6}
   & \multicolumn {2} {c|} {AiShell} & \multicolumn {3} {c|} {THCHS-30} \\
   &  dev & test & Clean & Car & Cafe \\
   \midrule
   TDNN-BLSTM    & \textbf{3.55} & \textbf{4.21} & \textbf{8.72} & \textbf{8.85} & \textbf{18.73} \\
   TDNN-LSTM      & 3.90 & 4.68 & 9.55 & 9.65 & 21.53 \\
   LSTM                  &  4.32 & 5.26 & 10.3 & 10.33 & 23.84 \\
   \midrule
   mGRUIP-Ctx-D   & 3.71 & 4.52 & 9.09 & 9.16 & 19.06 \\
   \midrule
   CERR (\%)    & 14.1 & 14.1 & 11.8 & 11.3 & 20.1 \\
 \bottomrule
  \end{tabular}
\end{table}}

According to Table 5,  the relative improvement of mGRUIP-Ctx-D over LSTM is 11\% to 20\%, and the gain over TDNN-LSTM is 3\% to 11\%. mGRUIP-Ctx-D performs slightly worse (2\% to 6\% relative) than the strong baseline, TDNN-BLSTM, but has two advantages: much less parameters and much lower latency.

\section{Conclusions}
In this paper, we improve our previous proposed model mGRUIP-Ctx with two revisions: applying BN methods and enlarging model context. After revising, mGRUIP-Ctx outperform LSTM with a large margin. It even performs slightly better than a superior BLSTM on one task, with much less parameters and much lower latency.

\bibliographystyle{IEEEtran}


\section{Introduction}
Recent research has dampened the consensus about the unpredictability of market prices.
As a result, predicting the stock market has garnered the attention of diverse fields and domains where different information channels have been explored,
such as news~\cite{tetlock2007giving, tetlock2008more, Tobias:2013}, search engines~\cite{PreisCurme:2014,Preis5707, citeulike:12299800, JOFI:JOFI1679}
and, more recently, social media~\cite{Bollen20111, citeulike:13108056, 1507.00784, 2016arXiv160104535S}.

Social media is of particular interest due to the high volume and velocity of activity of an ever-evolving network that is constantly providing and creating information.
It offers real-time information that can be attributed to specific people, events, markets, and securities.
This is channeled through the use of so-called \textit{cashtags} (e.g., `\$AAPL') in messages, enabling the creation of feeds associated with particular stock symbols.
The use of \textit{cashtags} has since been propagated onto Twitter, thus providing a means to direct social sentiment to the specific stock or
securities of that it references.
As a result, social media user sentiment can be harnessed as a source of information with respect to financial market activity.

Now, the opinions of traders, professional bloggers and analysts, along with laypeople's opinions,
are aggregated in a dynamic social network that can possibly explain some of the variance in market behavior.
Initial research~\cite{yahoo1} sought to quantify a relationship between social media analytics with financial market data such as daily returns.
The observed results outperformed baseline trading strategies, providing evidence that the volume of tweets can reduce uncertainty about financial returns.
However, volume-based methods disregard any possible predictive information from \textit{qualitative} aspects of the data (i.e., the actual content or content polarity).
By applying sentiment analysis techniques \cite{1507.00955} to a corpus of tweets,
a sentiment score or emotion classification can be derived to quantify this qualitative dimension~\cite{1507.00955}.
The effectiveness of this semantic approach has been examined in~\cite{Bollen20111}, where collective moods analyzed from large
collections of daily tweets were used to increase an existing financial predictor's performance to an accuracy of 87.6\%.

While a broad analysis across several financial securities might reveal that social signs are relevant for explaining financial dynamics to some extent~\cite{Bollen20111, citeulike:13108056},
little is known about confounding factors that distinguish assets with predictive social signs from assets with no extra information provided by social media.
Leveraged by a non-parametric analysis founded in information-theoretic measures, we demonstrate that social signs can be useful for most stocks from both NYSE and NASDAQ exchanges.
This alone is an interesting and very sound result, compared to current literature' however, we extend this analysis to provide, for the first time,
possible explanations of features that might be essential to distinguish predictive social signs from non-predictive ones.

\subsection{Research Questions}

\begin{itemize}
    \item \textbf{RQ1. \textit{Which stocks from the NASDAQ and NYSE exchanges exhibit a significant information surplus when using social media as a leading indicator of the stocks' future volatility?}} \\
    For each stock, lead-lag mutual information analysis is performed between financial and social media data.

    \item \textbf{RQ2. \textit{Under what configuration of social media and financial variables are social media analytics informative of future financial movements?}} \\
    We aim to determine the feature profile (using financial and social media variables) of companies that are most indicative of a statistically significant
    social media lead-time information.
\end{itemize}

\section{Methodology}

\subsection{Data}
The data correspond to the daily social media analytics and market quotes of NYSE and NASDAQ listed companies between 01-01-2012 and 01-01-2016.

Social media data were provided by PsychSignal \cite{PsychSignal}, which operates a customized
Twitter and StockTwits collection framework tracking messages containing \textit{cashtags} (e.g., \$AAPL).
State-of-the-art natural language processing algorithms were applied to relevant messages,
labeling each with a sentiment disposition (i.e., bullish or bearish) and a measure of disposition intensity.
A daily aggregate of these data was provided for each tracked stock.
Furthermore, for each stock with available social media data, we also considered the historic records of daily stock quotes.
The following data sets were utilized:
\begin{itemize}
    \item \textbf{PsychSignal Social Media Database.} Sentiment daily aggregates containing the following stock quotes attributes:
\begin{itemize}
  \item \texttt{SYMBOL}: the stock symbol (ticker) to which the sentiment data refers to;
  \item \texttt{TIMESTAMP\_UTC}: the date and time of the analyzed data in UTC format;
  \item \texttt{BULLISH\_INTENSITY}: positive sentiment polarity;
  \item \texttt{BEARISH\_INTENSITY}: negative sentiment polarity;
  \item \texttt{BULL\_SCORED\_} \texttt{MESSAGES}: positive sentiment volume, number of messages;
  \item \texttt{BEAR\_SCORED\_} \texttt{MESSAGES}: negative sentiment volume, number of messages;
  \item \texttt{TOTAL\_SCANNED\_} \texttt{MESSAGES}: total number of scanned messages.
\end{itemize}
    \item \textbf{Google Finance.} Daily Market Quotes:
    \begin{itemize}
    \item \texttt{OPEN}: daily opening price;
    \item \texttt{HIGH}: daily high price;
    \item \texttt{LOW}: daily low price;
    \item \texttt{CLOSE}: daily closing price;
    \item \texttt{VOLUME}: financial volume, number of daily trades.
    \end{itemize}
\end{itemize}

\subsubsection{Variables Analyzed}
We used a value of Daily True Range (TR) (see Equation \ref{TRequation}) as a measure of financial volatility.
A time series of volatility data for each stock was derived using financial quotes as follows:
\begin{equation}\label{TRequation}
    \begin{split}
        TR_{t} = \textup{max}[(\textup{High}_{t}-\textup{Low}_{t}), \\ (\textup{Low}_{t}-\textup{Close}_{t-1}),\\(\textup{High}_{t}-\textup{Close}_{t-1})]
    \end{split}
\end{equation}

We applied a principal component analysis (PCA) to a set of social media variables in order to obtain a time series that contains the majority of underlying information from the sentiment features considered.
\texttt{PCA\_SOCIAL\_CHANGE} is defined as the daily change in the time series obtained by applying the PCA and extracting
the first principal component from the set of social media variables considered, as defined in Table \ref{sentvariables}.
Principal component analysis is a dimensionality reduction technique that we utilized for feature extraction.
The PCA permits the reduction of numerous correlated, co-linear variables to a component (or feature set of components).
Tables \ref{sentvariables} and \ref{finvariables} list the features utilized in this work.
\input{Tables/sent_variables.txt}
\input{Tables/fin_variables.txt}

\subsection{Information Surplus}
Information surplus~\cite{citeulike:13108056} was derived from mutual information, a measure of the mutual dependence between two data sets.
Let $S$ be the variable \texttt{PCA\_SOCIAL\_CHANGE}, which is the daily change in the time series obtained by applying the PCA
and extracting the first principal component from the set of social media variables as defined in Table \ref{sentvariables} and
let $F$ be \texttt{LOG\_TR\_DIFF}, which is the log of the daily difference in the Daily True Range (see Equation \ref{TRequation}).
If the addition of series $S$ provides information about the movements of series $F$, then it is said that
a dependency or mutual information ($MI$) exists between $S$ and $F$. However, such a dependency is non-directional;
in order to determine whether $S$ \textit{leads} $F$, $S$ must provide more information on a lagged series of $F$ than the baseline $MI$ (i.e. non-lagged).
Determining whether a baseline dependency exists between the series of a social media feature $S_{l=0}$ and a
financial feature $F_{l=0}$ on the same day is the first step in identifying if $S$ leads or is predictive of $F$.
Mutual information tells us how much the information about $S$ reduces the uncertainty of $F$.
Equation (\ref{MI}) shows the formal form, where we attempt to reduce uncertainty or
increase information by taking the double integral over the log of the joint entropy for both series over each distribution.

\begin{equation}\label{MI}
MI(S;F)=\int\int f(s,\textup{f})\log\frac{f(s,\textup{f})}{f_{s}(s)f_{\textup{f}}(\textup{f})}\,ds\,df
\end{equation}

The data need to be grouped into bins to determine the mutual information between the two series. This is necessary
to calculate entropy because the probability of observing an instance $i$ in each bin $s$ and $f$ constitutes the probability distributions.
The number of bins is dependent on the size of the data; therefore, our bin sizes were typically identical
per feature (i.e., there are roughly 365 daily instances of tweets and financial data per security per year).
The bin size $k$ was calculated using Sturge's Rule (see Equation \ref{MI2}), which has been found to be more accurate than comparable methods when used to calculate entropy in the $MI$ algorithm~\cite{bin}.

\begin{equation}\label{MI2}
k = \textup{log}_{2}n + 1
\end{equation}

Mutual information was then computed at consecutive daily time lags. The information \textit{gain} at time lag $l=i$ is
calculated by finding the difference between mutual information at $l=i$ and $l=0$, where $i$ is a certain day lag
and $l=0$ corresponds to the baseline case. Information surplus is expressed as a percent of the $MI$ above what we would expect over the given time frame.
Therefore, if we achieve a surplus above the average $MI$ from $l=i$ to $l=0$, then the social media time series $S$ is said to lead $F$.

\begin{equation}\label{IS}
\textup{Information Surplus}_{l} = \frac{MI(S;F)_{l=i}-MI(S;F)_{l=0}}{MI(S;F)_{l=0}} \times 100
\end{equation}

\subsection{Validating Significant Information Surplus}
To validate whether the information surplus is statistically significant, we verified whether information is more leading than trailing, and we compared the
obtained results with those obtained via a randomly permuted time-series.

We first filtered companies whose surplus is more \textit{trailing} than it is \textit{leading}
by identifying stocks for which the daily changes in $S$ carry more information about the daily $F$ in hindsight ($l=-i$)
rather than during the same ($l=0$) or a previous day ($l=i$).
We calculated $MI$ on a forward shift (\textit{ex-post}) and on a backward shift (\textit{ex-ante}),
eliminating stocks where the \textit{ex-post} $MI$ is greater than the \textit{ex-ante} $MI$.
Where the $MI$ for a lag is less than the $MI$ for a retrospective advance of the series (i.e. $MI_{l=i}<MI_{l=-i}$),
we can assert that $MI(S;F)$ trails more trailing than it leads, and it is thus insignificant.

A random permutation of the remaining symbols was then performed 100 times. With $\alpha = 0.05$,
the stocks must outperform 95\% of the randomly permuted data to be considered to have a significant surplus.

\subsection{Clustering}
To identify the conditions under which social signs may be predictive of financial volatility,
we utilized a clustering method to determine the configuration of social media and financial variables that is
indicative of a high information surplus. Each stock was represented as a vector of social media and financial variables.
In addition, the respective scores of information surplus and the features that describe its nature
(e.g., the size of the lag it was obtained at) were included. Stocks with similar configurations of social media and financial variables were
grouped together in clusters.

Inspecting the feature profiles of the clusters (i.e., the average representation of constituent stocks' features)
allows us to identify the social signs of financial dynamics that are indicative of a high information surplus.
Clusters with a profile containing many significant lags and high overall information surplus will provide us insight into the
configuration of variables that characterizes a stock with predictable volatility.

We used k-means clustering, which is an unsupervised learning algorithm that partitions instances into \textit{k} clusters by minimizing the within-cluster sum of square error (WCSS) between instances in each set $S$ using a distance metric (see Equation \ref{kmeans}).

\begin{equation}\label{kmeans}
\textup{argmin}\, S\sum_{i=1}^{k}\sum_{x\in S_i}\left \| x - u_{i} \right \|^2
\end{equation}

\section{Descriptive Statistics}

The following descriptive statistics provide context for our results.
Figure \ref{pies} presents the sector breakdown of the top 250 stocks by market cap for both the NASDAQ and NYSE (i.e. 500 stocks in total).
In the context of this work, it is interesting to note that the NASDAQ is typically characterized as being more volatile than the NYSE exchange \cite{vol2}.

\begin{figure}[h]
    \centering
       \begin{subfigure}[t]{0.49\textwidth}
            \caption{\textbf{A) NASDAQ}}
     \includegraphics[width=\textwidth]{Figures/pie_nasdaq}
    \end{subfigure}
     \begin{subfigure}[t]{0.49\textwidth}
          \caption{\textbf{B) NYSE}}
     \includegraphics[width=\textwidth]{Figures/pie_nyse}
    \end{subfigure}
    \caption{\textbf{Sector breakdown of selected companies by exchange.} Percentage breakdown by sector highlights both that the majority of large cap stocks belong to technology and financial sectors and that the NYSE has a more balanced distribution.}
    \label{pies}
\end{figure}
\FloatBarrier

\begin{figure}[ht]
    \centering
     \begin{subfigure}[t]{0.49\textwidth}
     \caption{\textbf{A) NASDAQ}}
     \includegraphics[width=\textwidth]{Figures/tweet_vol_nasdaq}
    \end{subfigure}
     \begin{subfigure}[t]{0.49\textwidth}
     \caption{\textbf{B) NYSE}}
     \includegraphics[width=\textwidth]{Figures/tweet_vol_nyse}
    \end{subfigure}
    \caption{\textbf{Probability distribution of tweet volume.} There is a wide discrepancy between the NYSE and NASDAQ tweet volumes for the selected companies.
    The NASDAQ contains several outliers (most notably, \$APPL) that skew the distribution towards the high end.
    The NASDAQ contains a larger number and a more variable distribution of tweets, with a total of 324,239, and a standard deviation of 5,489,
    in contrast to 212,368 and 1,241, respectively, for the NYSE.}
    \label{hist}
\end{figure}
\FloatBarrier

\begin{figure}[!h]
    \centering
     \begin{subfigure}[t]{0.49\textwidth}
     \caption{\textbf{A) NASDAQ}}
     \includegraphics[width=\textwidth]{Figures/mktcap_vol_nasdaq}
    \end{subfigure}
     \begin{subfigure}[t]{0.49\textwidth}
     \caption{\textbf{B) NYSE}}
     \includegraphics[width=\textwidth]{Figures/mktcap_vol_nyse}
    \end{subfigure}
    \caption{\textbf{Market cap and Tweet Volume Positive Relationship .} The NYSE and NASDAQ selected companies exhibit a
    moderate positive correlation between market capitalization (x) and average monthly tweet volume (y).}
    \label{regression}
\end{figure}

Figures \ref{hist} and \ref{regression} compare the relationship between the volume of tweets regarding
exchange-specific securities and the size of those securities.
In summary, there is a strong right skew in tweet volume, which contributes to a moderately
positive relationship between the size of a security and its interest to investors as quantified by tweet volume.
In both exchanges, there are several notable outliers, such as \$AAPL that contains a disproportionate volume of tweets.

\begin{figure}
    \centering
    \includegraphics[width=.95\textwidth]{Figures/cor_mat_nasdaq}
    \caption{\textbf{NASDAQ correlation matrix.} Upper right (and low left) quadrant reveals weak to no correlation between social and financial features. The NYSE features exhibit comparable correlation.}
    \label{cor_plot}
\end{figure}

\begin{figure}[!h]
    \centering
    \begin{subfigure}[t]{0.45\textwidth}
    \centering
    \caption{\textbf{A)}}
     \includegraphics[width=\textwidth]{Figures/b_minus_b_PDF}
    \end{subfigure}
    ~
     \begin{subfigure}[t]{0.45\textwidth}
     \centering
     \caption{\textbf{B)}}
     \includegraphics[width=\textwidth]{Figures/log_tr_diff_PDF}
     \end{subfigure}
    \caption{\textbf{Probability Distribution Examples with Fitted Normal Curve.} The plots are exemplary of the non-normal distributions found in both our social media (A) and financial features (B) for the NASDAQ and NYSE. }
    \label{pdfplot}
\end{figure}

\iffalse
\begin{figure}[h]
    \centering
    \subfloat[]{
     \includegraphics[width=.23\textwidth]{Figures/PCA_sent_change_nowindow_ACPlot}
    }
    \subfloat[]{
     \includegraphics[width=.23\textwidth]{Figures/tr_ACPlot}
    }
    \caption{\textbf{Autocorrelation Plots.} Plots for the features used to calculate $MI$. $S_{PCA}$ (a) and $F_{TR}$ (b) in both the NASDAQ and NYSE exhibit no significant autocorrelation in these features as demonstrated by the example above for \$GE. Consequently, no rolling window is applied to the data.}
    \label{acfPlot}
\end{figure}
\fi

\iffalse
\begin{figure}[h]
    \centering
    \subfloat[NASDAQ]{
     \includegraphics[width=.23\textwidth]{Figures/regress_nasdaq}
    }
    \subfloat[NYSE]{
     \includegraphics[width=.23\textwidth]{Figures/regress_nyse}
    }
    \caption{\textbf{Primary Feature Relationship.} In both (a) and (b), a negative correlation is observed between $S_{PCA}$ (x) and $F_{TR}$ (y): the two feature series used to calculate $MI$.}
    \label{regress}
\end{figure}
\FloatBarrier
\fi

In Figures \ref{cor_plot} and \ref{pdfplot},
we present a range of plots related to the feature behavior of the data utilized for calculating the $MI$.
The correlation plot between all features (see tables \ref{sentvariables} and \ref{finvariables}) in Fig. \ref{cor_plot} reinforces the use of
PCA dimensionality reduction. As the features are correlated, it is useful to inject a compression of these variables into our MI calculation.
A correlation was found among social media features pertaining to, for instance, sentiment polarity, the volume of messages as well as with daily financial features.

The majority of features is also characterized by non-normal distributions (see Fig. \ref{pdfplot}).
For features that are highly dependent on market capitalization (e.g., mean average monthly message volume as (see Figure \ref{hist})),
log-normal behavior exists. Furthermore, NASDAQ and NYSE stocks exchanges exhibit comparable results.

\section{Results}
We used the outlined information surplus method~\cite{citeulike:13108056} and data from 01/01/2012 to 01/01/2016
to determine which stocks in the NYSE and NASDAQ exhibit, on average, a significant leading information surplus.
We then built on this method by clustering stocks and examining which configuration of variables is indicative of a high information surplus.

\subsection{Reducing Uncertainty about Volatility}

Experiments on the NASDAQ and NYSE were carried out in tandem, and they produced comparable results.
We found 101 stocks from the NASDAQ that exhibit a leading information surplus when using social media as an indicator of the daily change in True Range (our measure of volatility, see Equation \ref{TRequation}).
Of the original 250 stocks examined, 149 did not have a significant surplus, meaning that the surplus in each time lag over a 10-day
period did not exceed the expected surplus. Thus, for example, the periods in Fig. \ref{MI_sample} where the blue \textit{ex-ante}
series is below the average ex-post for the
10-day period do not contain a significant leading surplus.
All 101 stocks with a significant leading surplus, however, passed through the second validation test by performing better than 95\% of the randomly permuted data.

For the NYSE, 91 of the original 250 stocks exhibited a significant leading surplus. In addition, only one company did not pass the second validation test.

The full list of significant stocks from the NASDAQ and NYSE can be found in Appendix \ref{appendix}.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.8\textwidth]{Figures/MI_sample}
    \caption{\textbf{Sample Information Surplus.} For each lag, the surplus is calculated as the amount of $MI$ above the base case, $MI_{t=0}$.
    A significant \textit{ex-ante} surplus occurs where the surplus is greater than the average \textit{ex-post} surplus for the 10-day window.}
    \label{MI_sample}
\end{figure}

\subsection{Leading Surplus Indicators}

Stocks with a statistically significant information surplus were clustered using relevant numeric features studied (see Tables \ref{sentvariables} and \ref{finvariables}).
The aim of the clustering is to identify the cluster feature profile of stocks with a surplus exhibited at many time lags (i.e. high \texttt{POS\_LAG\_COUNT} values)
and the stocks with the highest surplus (i.e. high \texttt{MAX\_INF\_SURP\_PCT} values).
The results were robust across different values of number of clusters \textit{k} tested.
We found that moderate choices of \textit{k} all produced the noteworthy cluster profile presented in Fig. \ref{radar},
where a significant surplus is almost completely contained.
The two cluster profiles presented in Fig. \ref{radar} correspond to the cluster centroid feature profiles of both the NASDAQ and NYSE which contained the highest information surplus values.
\begin{figure}[!h]
    \centering
   \begin{subfigure}[t]{0.45\textwidth}
   \centering
     \includegraphics[width=\textwidth]{Figures/radar_nasdaq}
     \end{subfigure}
     ~
    \begin{subfigure}[t]{0.45\textwidth}
    \centering
     \includegraphics[width=\textwidth]{Figures/radar_nyse}
     \end{subfigure}
    \caption{\textbf{Radar Plot of Cluster Centroid Feature Profiles.} In both the NASDAQ and NYSE, we observe a cluster that contains the majority of leading surplus lags (\texttt{POS\_LAG\_COUNT}) and maximum surplus (\texttt{MAX\_INF\_SURP\_PCT}) for $k=\left \{ 2,3,4,5,6,7 \right \}$. The important relationship represented in this cluster is the high value of \texttt{BULL\_MINUS\_BEAR}.}
    \label{radar}
\end{figure}

Interestingly, \texttt{BULL\_MINUS\_BEAR} is maximized in the cluster,
indicating that stocks with many leading surpluses during the 10-day period (\texttt{POS\_LAG\_COUNT})
and large significant surpluses (\texttt{MAX\_INF\_SURP\_PERC}) also have a particular ratio of bullish to bearing intensity.
This suggests that in isolation, messages with a strong positive or negative sentiment are not as informative about future volatility;
rather, the combination of features is important in the context of volatility or daily risk.
Although we can speculate about the underlying factors contributing to this behavior, we note that this feature relationship seems to contradict the traditionally held assumption that \textit{negative} sentiment is more indicative of volatility.
It may be expected that a volatile security is lead by historically negative or bearish tweets because volatility is associated with risk,
and high risk is synonymous with negative sentiment; however, our clustering suggests otherwise:
the polarity between all sentiment-loaded tweets is important in predicting volatility.
Another noteworthy implication of the clustering is the absence of \texttt{VOLUME} (number of trades),
\texttt{LOG\_RETURN}, and \texttt{TOTAL\_SCANNED\_MESSAGES} (volume of messages) in the cluster profile associated with significant leading surpluses.
While those are features are commonly used in the literature, in this study they did not prove to be as relevant as the ratio of bullish to bearish messages.

\section{Conclusion}

Our results demonstrate that signals from social media can lead daily financial volatility in a large proportion of the 500 stocks
examined from the NASDAQ and NYSE. A total of 101 (40\%) stocks from the NASDAQ and a further 91 stocks (36\%) from the NYSE exhibited a
statistically significant information surplus. This was found by identifying an increase in the mutual information between social and financial time series.

While our framework of analysis is closely aligned with earlier works~\cite{citeulike:13108056}, our results are novel.
First, by identifying an information surplus in a large number of stocks,
we found that social media has the capability to lead financial markets in a much larger segment of the market than in previous works,
which only reported 12 stocks~\cite{citeulike:13108056} of significant lead-time information.

A key aim of this work was to go beyond the determination of the predictive capability of social media and attempt to determine the
configuration of features with which this occurs.
Our results revealed that stocks with the highest net sentiment polarity had the highest information surpluses.
Interestingly, in contrast to other works,  stocks with a high information surplus did not require a high volume of messages and did not have a high average log-return.
To identify this, we characterized each stock using an average representation of social and financial variables using PCA analysis, and we
applied a clustering algorithm to inspect the group of stocks that exhibited the highest maximum information surplus.

In summary, we have challenged the notion of the efficient market hypothesis by examining the effect of the continuously
evolving source of information embedded in social media and its effect in financial volatility.
Using a method rooted in information theory, we have presented results that have identified a large set
of stocks for which social media can be informative regarding financial volatility.
By clustering stocks based on the joint feature sets of social and financial variables, our research has taken an important first step in
characterizing the conditions under which this can be the case.
The results indicate that social media is most informative about financial market volatility when the ratio of bullish to bearish sentiment
is high, even when the number of messages is low.
The reported results are reproducible via an open-source library, which allows the methodology outlined in this
research paper to be re-used for sentiment analysis applied to finance.

\section*{Acknowledgment}
This work was supported by PsychSignal for providing social media sentiment analytics data. We also thank Prof. Tomaso Aste for valuable comments provided.

\bibliographystyle{unsrt}

\section{Introduction}
Recent research has dampened the consensus about the unpredictability of market prices.
As a result, predicting the stock market has garnered the attention of diverse fields and domains where different information channels have been explored,
such as news~\cite{tetlock2007giving, tetlock2008more, Tobias:2013}, search engines~\cite{PreisCurme:2014,Preis5707, citeulike:12299800, JOFI:JOFI1679}
and, more recently, social media~\cite{Bollen20111, citeulike:13108056, 1507.00784, 2016arXiv160104535S}.

Social media is of particular interest due to the high volume and velocity of activity of an ever-evolving network that is constantly providing and creating information.
It offers real-time information that can be attributed to specific people, events, markets, and securities.
This is channeled through the use of so-called \textit{cashtags} (e.g., `\$AAPL') in messages, enabling the creation of feeds associated with particular stock symbols.
The use of \textit{cashtags} has since been propagated onto Twitter, thus providing a means to direct social sentiment to the specific stock or
securities of that it references.
As a result, social media user sentiment can be harnessed as a source of information with respect to financial market activity.

Now, the opinions of traders, professional bloggers and analysts, along with laypeople's opinions,
are aggregated in a dynamic social network that can possibly explain some of the variance in market behavior.
Initial research~\cite{yahoo1} sought to quantify a relationship between social media analytics with financial market data such as daily returns.
The observed results outperformed baseline trading strategies, providing evidence that the volume of tweets can reduce uncertainty about financial returns.
However, volume-based methods disregard any possible predictive information from \textit{qualitative} aspects of the data (i.e., the actual content or content polarity).
By applying sentiment analysis techniques \cite{1507.00955} to a corpus of tweets,
a sentiment score or emotion classification can be derived to quantify this qualitative dimension~\cite{1507.00955}.
The effectiveness of this semantic approach has been examined in~\cite{Bollen20111}, where collective moods analyzed from large
collections of daily tweets were used to increase an existing financial predictor's performance to an accuracy of 87.6\%.

While a broad analysis across several financial securities might reveal that social signs are relevant for explaining financial dynamics to some extent~\cite{Bollen20111, citeulike:13108056},
little is known about confounding factors that distinguish assets with predictive social signs from assets with no extra information provided by social media.
Leveraged by a non-parametric analysis founded in information-theoretic measures, we demonstrate that social signs can be useful for most stocks from both NYSE and NASDAQ exchanges.
This alone is an interesting and very sound result, compared to current literature' however, we extend this analysis to provide, for the first time,
possible explanations of features that might be essential to distinguish predictive social signs from non-predictive ones.

\subsection{Research Questions}

\begin{itemize}
    \item \textbf{RQ1. \textit{Which stocks from the NASDAQ and NYSE exchanges exhibit a significant information surplus when using social media as a leading indicator of the stocks' future volatility?}} \\
    For each stock, lead-lag mutual information analysis is performed between financial and social media data.

    \item \textbf{RQ2. \textit{Under what configuration of social media and financial variables are social media analytics informative of future financial movements?}} \\
    We aim to determine the feature profile (using financial and social media variables) of companies that are most indicative of a statistically significant
    social media lead-time information.
\end{itemize}

\section{Methodology}

\subsection{Data}
The data correspond to the daily social media analytics and market quotes of NYSE and NASDAQ listed companies between 01-01-2012 and 01-01-2016.

Social media data were provided by PsychSignal \cite{PsychSignal}, which operates a customized
Twitter and StockTwits collection framework tracking messages containing \textit{cashtags} (e.g., \$AAPL).
State-of-the-art natural language processing algorithms were applied to relevant messages,
labeling each with a sentiment disposition (i.e., bullish or bearish) and a measure of disposition intensity.
A daily aggregate of these data was provided for each tracked stock.
Furthermore, for each stock with available social media data, we also considered the historic records of daily stock quotes.
The following data sets were utilized:
\begin{itemize}
    \item \textbf{PsychSignal Social Media Database.} Sentiment daily aggregates containing the following stock quotes attributes:
\begin{itemize}
  \item \texttt{SYMBOL}: the stock symbol (ticker) to which the sentiment data refers to;
  \item \texttt{TIMESTAMP\_UTC}: the date and time of the analyzed data in UTC format;
  \item \texttt{BULLISH\_INTENSITY}: positive sentiment polarity;
  \item \texttt{BEARISH\_INTENSITY}: negative sentiment polarity;
  \item \texttt{BULL\_SCORED\_} \texttt{MESSAGES}: positive sentiment volume, number of messages;
  \item \texttt{BEAR\_SCORED\_} \texttt{MESSAGES}: negative sentiment volume, number of messages;
  \item \texttt{TOTAL\_SCANNED\_} \texttt{MESSAGES}: total number of scanned messages.
\end{itemize}
    \item \textbf{Google Finance.} Daily Market Quotes:
    \begin{itemize}
    \item \texttt{OPEN}: daily opening price;
    \item \texttt{HIGH}: daily high price;
    \item \texttt{LOW}: daily low price;
    \item \texttt{CLOSE}: daily closing price;
    \item \texttt{VOLUME}: financial volume, number of daily trades.
    \end{itemize}
\end{itemize}

\subsubsection{Variables Analyzed}
We used a value of Daily True Range (TR) (see Equation \ref{TRequation}) as a measure of financial volatility.
A time series of volatility data for each stock was derived using financial quotes as follows:
\begin{equation}\label{TRequation}
    \begin{split}
        TR_{t} = \textup{max}[(\textup{High}_{t}-\textup{Low}_{t}), \\ (\textup{Low}_{t}-\textup{Close}_{t-1}),\\(\textup{High}_{t}-\textup{Close}_{t-1})]
    \end{split}
\end{equation}

We applied a principal component analysis (PCA) to a set of social media variables in order to obtain a time series that contains the majority of underlying information from the sentiment features considered.
\texttt{PCA\_SOCIAL\_CHANGE} is defined as the daily change in the time series obtained by applying the PCA and extracting
the first principal component from the set of social media variables considered, as defined in Table \ref{sentvariables}.
Principal component analysis is a dimensionality reduction technique that we utilized for feature extraction.
The PCA permits the reduction of numerous correlated, co-linear variables to a component (or feature set of components).
Tables \ref{sentvariables} and \ref{finvariables} list the features utilized in this work.
\input{Tables/sent_variables.txt}
\input{Tables/fin_variables.txt}

\subsection{Information Surplus}
Information surplus~\cite{citeulike:13108056} was derived from mutual information, a measure of the mutual dependence between two data sets.
Let $S$ be the variable \texttt{PCA\_SOCIAL\_CHANGE}, which is the daily change in the time series obtained by applying the PCA
and extracting the first principal component from the set of social media variables as defined in Table \ref{sentvariables} and
let $F$ be \texttt{LOG\_TR\_DIFF}, which is the log of the daily difference in the Daily True Range (see Equation \ref{TRequation}).
If the addition of series $S$ provides information about the movements of series $F$, then it is said that
a dependency or mutual information ($MI$) exists between $S$ and $F$. However, such a dependency is non-directional;
in order to determine whether $S$ \textit{leads} $F$, $S$ must provide more information on a lagged series of $F$ than the baseline $MI$ (i.e. non-lagged).
Determining whether a baseline dependency exists between the series of a social media feature $S_{l=0}$ and a
financial feature $F_{l=0}$ on the same day is the first step in identifying if $S$ leads or is predictive of $F$.
Mutual information tells us how much the information about $S$ reduces the uncertainty of $F$.
Equation (\ref{MI}) shows the formal form, where we attempt to reduce uncertainty or
increase information by taking the double integral over the log of the joint entropy for both series over each distribution.

\begin{equation}\label{MI}
MI(S;F)=\int\int f(s,\textup{f})\log\frac{f(s,\textup{f})}{f_{s}(s)f_{\textup{f}}(\textup{f})}\,ds\,df
\end{equation}

The data need to be grouped into bins to determine the mutual information between the two series. This is necessary
to calculate entropy because the probability of observing an instance $i$ in each bin $s$ and $f$ constitutes the probability distributions.
The number of bins is dependent on the size of the data; therefore, our bin sizes were typically identical
per feature (i.e., there are roughly 365 daily instances of tweets and financial data per security per year).
The bin size $k$ was calculated using Sturge's Rule (see Equation \ref{MI2}), which has been found to be more accurate than comparable methods when used to calculate entropy in the $MI$ algorithm~\cite{bin}.

\begin{equation}\label{MI2}
k = \textup{log}_{2}n + 1
\end{equation}

Mutual information was then computed at consecutive daily time lags. The information \textit{gain} at time lag $l=i$ is
calculated by finding the difference between mutual information at $l=i$ and $l=0$, where $i$ is a certain day lag
and $l=0$ corresponds to the baseline case. Information surplus is expressed as a percent of the $MI$ above what we would expect over the given time frame.
Therefore, if we achieve a surplus above the average $MI$ from $l=i$ to $l=0$, then the social media time series $S$ is said to lead $F$.

\begin{equation}\label{IS}
\textup{Information Surplus}_{l} = \frac{MI(S;F)_{l=i}-MI(S;F)_{l=0}}{MI(S;F)_{l=0}} \times 100
\end{equation}

\subsection{Validating Significant Information Surplus}
To validate whether the information surplus is statistically significant, we verified whether information is more leading than trailing, and we compared the
obtained results with those obtained via a randomly permuted time-series.

We first filtered companies whose surplus is more \textit{trailing} than it is \textit{leading}
by identifying stocks for which the daily changes in $S$ carry more information about the daily $F$ in hindsight ($l=-i$)
rather than during the same ($l=0$) or a previous day ($l=i$).
We calculated $MI$ on a forward shift (\textit{ex-post}) and on a backward shift (\textit{ex-ante}),
eliminating stocks where the \textit{ex-post} $MI$ is greater than the \textit{ex-ante} $MI$.
Where the $MI$ for a lag is less than the $MI$ for a retrospective advance of the series (i.e. $MI_{l=i}<MI_{l=-i}$),
we can assert that $MI(S;F)$ trails more trailing than it leads, and it is thus insignificant.

A random permutation of the remaining symbols was then performed 100 times. With $\alpha = 0.05$,
the stocks must outperform 95\% of the randomly permuted data to be considered to have a significant surplus.

\subsection{Clustering}
To identify the conditions under which social signs may be predictive of financial volatility,
we utilized a clustering method to determine the configuration of social media and financial variables that is
indicative of a high information surplus. Each stock was represented as a vector of social media and financial variables.
In addition, the respective scores of information surplus and the features that describe its nature
(e.g., the size of the lag it was obtained at) were included. Stocks with similar configurations of social media and financial variables were
grouped together in clusters.

Inspecting the feature profiles of the clusters (i.e., the average representation of constituent stocks' features)
allows us to identify the social signs of financial dynamics that are indicative of a high information surplus.
Clusters with a profile containing many significant lags and high overall information surplus will provide us insight into the
configuration of variables that characterizes a stock with predictable volatility.

We used k-means clustering, which is an unsupervised learning algorithm that partitions instances into \textit{k} clusters by minimizing the within-cluster sum of square error (WCSS) between instances in each set $S$ using a distance metric (see Equation \ref{kmeans}).

\begin{equation}\label{kmeans}
\textup{argmin}\, S\sum_{i=1}^{k}\sum_{x\in S_i}\left \| x - u_{i} \right \|^2
\end{equation}

\section{Descriptive Statistics}

The following descriptive statistics provide context for our results.
Figure \ref{pies} presents the sector breakdown of the top 250 stocks by market cap for both the NASDAQ and NYSE (i.e. 500 stocks in total).
In the context of this work, it is interesting to note that the NASDAQ is typically characterized as being more volatile than the NYSE exchange \cite{vol2}.

\begin{figure}[h]
    \centering
       \begin{subfigure}[t]{0.49\textwidth}
            \caption{\textbf{A) NASDAQ}}
     \includegraphics[width=\textwidth]{Figures/pie_nasdaq}
    \end{subfigure}
     \begin{subfigure}[t]{0.49\textwidth}
          \caption{\textbf{B) NYSE}}
     \includegraphics[width=\textwidth]{Figures/pie_nyse}
    \end{subfigure}
    \caption{\textbf{Sector breakdown of selected companies by exchange.} Percentage breakdown by sector highlights both that the majority of large cap stocks belong to technology and financial sectors and that the NYSE has a more balanced distribution.}
    \label{pies}
\end{figure}
\FloatBarrier

\begin{figure}[ht]
    \centering
     \begin{subfigure}[t]{0.49\textwidth}
     \caption{\textbf{A) NASDAQ}}
     \includegraphics[width=\textwidth]{Figures/tweet_vol_nasdaq}
    \end{subfigure}
     \begin{subfigure}[t]{0.49\textwidth}
     \caption{\textbf{B) NYSE}}
     \includegraphics[width=\textwidth]{Figures/tweet_vol_nyse}
    \end{subfigure}
    \caption{\textbf{Probability distribution of tweet volume.} There is a wide discrepancy between the NYSE and NASDAQ tweet volumes for the selected companies.
    The NASDAQ contains several outliers (most notably, \$APPL) that skew the distribution towards the high end.
    The NASDAQ contains a larger number and a more variable distribution of tweets, with a total of 324,239, and a standard deviation of 5,489,
    in contrast to 212,368 and 1,241, respectively, for the NYSE.}
    \label{hist}
\end{figure}
\FloatBarrier

\begin{figure}[!h]
    \centering
     \begin{subfigure}[t]{0.49\textwidth}
     \caption{\textbf{A) NASDAQ}}
     \includegraphics[width=\textwidth]{Figures/mktcap_vol_nasdaq}
    \end{subfigure}
     \begin{subfigure}[t]{0.49\textwidth}
     \caption{\textbf{B) NYSE}}
     \includegraphics[width=\textwidth]{Figures/mktcap_vol_nyse}
    \end{subfigure}
    \caption{\textbf{Market cap and Tweet Volume Positive Relationship .} The NYSE and NASDAQ selected companies exhibit a
    moderate positive correlation between market capitalization (x) and average monthly tweet volume (y).}
    \label{regression}
\end{figure}

Figures \ref{hist} and \ref{regression} compare the relationship between the volume of tweets regarding
exchange-specific securities and the size of those securities.
In summary, there is a strong right skew in tweet volume, which contributes to a moderately
positive relationship between the size of a security and its interest to investors as quantified by tweet volume.
In both exchanges, there are several notable outliers, such as \$AAPL that contains a disproportionate volume of tweets.

\begin{figure}
    \centering
    \includegraphics[width=.95\textwidth]{Figures/cor_mat_nasdaq}
    \caption{\textbf{NASDAQ correlation matrix.} Upper right (and low left) quadrant reveals weak to no correlation between social and financial features. The NYSE features exhibit comparable correlation.}
    \label{cor_plot}
\end{figure}

\begin{figure}[!h]
    \centering
    \begin{subfigure}[t]{0.45\textwidth}
    \centering
    \caption{\textbf{A)}}
     \includegraphics[width=\textwidth]{Figures/b_minus_b_PDF}
    \end{subfigure}
    ~
     \begin{subfigure}[t]{0.45\textwidth}
     \centering
     \caption{\textbf{B)}}
     \includegraphics[width=\textwidth]{Figures/log_tr_diff_PDF}
     \end{subfigure}
    \caption{\textbf{Probability Distribution Examples with Fitted Normal Curve.} The plots are exemplary of the non-normal distributions found in both our social media (A) and financial features (B) for the NASDAQ and NYSE. }
    \label{pdfplot}
\end{figure}

\iffalse
\begin{figure}[h]
    \centering
    \subfloat[]{
     \includegraphics[width=.23\textwidth]{Figures/PCA_sent_change_nowindow_ACPlot}
    }
    \subfloat[]{
     \includegraphics[width=.23\textwidth]{Figures/tr_ACPlot}
    }
    \caption{\textbf{Autocorrelation Plots.} Plots for the features used to calculate $MI$. $S_{PCA}$ (a) and $F_{TR}$ (b) in both the NASDAQ and NYSE exhibit no significant autocorrelation in these features as demonstrated by the example above for \$GE. Consequently, no rolling window is applied to the data.}
    \label{acfPlot}
\end{figure}
\fi

\iffalse
\begin{figure}[h]
    \centering
    \subfloat[NASDAQ]{
     \includegraphics[width=.23\textwidth]{Figures/regress_nasdaq}
    }
    \subfloat[NYSE]{
     \includegraphics[width=.23\textwidth]{Figures/regress_nyse}
    }
    \caption{\textbf{Primary Feature Relationship.} In both (a) and (b), a negative correlation is observed between $S_{PCA}$ (x) and $F_{TR}$ (y): the two feature series used to calculate $MI$.}
    \label{regress}
\end{figure}
\FloatBarrier
\fi

In Figures \ref{cor_plot} and \ref{pdfplot},
we present a range of plots related to the feature behavior of the data utilized for calculating the $MI$.
The correlation plot between all features (see tables \ref{sentvariables} and \ref{finvariables}) in Fig. \ref{cor_plot} reinforces the use of
PCA dimensionality reduction. As the features are correlated, it is useful to inject a compression of these variables into our MI calculation.
A correlation was found among social media features pertaining to, for instance, sentiment polarity, the volume of messages as well as with daily financial features.

The majority of features is also characterized by non-normal distributions (see Fig. \ref{pdfplot}).
For features that are highly dependent on market capitalization (e.g., mean average monthly message volume as (see Figure \ref{hist})),
log-normal behavior exists. Furthermore, NASDAQ and NYSE stocks exchanges exhibit comparable results.

\section{Results}
We used the outlined information surplus method~\cite{citeulike:13108056} and data from 01/01/2012 to 01/01/2016
to determine which stocks in the NYSE and NASDAQ exhibit, on average, a significant leading information surplus.
We then built on this method by clustering stocks and examining which configuration of variables is indicative of a high information surplus.

\subsection{Reducing Uncertainty about Volatility}

Experiments on the NASDAQ and NYSE were carried out in tandem, and they produced comparable results.
We found 101 stocks from the NASDAQ that exhibit a leading information surplus when using social media as an indicator of the daily change in True Range (our measure of volatility, see Equation \ref{TRequation}).
Of the original 250 stocks examined, 149 did not have a significant surplus, meaning that the surplus in each time lag over a 10-day
period did not exceed the expected surplus. Thus, for example, the periods in Fig. \ref{MI_sample} where the blue \textit{ex-ante}
series is below the average ex-post for the
10-day period do not contain a significant leading surplus.
All 101 stocks with a significant leading surplus, however, passed through the second validation test by performing better than 95\% of the randomly permuted data.

For the NYSE, 91 of the original 250 stocks exhibited a significant leading surplus. In addition, only one company did not pass the second validation test.

The full list of significant stocks from the NASDAQ and NYSE can be found in Appendix \ref{appendix}.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.8\textwidth]{Figures/MI_sample}
    \caption{\textbf{Sample Information Surplus.} For each lag, the surplus is calculated as the amount of $MI$ above the base case, $MI_{t=0}$.
    A significant \textit{ex-ante} surplus occurs where the surplus is greater than the average \textit{ex-post} surplus for the 10-day window.}
    \label{MI_sample}
\end{figure}

\subsection{Leading Surplus Indicators}

Stocks with a statistically significant information surplus were clustered using relevant numeric features studied (see Tables \ref{sentvariables} and \ref{finvariables}).
The aim of the clustering is to identify the cluster feature profile of stocks with a surplus exhibited at many time lags (i.e. high \texttt{POS\_LAG\_COUNT} values)
and the stocks with the highest surplus (i.e. high \texttt{MAX\_INF\_SURP\_PCT} values).
The results were robust across different values of number of clusters \textit{k} tested.
We found that moderate choices of \textit{k} all produced the noteworthy cluster profile presented in Fig. \ref{radar},
where a significant surplus is almost completely contained.
The two cluster profiles presented in Fig. \ref{radar} correspond to the cluster centroid feature profiles of both the NASDAQ and NYSE which contained the highest information surplus values.
\begin{figure}[!h]
    \centering
   \begin{subfigure}[t]{0.45\textwidth}
   \centering
     \includegraphics[width=\textwidth]{Figures/radar_nasdaq}
     \end{subfigure}
     ~
    \begin{subfigure}[t]{0.45\textwidth}
    \centering
     \includegraphics[width=\textwidth]{Figures/radar_nyse}
     \end{subfigure}
    \caption{\textbf{Radar Plot of Cluster Centroid Feature Profiles.} In both the NASDAQ and NYSE, we observe a cluster that contains the majority of leading surplus lags (\texttt{POS\_LAG\_COUNT}) and maximum surplus (\texttt{MAX\_INF\_SURP\_PCT}) for $k=\left \{ 2,3,4,5,6,7 \right \}$. The important relationship represented in this cluster is the high value of \texttt{BULL\_MINUS\_BEAR}.}
    \label{radar}
\end{figure}

Interestingly, \texttt{BULL\_MINUS\_BEAR} is maximized in the cluster,
indicating that stocks with many leading surpluses during the 10-day period (\texttt{POS\_LAG\_COUNT})
and large significant surpluses (\texttt{MAX\_INF\_SURP\_PERC}) also have a particular ratio of bullish to bearing intensity.
This suggests that in isolation, messages with a strong positive or negative sentiment are not as informative about future volatility;
rather, the combination of features is important in the context of volatility or daily risk.
Although we can speculate about the underlying factors contributing to this behavior, we note that this feature relationship seems to contradict the traditionally held assumption that \textit{negative} sentiment is more indicative of volatility.
It may be expected that a volatile security is lead by historically negative or bearish tweets because volatility is associated with risk,
and high risk is synonymous with negative sentiment; however, our clustering suggests otherwise:
the polarity between all sentiment-loaded tweets is important in predicting volatility.
Another noteworthy implication of the clustering is the absence of \texttt{VOLUME} (number of trades),
\texttt{LOG\_RETURN}, and \texttt{TOTAL\_SCANNED\_MESSAGES} (volume of messages) in the cluster profile associated with significant leading surpluses.
While those are features are commonly used in the literature, in this study they did not prove to be as relevant as the ratio of bullish to bearish messages.

\section{Conclusion}

Our results demonstrate that signals from social media can lead daily financial volatility in a large proportion of the 500 stocks
examined from the NASDAQ and NYSE. A total of 101 (40\%) stocks from the NASDAQ and a further 91 stocks (36\%) from the NYSE exhibited a
statistically significant information surplus. This was found by identifying an increase in the mutual information between social and financial time series.

While our framework of analysis is closely aligned with earlier works~\cite{citeulike:13108056}, our results are novel.
First, by identifying an information surplus in a large number of stocks,
we found that social media has the capability to lead financial markets in a much larger segment of the market than in previous works,
which only reported 12 stocks~\cite{citeulike:13108056} of significant lead-time information.

A key aim of this work was to go beyond the determination of the predictive capability of social media and attempt to determine the
configuration of features with which this occurs.
Our results revealed that stocks with the highest net sentiment polarity had the highest information surpluses.
Interestingly, in contrast to other works,  stocks with a high information surplus did not require a high volume of messages and did not have a high average log-return.
To identify this, we characterized each stock using an average representation of social and financial variables using PCA analysis, and we
applied a clustering algorithm to inspect the group of stocks that exhibited the highest maximum information surplus.

In summary, we have challenged the notion of the efficient market hypothesis by examining the effect of the continuously
evolving source of information embedded in social media and its effect in financial volatility.
Using a method rooted in information theory, we have presented results that have identified a large set
of stocks for which social media can be informative regarding financial volatility.
By clustering stocks based on the joint feature sets of social and financial variables, our research has taken an important first step in
characterizing the conditions under which this can be the case.
The results indicate that social media is most informative about financial market volatility when the ratio of bullish to bearish sentiment
is high, even when the number of messages is low.
The reported results are reproducible via an open-source library, which allows the methodology outlined in this
research paper to be re-used for sentiment analysis applied to finance.

\section*{Acknowledgment}
This work was supported by PsychSignal for providing social media sentiment analytics data. We also thank Prof. Tomaso Aste for valuable comments provided.

\bibliographystyle{unsrt}