Stance detection in twitter aims at mining user stances expressed in a tweet towards a single or multiple target entities. To tackle this problem, most of the prior studies have been explored the traditional deep learning models, e.g., LSTM and GRU. However, in compared to these traditional approaches, recently proposed densely connected Bi-LSTM and nested LSTMs architectures effectively address the vanishing-gradient and overfitting problems as well as dealing with long-term dependencies. In this paper, we propose a neural ensemble model that adopts the strengths of these two LSTM variants to learn better long-term dependencies, where each module coupled with an attention mechanism that am-plifies the contribution of important elements in the final representation. We also employ a multi-kernel convolution on top of them to extract the higher-level tweet representations. Results of extensive experiments on single and multi-target stance detection datasets show that our proposed method achieves substantial improvement over the current state-of-the-art deep learning based methods. Tweet stance detection is the task of automatically determining the stance of a tweet whether the tweet is in favor of, against, or none towards a target (Mohammad et al., 2017). We can consider it as a sub-domain of sentiment analysis. However, the goal of sentiment analysis is to classify the polarity of a tweet sentiment based on its contents, whereas identification of stance is dependent on the specific target. For example, Figure 1 depicts the stance of a tweet towards different targets. Stance detection in twitter poses unique challenges to the research community since tweets are short and informal user-generated text, which usually tend not to follow the grammatical rules. Target I: Hillary Clinton Stance I: Against Moreover, tweets contain plenty of idiosyncratic abbreviations as well as other twitter specific syntaxes such as #hashtags and emoticons. To address the challenges of stance detection in twitter, Mohammad et al. (2016) presented a tweet stance detection task that focused on a single target in SemEval-2016. Top performing systems in this task proposed several deep learning based approaches by using CNN (Wei et al., 2016), RNN (Zarrella and Marsh, 2016), and so on. Later, (Du et al., 2017) utilized the target-augmented embeddings in an attention based neural network, whereas (Zhou et al., 2017) proposed an attention mechanism at the semantic level in the bidirectional GRU-CNN structure to perform target-specific stance detection on tweets. More recently, (Dey et al., 2018) proposed a two-phase LSTM based model with attention and (Wei et al., 2018b) proposed an end-to-end neural memory model via target and tweet interactions. By considering the dependency of related targets, Sobhani et al. (2017) introduced a multi-target stance detection (MTSD) task and proposed an attentive encoder-decoder network to capture the dependencies among stance labels regarding multiple targets. Later, (Wei et al., 2018a) proposed a dynamic memory-augmented network that utilized a shared external memory to capture and store multi-targets stance indicative clues. However, most of the related work of tweet stance detection explored the traditional deep learning models in their methods. In this paper, we propose a neural ensemble method that combines the attention based state-of-the-art densely connected Bi-LSTM and nested LSTMs models with the multi-kernel convolution in a unified architecture. Experimental results on both the single and multi-target benchmark stance detection datasets demonstrate the efficacy of our method over the state-of-the-art deep learning based methods discussed above. The rest of the paper is structured as follows: In Section 2 , we introduce our proposed stance detection framework. Section 3 includes experiments and evaluations as well as the comparisons with the state-of-the-arts to show the effectiveness of our proposed method. Some concluded remarks and future directions of our work are described in Section 4 . In this section, we describe the details of our proposed neural ensemble model (PNEM) for twitter stance detection. Figure 2 depicts an overview of our proposed framework. At first, we utilize the multi-kernel convolution filters to extract higher-level feature sequences from the target appended tweet embeddings. These feature sequences are fed into the attention based densely connected Bi-LSTM and nested LSTMs to learn long-term dependencies. Final representations of these modules are concatenated and pass to the stance prediction module to determine the stance label. Next, we describe each component elaborately. Embedding Layer: Prior works already established the significance of target information for stance detection. To integrate the target information, we generate a unified word vector matrix by concatenating the vector representations of the target and tweet. The dimensionality of the matrix will be L  D , where length L is the sum of the target length and tweet length, and D denotes the word-vector dimension. We utilize a pre-trained word embedding model for obtaining the vector representation of words. Multi-Kernel Convolution: In our multi-kernel convolution, we adopt the idea proposed by (Kim, 2014) to extract the higher-level features. The input of this module is the target appended tweet matrix generated in the embedding layer. We then perform the convolution on it by using a filter. We apply multiple convolutions based on four different kernel sizes, i.e., the size of the convolution filters: 2, 3, 4, and 5. After performing convolutions, each filter generates the corresponding feature maps and a max pooling function is then applied to generate a univariate feature vector. Finally, the feature vectors generated from each kernel are concatenated to form a single high-level feature vector. Densely Connected Bi-LSTM: With the emerging trend of deep learning, LSTM based models are the most popular for sequential tasks. Recently, the densely connected structure of LSTM models gets attention among the researchers (Li et al., 2018; Wu et al., 2017) that enable the effective connection from lower to upper layers features without any loss of information on lower-layer features thus alleviate the vanishing-gradient and overfitting problems effectively. In our proposed framework, we utilize the densely connected Bi-LSTM (Ding et al., 2018) (DC-Bi-LSTM) model. A DC-Bi-LSTM model consists of multiple Bi-LSTM layers, where representation of each layer is estimated by concatenating its hidden states and all the preceding layers' hidden states. Hence, for the first Bi-LSTM layer, the input is a higher-level features sequences generated from multi-kernel convolution (MKC) and the output is { h 1 = h 11 , h 12 , ..., h 1 s } . For the second Bi-LSTM layer, the input is the concatenation of higher-level feature sequences from MKC and the output from first Bi-LSTM layer to generate the corresponding output. Rest of the layers are processed accordingly. We can define the above process as follows: h lt = [  h lt ;  h lt ] , h 0 t = MKC feature sequence ,  h lt = lstm (  h lt  1 , M l  1 t ) ,  h lt = lstm (  h lt +1 , M l  1 t ) , M l  1 t = [ h 0 t ; h 1 t ; ... ; h l  1 t ] . Therefore, from a L layer DC-Bi-LSTM model, the output is { h L = h L 1 , h L 2 , ..., h Ls } . Nested LSTMs: The nested LSTMs (NL-STMs) architecture (Moniz and Krueger, 2017) creates a temporal hierarchy of memories that achieved significant improvement over the single-layer or stacked LSTM architectures to learn longer-term dependencies. In NLSTMs, the LSTM memory cells have access to their inner memory, where they can selectively read and write relevant long-term information. While the value of the outer memory cell in the LSTM is estimated as c outert = f t (cid:12) c t  1 + i t (cid:12) g t , memory cells of the NLSTMs use the concatenation ( f t (cid:12) c t  1 , i t (cid:12) g t ) as input to an inner LSTM (or NLSTM) memory cell, and set c outert = h innert . Therefore, in compared to the LSTM and stacked LSTM, the inner memories of NLSTMs operate on longer time-scales and effectively capture the context information from the input texts. Feed-Forward Attention: Recently, the attention mechanism has been introduced in the neural network models for effectively modeling the long-term dependencies by enabling the model to learn what to attend based on the input text (Vaswani et al., 2017). In order to amplify the contribution of important elements in the final representation of both the DC-Bi-LSTM and NLSTMs module, we employ a feed-forward attention mechanism (Raffel and Ellis, 2015) to aggregate all the hidden states according to their relative importance weight. Stance Prediction and Model Training: We concatenate the final tweet representation from the attention based DC-Bi-LSTM and NLSTMs module and pass it to a fully connected softmax layer for stance detection. We consider cross-entropy as the loss function and train the model by minimizing the error, which is defined as: E ( x ( i ) , y ( i ) ) = k (cid:88) j =1 1 { y ( i ) = j } log( y  ( i ) j ) where x ( i ) is the training sample with its true label y ( i ) . y  ( i ) j is the estimated probability in [0 , 1] for each label j . 1 { condition } is an indicator which is 1 if true and 0 otherwise. We use the stochastic gradient descent (SGD) to learn the model parameter and adopt the Adam optimizer (Kingma and Ba, 2014). In the following, we describe the set of parameters that we have used in our proposed neural ensemble model (PNEM) during experiments. We used the 300-dimensional fastText embedding model pre-trained on Wikipedia with skip-gram (Bojanowski et al., 2017) to initialize the word embeddings in the embedding layer. For the multi-kernel convolution, we employed 4 kernel sizes (2,3,4,5), and the number of filters was set to 600. In our model, DC-Bi-LSTM module contains 5 layers and NLSTMs module contains 2 layers. We trained all models for max 45 epochs with a batch size of 32 and an initial learning rate 0.001 by Adam optimizer. L 2 regularization with a factor of 0.01 was applied to the weights in the softmax layer. In this paper, we reported the results based on these settings. Unless otherwise stated, default settings were used for the other parameters. To preprocess the data, we removed the stop words based on NLTK's standard stoplist, special characters removal, and performed hashtag segmentation according to (Baziotis et al., 2017). Dataset and Setup: To validate the effectiveness of our proposed method for the single target stance detection, we made use of a widely popular benchmark twitter dataset used in the SemEval-2016 Task 6-A (Mohammad et al., 2016). The training set consists of 2914 tweets and the test set consists of 1249 tweets relevant to 5 targets. Each tweet was annotated as Favor , Against or None towards the specific target. Following the SemEval-2016 Task 6-A benchmark, we employed the macro-average of F1-score for the Favor and Against stance classes as the evaluation measure. To estimate it, we used the evaluation script provided by the organizer. Results and Analysis: We divided the whole dataset across targets and trained the model accordingly. We used 5% of the training samples as the validation set. The summarized experimental results of our proposed neural ensemble model (PNEM) on single target stance detection are presented in Table 1. At first, we report the results based on the baseline, which is the combination of CNN and LSTM (Zhou et al., 2015) and obtained competitive performances on several text classification tasks. Next, we report the results of our proposed PNEM model. It showed that our PNEM method outperformed the baseline by a large margin. In order to estimate the effect of each component of our model, we performed the component ablation study on our proposed model. In this regard, we removed one component each time and repeated the experiment. From the results, it can be observed that when removing target embedding, multi-kernel convolution (MKC), attention mechanism (ATT), NLSTMs with corresponding attention (NLSTMs+ATT), and DC-Bi-LSTM with corresponding attention (DC-Bi-LSTM+ATT), the results decreased by 0.99%, 4.98%, 1.68%, 4.14%, and 1.11%, respectively. Thus deduced the importance of each of the component in our model. Moreover, the comparative performance of our proposed model, PNEM with the state-of-the-art methods are presented in Table 2. It showed that our model gained 3.13% and 4.29% improvement while compared with the SemEval-2016 baseline (SVM-ngrams) and the best performing system MITRE, respectively. Furthermore, in comparison with the related deep learning based methods, PNEM achieved at least 1.07% and at best 3.44% improvement. Overall, our proposed PNEM method greatly surpassed the previous works. The results show that attentive neural ensemble model could benefit stance detection. Dataset and Setup: In order to assess our method for the multi-target stance detection, we made use of a benchmark twitter dataset (Sobhani et al., 2017), where each tweet is annotated with two stance labels towards two targets of a pair. The tweets in this dataset are related to the 2016 US election. Overall, the dataset contains 4455 tweets for the three target-pairs including Hillary Clinton Bernie Sanders, Hillary Clinton Donald Trump, and Ted Cruz Donald Trump. The train, development, and test set contains 3119, 446, and 890 tweets, respectively. As the evaluation measure, we used a similar kind of approach reported in the dataset paper (Sobhani et al., 2017). Following this, the F avg of each target is estimated according to the SemEval-2016 Task-A benchmark (Mohammad et al., 2016). Then, the F avg of the two targets of a target-pair is averaged to estimate the average score of a target-pair. Finally, the average scores of all the target-pairs are averaged to estimate the overall score. Results and Analysis: We divided the whole dataset based on each target-pairs. The dataset is then separated into two parts for each target in the target-pair. We trained and evaluated the model for each target in the target-pair and combined the results to estimate the overall performance. We used a similar kind of baseline (CNN+LSTM) for comparison that we used in the single target stance detection as well as compared with the state-of-the-art deep learning based methods. The comparative results of our proposed PNEM model on multi-target stance dataset are presented in Table 3. It showed that our method surpassed the baseline by a large margin and gained 3.91% and 1.99% improvement over the state-of-the-art methods Seq2Seq and DMAN, respectively. In this paper, we proposed an attention based neural ensemble model for the target-specific tweet stance detection. The main contribution of our unified model is to learn the contextual information effectively which in turns improved the stance detection performance and outperformed the state-of-the-art deep learning based methods for both the single and multi-target stance detection benchmark datasets. In the future, we have a plan to leverage external knowledge and generalize our model for target-independent stance detection in the same domain. The part of this research is supported by MEXT KAKENHI, Grant-in-Aid for Scientific Research (B), Grant Number 17H01746 and by grants from the KDDI Foundation.
abstain abstain abstain objective method objective abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective objective objective abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain other abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain result objective objective abstain other