The stance detection task aims at detecting the stance of a tweet or a text for a target. These targets can be named entities or free-form sentences (claims). Though the task involves reasoning of the tweet with respect to a target, we find that it is possible to achieve high accuracy on several publicly available Twitter stance detection datasets without looking at the target sentence. Specifically, a simple tweet classification model achieved human-level performance on the WTWT dataset and more than two-third accuracy on various other datasets. We investigate the existence of biases in such datasets to find the potential spurious correlations of sentiment-stance relations and lexical choice associated with the stance category. Furthermore, we propose a new large dataset free of such biases and demonstrate its aptness on the existing stance detection systems. Our empirical findings show much scope for research on the stance detection task and proposes several considerations for creating future stance detection datasets. 1 1 Introduction Stance detection is a vital sub-task for fake news detection (Pomerleau and Rao, 2017), automated fact checking (Vlachos and Riedel, 2014; Ferreira and Vlachos, 2016), social media analysis (Zhang et al., 2017), analyzing online debates (Bar-Haim et al., 2017) and rumour verification (Derczynski et al., 2017; Gorrell et al., 2019). Furthermore, it is also an essential measure for progress in Natural Language Understanding, especially in the noisy-text domain. Over the recent years, several stance detection datasets have been proposed. These datasets, in turn, facilitated progress in stance detection research, with some systems achieving up to 93.7% accuracy (Dulhanty et al., 2019). However, most 1 Code: https://github.com/Ayushk4/bias-stance Dataset: https://github.com/Ayushk4/stance-dataset Analysts: Aetna-Humana Deal still Probable, Anthem-Cigna Unlikely @InsuranceNewsNet <URL> 11:03 AM   Twitter for iPhone [Aetna-Humana] Support [Anthem-Cigna] Refute Figure 1: An example tweet from WTWT dataset with different targets. of these state-of-the-art systems are complex deep neural networks, making them difficult to interpret. Lack of explainability raises concern since previous works (Gururangan et al., 2018; Goyal et al., 2017; Cirik et al., 2018; Geva et al., 2019) on other tasks demonstrated that superficial dataset biases could result in inflated test-set performance. With this motivation, we carry out the first study analyzing several publicly available Twitter stance detection datasets. Our experiments reveal rampant biases in datasets through which even target-oblivious models can achieve impressive performance. Various existing works have hinted at the presence of such dataset biases. For example, TAN model (Du et al., 2017) is a very competitive stance detection model.However, Ghosh et al. (2019) recently proved that TAN does not take advantage of target information at all. In RumourEval-2017 (Derczynski et al., 2017), models delivered up to 0.74 accuracy without any knowledge of the target, being only short of 0.004 from the best model considering the context. Similarly, in RumourEval2019 (Gorrell et al., 2019), the runner-up model (Fajcik et al., 2019) observed a 0.43 decrease in accuracy by considering the target information. Schiller et al. (2020) discovered that stance detection models are prone to adversarial attacks of paraphrasing, spelling error and negation similar to other NLP tasks (Ribeiro et al., 2020). However, Target Tweets Dataset size Number Domain Type # Stance Unique Scrapped DT/T WTWT 51284 5 Finance (M&A) fixed 4 50210 45865 2% SE16 4162 5 Various fixed 3 4162 0% M-T 4455 3 Political fixed (pairs) 3 4413 2688 0.9% RE17 5568 -Rumour-claims free-form 4 556k 0% RE19 8574 -Rumour-claims free-form 4 8574 0% Encryption 2999 1 Encryption-debate fixed 3 2522 1634 0% Table 1: Statistics of the Twitter stance detection datasets considered. Target plays a crucial role in deciding stance. Consider the example in Figure 1. Here, the tweet stance varies for the two targets. The existing datasets have very few examples with different target labels. Models can pick up on pseudo signals in the tweet content and shortcut the task without looking at the targets. These signals or biases occur due to inherent biases in our language and human nature. For example, certain lexical choices can correlate with their respective stance classes. Upon discovering and studying such correlations, we augment the WTWT dataset addressing these issues and re-evaluate the stance detection systems. We make the following contributions. We empirically demonstrate biases across a variety of Twitter stance detection datasets and carry out a detailed analysis of these datasets. Consequently, we propose a new large scale dataset free of such spurious cues and re-evaluate the stance detection systems to show the usefulness of this dataset. We first discuss the datasets considered (Section 2.1), followed by our experiments (Section 2.2) and analysis (Section 2.3). We consider a wide variety of publicly available Twitter stance detection datasets including cross-target, multi-target, rumour-claim variants of stance detection. These datasets have a diverse set of targets ranging from free-form sentences to fixed target entities. Over the past few years, several more variants of this task have been proposed, such as in non-English language (Darwish et al., 2017; Kk and Can, 2018; Lai et al., 2018) and multi-lingual settings (Zotova et al., 2020; Vamvas and Sen-nrich, 2020), different learning paradigms of unsupervised (Darwish et al., 2019) semi-supervised (Mohammad et al., 2016b), zero-shot (Allaway and McKeown, 2020) and non-Twitter tasks of debate-argument stance (Bar-Haim et al., 2017) and headline-body stance detection (Pomerleau and Rao, 2017). Here, however, we only study the English Twitter stance detection tasks in fully supervised learning settings. Specifically, we consider 6 datasets WTWT (Conforti et al., 2020), SE16 (task-A) (Mohammad et al., 2016b,a) M-T (Sobhani et al., 2017), RE17 (Derczynski et al., 2017), RE19 (Gor-rell et al., 2019) and Encryption (Addawood et al., 2017) with their statistics mentioned in Table 7. This table also reports the percentage of tweets in the entire dataset labelled for different targets (DT) given by DT/T in the last column. We can see that these datasets have very few tweets annotated for different targets. The M-T dataset's targets are a pair of politicians, and for each of its tweet-targets, the label is a pair of stances. We formulate detecting these two stance-pair as separate tasks for the experiments in the following section. Method: Given a tuple ( tweet, target, stance ) , a target-oblivious classifier f ( tweet )  stance is trained in a supervised setting. It is expected that such a classifier would generalize poorly for an unbiased dataset. We set this target-oblivious classifier as the standard Bert classifier (Devlin et al., 2019) pre-trained on Tweets (Nguyen et al., 2020). It receives the input  [ CLS ] tweet [ SEP ] ". Additionally, we train a strong target-aware Bert classifier model for stance detection (Ghosh et al., 2019). This model takes input  [ CLS ] tweet [ SEP ] target [ SEP ] ". We use PyTorch (Paszke et al., 2019), HuggingFace (Wolf et al., 2019), Wandb (Biewald, 2020) and Scikit-Learn (Pedregosa et al., 2011) for our experiments. We use Adam optimizer (Kingma and Ba, 2014). We elaborate the full experimental settings in the appendix A. F 1 Macro across healthcare merger operations Entertainment Models CVS_AET CI_ESRX ANTM_CI AET_HUM avgF 1 avg w F 1 F 1 Macro Bert (no-target) 0.673 0.703 0.745 0.759 0.720 0.720 0.347 Human Upperbound 0.753 0.712 0.744 0.737 0.736 0.743 N/A Bert (with target) 0.668 0.709 0.746 0.756 0.720 0.719 0.433 Random guessing 0.222 0.237 0.231 0.236 0.230 0.232 0.201 Majority guessing 0.162 0.139 0.155 0.134 0.151 0.148 0.161 Table 2: Results on WTWT dataset (Conforti et al., 2020). Results and Discussion: The WTWT dataset is a cross-target dataset containing four in-domain (healthcare) and one out-of-domain (entertainment) target. For the in-domain evaluation, training is done on three health mergers while testing is done on the fourth unseen target. For out of domain, training is on all four health mergers and testing on the entertainment domain. Table 2 shows the performance of target-oblivious Bert, target-aware Bert and the human upper-bound. The human expert upper-bound values were taken from the WTWT dataset. We observe that the target-oblivious model consistently performs very close to the target-aware model for all the targets. Both these models achieve near-human performance overall on in-domain targets. The target-oblivious Bert surpasses human upper-bound for two mergers individually. Such a feat is alarming, especially because cross-target stance is a more challenging variant (Kk and Can, 2020; Wang et al., 2020) of the task. Results on the other datasets are shown in Table 3. We compare these results with random guessing, predicting majority class and the target-aware Bert. Additionally, RE17, R19, and Encryption datasets are heavily skewed datasets, so Macro-F1 is the proposed metric (Gorrell et al., 2019). The target-oblivious Bert delivers more than two-third classification accuracy consistently across all these datasets. This model achieves impressive performance for all metrics in SE16 and M-T datasets, while also performing significantly above majority class for datasets with skewed distributions on the Macro-F1 metric. The performance delivered by target oblivious Bert is also very close to the target-aware Bert model on every metric. These surprising numbers across all the datasets indicates the presence of spurious cues that encourages the models to bypass the need for looking at the target. After the finding from our previous section, we sought to discover the form in which spurious cues exists and use those findings to create a new dataset. We mainly consider the largest and most recent dataset, WTWT for analysis. We first discuss target-independent lexical choices associated with stance, followed by target-independent sentiment-stance correlations. between tweet and stance following the exact same procedure as (Gururangan et al., 2018) after removing stopwords. Table 4 shows that top 5 stance-wise words along with the fraction of tweets containing those words. We observer that certain groups of target-independent lexicons are highly correlated with stances in some cases occurring in more 29% of the tweet. For Support and Refute classes respectively, we find the co-occurrence of indicative words for the status of merger, such as approves' or blocks'. The Comments relating to these health companies' mergers often talk about its impact, leading to the choice of lexicons containing words like healthcare' and mean' with this stance. Similarly, Unrelated tweets often talk about things related to the companies but unrelated to the merger operation such as stocks' or bids'. Sentiment-stance correlation: Stance detection differs from the sentiment analysis task (Moham-mad et al., 2016b). However, we observe a strong correlation of sentiment with stance. Formally, we obtain a sentiment score between 0 (negative) and 1 (positive) for each tweet using XLNet model (Yang et al., 2019) trained on SST (Socher et al., 2013; Pang and Lee, 2005) and Imdb (Maas et al., 2011). The average sentiment scores of these tweets across Support, Refute, Comment and Unrelated stances were found to be 0.237, 0.657, 0.492 and 0.485 respectively, while their variance were 0.087, 0.056, 0.110, 0.108. The tweets with Support and Refute stance have strong negative or positive sentiment on average while for the other two is it neutral on average but having a high variance. These serve as strong evidences for stance-sentiment correlations. such cues in the remaining datasets, varying with their domains. For example, RE19 has a question mark in more than 75% query' stance tweets, while it is present only in 11% of the entire remaining dataset. Similarly 75% of tweets with deny' stance have highly negative sentiment of less than 0.1 score. In SE16 dataset, had 91.4% of tweets without any opinion 2 had None' stance despite the stance detection task being different from opinion mining task (Mohammad et al., 2016b). With the understanding from the previous section, we propose a new stance detection dataset on which target-unaware models will not perform well. We use the following reasoning for creating the new dataset. If a tweet in the dataset has different stances depending on different targets, then simple tweet classification models will not be able to perform well. Thus we attempt to increase DT/T ratio from Table 7. Formally, we take the WTWT dataset, which is the largest dataset of its kind, with high-quality experts labels of 0.88 Cohen (Co-hen, 1960), and generate new (tweet, target, stance) triplets in three ways. First , we attempt to remove the sentiment-stance correlation by making the stance-wise average sentiment neutral. The WTWT dataset has 5 targets, one target for each merger. We introduce 5 new additional targets which are negations of the original ones. Formally, if the tweet has a Support (Refute) stance to the target CV S _ AET , then its stance to the negated target NEG _ CV S _ AET will be inverted to Refute (Support). This is done only 2 Tweets have gold labels for opinion-class in the dataset. for the two stance classes with non-neutral average sentiment score. Introducing such negated targets reduces their sentiment to near neutral. Second , we remove lexicon-stance correlations by creating multiple targets with different stances for each tweet. Formally, for each tweet t with only one labelled target tgt , if the tweet-target pair ( t, tgt ) has the stance (cid:54) = Unrelated', then pick a target tgt (cid:48) where tgt (cid:48) (cid:54) = tgt and add the tuple ( t, tgt (cid:48) , Unrelated ) to the dataset. Due to WTWT data collection and annotation procedure, this will not generate any wrong labels. This augmentation reduces the lexicon-stance correlations, by having similar sets of lexicons introduced for different stances. Hence, it guarantees target-oblivious shortcuts to result in poor performance. Last , we balance the target-wise class-distributions. For the tuples with Comment' and Unrelated' stances, we create a new tuple with inverted target (same as the first step) for 50% and 75% such examples randomly. The resulting dataset contains 111596 tweet-target pairs each belonging to a stance class. Each merger has at least 10000 data points. The class distribution is also somewhat balanced with more than 10k examples for the least occurring class. Among the tweet-target pairs, the pairs classified as Support, Refute, Comment and Unrelated are distributed in the ratio 1:1:3:5 approximately, having a similar distribution to the WTWT dataset. We propose a similar cross-target evaluation setting for tWTWT as WTWT. For the in domain (health) mergers, we train on three health merger (total six targets including negated target for each merger) and test on the fourth health merger. For the out-of-domain evaluation, we train on the eight targets corresponding to the 4 health mergers and test on the two targets for entertainment merger. target), target-oblivious Bert from 2.2, along with the two strongest baselines from the WTWT paper SiamNet (Santosh et al., 2019) and TAN (Du et al., 2017). For SiamNet and TAN models, we replace the Glove (Pennington et al., 2014) and LSTM (Hochreiter and Schmidhuber, 1997) features with better features from Bert. Table 5 shows the performance of these models. Bert (no-target) gives very low performance, showing that target oblivious models perform poorly on this dataset. Similarly, TAN which has been proven to not at take advantage of the target information (Ghosh et al., 2019) also performs very poorly on the dataset. The target aware Bert offers a competitive performance still being only at 0.51 F1 score. SiamNet follows next at 0.31 F1 score. Both these models have their performance reduced significantly from WTWT dataset. In this paper we demonstrated the presence of biases across several Twitter stance detection datasets, which aid simple tweet classifiers to achieve impressive performance. We carried out an investigation for presence of bias for the WTWT dataset and found correlations of stance-class with sentiment and lexical choice. Consequently, we proposed a new bias-free stance detection dataset tWTWT, the largest of its kind. Evaluation of our baselines on this new dataset demonstrates scope for future research on stance detection. The observations are also crucial for the creation of new stance detection datasets. Our future work includes analysing multilingual datasets and exploring explainable target aware stance detection models. We would like to thank the anonymous reviewers for their valuable feedback and suggestions. We also thank the Computer Science and Engineering Department at the IIT Kharagpur for providing us with the compute facilities for our research.
abstain abstain result abstain objective objective objective other abstain abstain abstain other abstain abstain objective result abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain objective objective objective objective abstain method abstain other method method abstain abstain abstain method abstain abstain method method method method abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain objective result objective objective abstain objective other other