Natural disasters (e.g., hurricanes) affect millions of people each year, causing widespread destruction in their wake. People have recently taken to social media websites (e.g., Twitter) to share their sentiments and feelings with the larger community. Consequently, these platforms have become instrumental in understanding and perceiving emotions at scale. In this paper, we introduce HURRICANEEMO , an emotion dataset of 15,000 English tweets spanning three hurricanes: Harvey, Irma, and Maria. We present a comprehensive study of fine-grained emotions and propose classification tasks to discriminate between coarse-grained emotion groups. Our best BERT (De-vlin et al., 2019) model, even after task-guided pre-training which leverages unlabeled Twitter data, achieves only 68% accuracy (averaged across all groups). HURRICANEEMO serves not only as a challenging benchmark for models but also as a valuable resource for analyzing emotions in disaster-centric domains. Natural disasters cause thousands of deaths and displace hundreds of millions each year (Ritchie and Roser, 2020). These catastrophic events not only induce material destruction but also stir an integral part of being human: our emotions. Disasters adversely affect individuals' mental states (Fritz and Marks, 1954; Kinston and Rosser, 1974), and therefore it is no surprise that many take to social media (e.g., Twitter) to share their feelings. Social media websites, as a result, have become an essential platform for understanding the expression and perception of emotions at a significantly larger scale (Mohammad, 2012; Wang et al., 2012; Mohammad and Kiritchenko, 2015; Volkova and Bachrach, 2016; Abdul-Mageed and Ungar, 2017), with far reaching potential influences from academic research to public policy (Dennis et al., 2006; Fritze et al., 2008; Fraustino et al., 2012). While natural language processing methods have been effective for emotion detection (Strapparava and Mihalcea, 2007), existing resources struggle in disaster-centric domains, in part due to distributional shifts. Emotion detection in natural disasters (e.g., hurricanes) requires implicit reasoning not available as surface-level lexical information. For example, in of course, [we] 1 still have the [storm surge] 2 coming, given the context, we can reasonably infer discontent towards the storm surge despite the absence of polarizing words. Therefore, distantly supervised techniques largely based on lexical units (Mohammad and Turney, 2013; Abdul-Mageed and Ungar, 2017) fail to capture this type of deeper semantic phenomena. Our paper presents a comprehensive investigation into perceived emotions in hurricane disasters. To this end, we introduce HURRICANEEMO , a dataset of 15,000 disaster-related tweets (in English) streamed during Hurricanes Harvey, Irma, and Maria, which were devastating tropical storms occurring in the 2017 Atlantic hurricane season (Belles, 2017). Our samples are annotated with fine-grained emotions derived from the Plutchik Wheel of Emotions (Plutchik, 2001), a well-defined ontology of emotion classes commonly used in computational social science (Abdul-Mageed and Ungar, 2017). 1 To measure inter-annotator agreement on fine-grained emotion labels, we conceptualize the P lutchik E motion A greement (PEA) metric (3). PEA is intuitively grounded; our human evaluation shows workers agree with PEA's rankings 88% of the time. Furthermore, we perform insightful analyses on implicit and explicit emotions in hurricane tweets (4). Quite surpris-1 Specifically, we use Plutchik-8 and Plutchik-24 emotions. We refer readers to Plutchik (2001) for an in-depth discussion on their conception. ingly, we find consistencies in Plutchik-24 emotion distributions across Hurricanes Harvey, Irma, and Maria. HURRICANEEMO also serves as a challenging new benchmark for large-scale, pre-trained language models. We establish baselines for a coarser Plutchik-8 emotion detection task using BERT (De-vlin et al., 2019) and RoBERTa (Liu et al., 2019) (5). Our experiments reveal: (1) BERT only achieves 64% (averaged) accuracy; and (2) using better pre-trained models (e.g., RoBERTa) does not help, which is a strikingly different trend than most leaderboards (Wang et al., 2018). To better understand their pitfalls, in particular BERT, we conduct a comprehensive error analysis of 200 incorrectly predicted samples. In addition, we incorporate stronger inductive biases into BERT via pretraining on related tasks, which culminates in (av-eraged, absolute) +4% accuracy (6). Finally, we propose unsupervised domain adaptation to bridge the domain gap between existing large-scale emotion datasets (e.g., EMONET (Abdul-Mageed and Ungar, 2017)) and HURRICANEEMO (7). Our code and datasets are made publicly available. 2 2 Related Work Emotion detection has been extensively studied in news headlines (Strapparava and Mihalcea, 2007; Katz et al., 2007), blog posts (Aman and Szpakow-icz, 2007), health-related posts (Khanpour and Caragea, 2018), and song lyrics (Strapparava et al., 2012), but only recently, in social media websites (e.g., Twitter, Facebook) (Mohammad, 2012; Wang et al., 2012; Mohammad and Kiritchenko, 2015; Volkova and Bachrach, 2016; Abdul-Mageed and Ungar, 2017). However, emotion detection in disaster-centric domains, despite its practical importance, is limited. Schulz et al. (2013) (single-handedly) annotate 2,200 Hurricane Sandy tweets using Ekman-6 emotions (Ekman, 1992). In contrast, we introduce 15,000 annotated tweets from multiple hurricanes with (much more fine-grained) Plutchik-24 emotions. Unlike Abdul-Mageed and Ungar (2017), we focus on readers' perceived emotions rather than writers' intended emotions. Furthermore, in disaster-centric domains, the lack of labeled data required to train reliable models precludes the use of supervised learning techniques. Several works propose to use labeled data 2 https://github.com/shreydesai/ hurricane from prior (source) disasters to learn classifiers for new (target) disasters (Verma et al., 2011; Nguyen et al., 2017; Imran et al., 2013, 2016; Caragea et al., 2016). However, due to the unique nature of each disaster (e.g., type, geographical location, season, cultural differences among the affected pop-ulation), the source disaster may not accurately re-flect the characteristics of the target disaster (Palen and Anderson, 2016; Imran et al., 2015). Domain adaptation techniques address these challenges by efficiently using large amounts of unlabeled target domain data, consequently outperforming the aforementioned supervised techniques (Alam et al., 2018; Li et al., 2017). Our work contributes to disaster-centric emotion detection in three ways by: (1) introducing a dataset large enough to train supervised classifiers; (2) exploring various forms of pre-training to instill strong inductive biases; and (3) establishing domain adaptation baselines by leveraging emotive samples obtainable via distant supervision. In this section, we present HURRICANEEMO , an annotated dataset of 15,000 English tweets from Hurricanes Harvey, Irma, and Maria. We detail each component, including the initial preprocessing (3.1), annotation procedures (3.2), and the formulation and calculation of inter-annotator agreement (3.3). Ray Chowdhury et al. (2019) release a repository of large-scale Twitter datasets consisting of tweets streamed during the Harvey, Irma, and Maria hurricanes, which we will refer to as HURRICANEEXT (i.e., extended). We use their tweets as a starting point for the construction of our dataset. We perform two types of preprocessing. First, we replace usernames and links with <USER> and <URL> , respectively, then eliminate duplicate tweets. Second, we use filtering techniques to ensure the resulting tweets contain emotive content. We assume a lexical prior over emotion tweets, that is, requiring that an emotive tweet consist of at least one word derived from EMOLEX (Mo-hammad and Turney, 2013). EMOLEX consists of 14,182 crowdsourced words associated with several emotion categories. Critically, these words appear in emotional contexts, but are not necessarily emotion words themselves. For example, payback is related to the emotion anger, but is also used extensively in finance. Significant past work (Bravo-Marquez et al., 2014; Majumder et al., 2017; Giat-soglou et al., 2017) has used this lexicon to bootstrap their emotion datasets, since the alternatives are (1) using unlabeled tweets as-is or (2) using a model to classify emotional tweets. Initially, we started with (1) and did no emotion-related preprocessing. However, the dataset contained many spurious tweets, such as snippets of news articles, that had little to do with emotions. The level of noise rendered the data prohibitively costly to annotate. For (2), there is simply no such large-scale data to train on, and existing resources like EMONET manifest an even stronger prior where tweets are only included if they explicitly contain an emotion hashtag (e.g., #sad , #angry , #happy ). We randomly sample 5,000 tweets each for annotation from the filtered datasets for Harvey, Irma, and Maria; in total, this yields 15,000 annotations. We request workers on Amazon Mechanical Turk to label tweets with a list of Plutchik-24 emotions. Furthermore, to enable fine-grained emotion analysis, we do not crowdsource Plutchik-8 emotions directly. We require that workers reside in the US and have completed 500+ HITs with an acceptance rate  95%. Each HIT is completed by 5 workers. In this section, we elaborate on our PEA metric for computing inter-annotator agreement with fine-grained emotion labels. Challenges. Fine-grained emotion annotation presents several challenges for evaluating inter-annotator agreement. First, because a tweet can convey multiple emotions, we allow workers to select more than one Plutchik-24 emotion. This implies an agreement metric must support scoring sets of categorical values. Passonneau (2004) use set distance metrics for capturing agreement between coreference cluster annotations. Similarly, Wood et al. (2018) incorporate Jaccard's similarity in Krippendorff's alpha. However, these methods would penalize fine-grained emotions equally, which is not ideal. For the Plutchik wheel, the proximity of any two emotions represents their relatedness. For example, TRUST and ADMIRATION belong to the same emotion group while LOATHING and ADMIRATION are orthogonal to each other. PEA Scores. We introduce the P lutchik E motion A greementhereafter referred to as PEAto address these challenges. We superimpose a unit circle onto the Plutchik wheel, representing each Plutchik-8 emotion as a polar coordinate (e.g., DISAPPROVAL = (  22 ,  2 2 ) ). Intuitively, the an-gles between Plutchik-8 emotions represent how similar or dissimilar they are. If two Plutchik-24 annotations belong to the same Plutchik-8 group, we do not penalize them (e.g., JOY and ECSTASY incur no penalty). Otherwise, we enforce a linear penalty based on how radially separate the annotations are (e.g., ECSTASY and GRIEF incur the highest penalty). Higher PEA scores imply more agreement. Example. Figure 1 visualizes our metric. In this example, two annotators select emotions with radians 3  2 and  4 , respectively. The | f ( e ( i ) x )  f ( e ( j ) y ) | term evaluates to 5  4 . Then, it is normalized using 1  , yielding 54 = 1 . 25 . Finally, we subtract to obtain the agreement score: | 1  1 . 25 | = 0 . 25 . Intuitively, this makes sense as the decisions are only slightly better than being in complete disagreement (i.e., orthogonal). Formulation. For clarity, we introduce notation. Let w x and w y denote workers with (categorical) annotation sets { e ( i ) x } ni =1 and { e ( j ) y } mj =1 , respectively. The pairwise agreement d ( w x , w y ) between the workers is computed as: 1 n n (cid:88) i =1 max j (cid:0) | 1  1  | f ( e ( i ) x )  f ( e ( j ) y ) || (cid:1) Vocabulary Features (%) Hurricane Orig. where 1  is a normalizing constant and f :   R is a map from Plutchik-8 emotions to radians. Given a collection of workers that annotated a tweet, we obtain per-worker PEA scores by averaging over all possible pairwise agreements. For example, if workers w 1  3 annotated the same tweet, PEA( w 1 ) = 12 ( d ( w 1 , w 2 ) + d ( w 1 , w 3 )) . For quality control, we filter annotations from workers with PEA  0.55. This threshold is determined through manual inspection of 50 workers and their annotations. The (averaged, per-worker) PEA scores for each hurricane are: Harvey (65.7), Maria (67.3), and Irma (70.3). 3 Human Evaluation. We perform a human evaluation with our proposed metric, which is absent in previous work for measuring inter-annotator agreement for emotion annotations (Wood et al., 2018; hman et al., 2018). Crowdsourced workers are asked to determine the agreement between two annotation pairs constructed from three annotators, that is, A: ( e 1 , e 2 ) and B: ( e 1 , e 3 ) . They choose between three options: (1) A has higher agreement than B; (2) A and B have (roughly) the same agreement; and (3) B has higher agreement than A. 88.2% of the worker rankings match with PEA's rankings, pointing towards strong human agreement. The workers themselves in this study also show good agreement according to Krippendorff's alpha (  = 74.0) (Artstein and Poesio, 2008). 4 4 Qualitative Analysis 4.1 Dataset Overview Table 1 presents several statistics of HURRICANEEMO . We make three observations. First, the 3 A reasonable interpretation of PEA scores may be as follows: 025 (no agreement), 2550 (poor agreement), 5075 (moderate agreement), 75100 (high agreement). 4 See Appendix B for details on our procedures. vocabularies across all datasets are large considering there are only 5,000 tweets per hurricane. The vocabularies do decrease by about 30% after preprocessing, although the resulting sizes still suggest users use a myriad of words to express their emotions. Second, only about 50% of Harvey tweets and 40% of Irma/Maria tweets contain hashtags. Hashtags are a unique marker of Twitter discourse (Ritter et al., 2011), but in our dataset specifically, hashtags are used to tag particular entities, spread disaster-relief awareness, and create trending content. This phenomena alone makes our tweets different from those collected through distant supervision (Abdul-Mageed and Ungar, 2017). Third, roughly 80-85% of tweets contain links to third-party content. Users commonly use links to share news articles, resources for humanitarian aid, and other miscellaneous multimedia. Table 2 shows three samples from HURRICANEEMO . Unlike EMONET (Abdul-Mageed and Ungar, 2017), our dataset does not have the strong assumption that only one emotion can be expressed in a tweet. For example, the first tweet lexically points towards the expression of more than one emotion. The predicate helped us implies the user admires Mexico for providing aid, and the exclamation mark is indicative of JOY . In addition, our samples contain a mix of implicit and explicit emotions, which lexical information alone cannot resolve. In the third tweet, there are no particular words that point towards ANGER and ANNOYANCE , but we can infer the user is upset that the media is not prioritizing Hurricane Maria. Finally, our emotion prediction tasks cannot be solved by simply retrofitting pre-trained word embeddings (Mikolov et al., 2013; Pennington et al., 2014) or contextualized representations (Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019), which we also empirically show in our experiments (5). These methods work best for explicit emotion detection as they largely overfit to sparse lex-Plutchik-8 Plutchik-24 Emotion Abbrv. ical features. Rather, in order to capture implicit emotions, models must carry an inductive bias that appropriately reasons over the context (e.g., what event(s) occurred?) and semantic roles (e.g., what happened to whom?) while balancing the aforementioned features. We begin to analyze the fine-grained emotions present in our datasets. We ask the following questions: What is the general distribution of emotions? Are certain emotion groups highlighted more than others? How does the distribution change across hurricanes? Figure 2 shows Plutchik-24 emotion distributions for Hurricanes Harvey, Irma, and Maria. From these plots, a couple of trends emerge. First, the Plutchik-24 emotion counts are within the ballpark of each other with the notable exceptions of ADMIRATION and FEAR . This suggests that, on average, hurricane disasters evoke a similar spread of implicit and explicit emotions among most emotion categories. Second, users tend to post more optimistic content during hurricane disasters. We hy-Figure 2: Per-hurricane emotion counts where each box's Plutchik-8 emotion is broken down into its respective Plutchik-24 emotions. Plutchik-24 emotions are abbreviated using the codes in Table 3. pothesize that users use Twitter as a social platform to spread awareness of the hurricanes themselves or post-disaster relief efforts, commonly using hashtags like #prayfortexas , #floridaevacuation , and #donationdrive . It is encouraging to see that although users do express natural emotions such as fear, sadness, and anger, many seek to help others in the face of adversity. Third, sharp changes in emotion counts between Harvey and Irma may be tied to their history. In the 2017 Atlantic hurricane season, Harvey materialized as a Cat-4 hurricane, and Irma followed around two weeks later as a Cat-5 hurricane. 5 Through side-by-side comparisons of both hurricanes' tweets, we found the Irma tweets had more descriptions of destruction and its aftermath. These changes in discourse potentially explain shifts between the emotion distributions. Thus far, we have analyzed each Plutchik-24 emotion in isolation. In this section, we ask the following questions: How do Plutchik-8 emotion groups co-occur with one another? Do co-occurrence patterns change across hurricanes? Figure 3 shows co-occurrence heatmaps for each hurricane. Intuitively, we see strong correlations between polarized emotions, that is, emo-5 Abbreviations for Categoryx . This refers to the Saffir-Simpson scale for classifying hurricanes based on sustained wind speed, which ranges from 1-5 in order of severity. tions categorized as positive and negative . For example, ( LOVE , AGGRESSIVENESS ) does not appear as frequently as ( LOVE , OPTIMISM ) or ( CONTEMPT , AGGRESSIVENESS ). However, this premise does not always hold; the pairs ({ DISAPPROVAL , REMORSE }, OPTIMISM ) also co-occur across all hurricanes. Representative of this phenomenon is the tweet: I'm raising money for Hurricane Maria Destroyed Everything. Click to Donate: <URL> via <USER> . The user indicates disapproval towards the hurricane by evoking pathos, but also shows optimism by donating money to a relief effort. Finally, similar to our previous observations (4.2), we notice an increase in co-occurrence frequencies from Harvey  Irma. This increase is, somewhat surprisingly, most apparent with ( AWE , OPTIMISM ), although ({ DISAPPROVAL , REMORSE }, AWE ) frequencies also exhibit a noticeable gain. Once again, we posit that users may be expressing their sadness regarding the Cat-4  Cat-5 jump, but at the same time, offering solidarity to those affected by the hurricanes. We now turn to modeling the emotions in HURRICANEEMO . Because Plutchik-24 emotion counts are heavily imbalanced, we group them into Plutchik-8 emotions and consequently create 8 binary classification tasks. The tweets are assorted into their respective label buckets; because tweets may be labeled with more than one emotion, each belongs to one or more buckets. These buckets represent positive samples (i.e., tweets labeled with that emotion). To create negative samples, we sample an equal amount from Plutchik-8 Emotion Train Valid Test Aggressiveness 4,209 526 527 Optimism 11,902 1,488 1,488 Love 2,569 321 322 Submission 6,092 762 762 Awe 7,324 916 916 Disapproval 5,931 741 742 Remorse 7,732 967 967 Contempt 3,763 470 471 Table 4: Train, validation, and test splits for each Plutchik-8 emotion. other buckets. From here, we shuffle the positive and negative samples and perform an 80/10/10 split to create the train, validation, and test sets. 6 Table 4 enumerates the splits. We consider both traditional neural models and pre-trained language models. We implement our models in PyTorch (Paszke et al., 2019) and perform all experiments on an NVIDIA Titan V GPU. Training and optimization hyperparameters are detailed in Appendix C. We report mean performance across 10 runs, each with a different random initialization. Below, we elaborate on our models: Traditional Neural Models. Each is equipped with 200D GloVe embeddings pre-trained on 2B tweets (Pennington et al., 2014): (1) Logistic Regression: We average the word embeddings of each token in the sequence (Iyyer et al., 2015); (2) CNN: A word-level CNN (Kim, 2014) with 100 filters of size [3, 4, 5] obtains representations. They are max-pooled and concatenated row-wise. We also experiment with a character-level CNN with filter sizes [5, 6, 7]; (3) GRU: A one-layer, unidirectional GRU (Cho et al., 2014) with a hidden dimension of 100 obtains features, which are mean pooled. For all models, penultimate representations are projected with a weight matrix W  R d  2 . Pre-trained Language Models. We fine-tune base versions of BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) using the Hugging-Face Transformers library (Wolf et al., 2019). We 6 We also experimented with keeping all negative samples as opposed to sampling an equal amount. Each binary task had around 5-7x more negative samples; this significantly hurt model performance. Even with a class imbalance penalty, the models almost never predicted positive samples. Note that although, in aggregate, the number of positive and negative samples match, they do not necessarily match in the train, validation, and test splits. use the sentence representations embedded in the [CLS] token, then project it with a weight matrix W  R d  2 . The language model and classification parameters are jointly fine-tuned. Table 5 presents our classification results. We make the following observations: BERT consistently outperforms other models on most emotion tasks. BERT shows strong performance across all 8 binary tasks in comparison to traditional neural models and RoBERTa. Unlike most traditional neural models, its accuracy never falls below random chance, showing it captures at least some of the complex phenomena present in our dataset. However, our tasks remain challenging for both types of models alike. For traditional models, word embeddings alone do not provide enough representational power to model our emotional contexts. Although GRUs perform well on EMONET (Abdul-Mageed and Ungar, 2017), we suspect that they simply memorize emotion lexicons (4.1), which is not a notable strategy for capturing implicit emotions. Nevertheless, BERT only obtains an average accuracy of about 64%. This leaves plenty of room for future work; we perform a comprehensive error analysis as a step towards this goal (5.3). Better pre-trained models (e.g., RoBERTa) do not necessarily help performance. Unlike popular benchmarks such as GLUE (Wang et al., 2018) where more pre-training monotonically increases performance, rather encouragingly, we do not observe the same trend. RoBERTa's average performance is around 5% better than GRU's, but still around 6% worse than BERT's. We hypothesize that this drop in performance is attributed to pre-training  fine-tuning domain discrepancies. That is, RoBERTa's (additional) pre-training data (e.g., CC-News) may be too distant from Twitter data, which is known for its short contexts and unique vernacular (Ritter et al., 2011). We encourage practitioners to avoid applying state-of-the-art models without augmenting them with task-guided pre-training objectives, as we explore later (6). Using our BERT model, we sample 25 test errors from each of the 8 emotion tasks, yielding a total of 200 errors. We group the errors into the following categories: lexical and syntactic cues (45%), insufficient context (24%), entity mentions (15%), subjective labeling (10%), and unknown reasons (6%). The top three categories are discussed below: Lexical and Syntactic Cues. BERT often relies on surface-level lexical features to make predictions, as do most emotion prediction models. This bias also extends to certain syntactic features, such as punctuation. In pls be safe everyone!!!! , BERT associates the exclamation mark with a positive emotion, but here, the speaker is more concerned. Insufficient Context. Users often comment on events, public policies, or linked content that, by themselves, do not carry features for supervised learning. This type of error is not necessarily a shortcoming of BERT, but rather our dataset. For example, in for [tracy mcgrady] 1 , [hall induction] 2 muted by effects of [hurricane harvey] 3 at home , one use external knowledge to reason between the noun phrases and discern the latent emotions. Entity Mentions. BERT also makes erroneous predictions in the presence of certain entity mentions. For example, BERT classifies this tweet as AGGRESSIVENESS : nytimesworld: mexico offered aid to texas after harvey. but after an earthquake and hurricane, it says all help is needed at home. Here, the user is merely quoting a AGR OPT LOV SBM AWE DSP RMR CNT AVG NO-PRETRAIN 67.6 75.0 54.0 67.4 68.3 55.7 58.5 66.8 64.1 Supervised Transfer EMONET 73.5 75.2 55.2 68.8 67.5 53.1 60.0 71.7 65.6 SENTIMENT 72.8 75.8 62.7 71.0 65.6 53.4 57.0 67.3 65.7 Unsupervised Transfer EMONET 72.1 75.1 54.0 61.0 65.1 54.2 60.7 69.4 63.9 SENTIMENT 69.1 74.9 53.6 66.2 67.3 54.3 57.9 64.4 63.5 HURRICANEEXT 73.6 75.4 69.8 68.9 69.7 57.9 60.2 70.2 68.2 Table 6: Task-guided pre-training accuracies (abbreviations defined in Table 5). news statement as opposed to formulating opinions regarding NY Times' discourse. Because the sentiment towards NY Times is negative in our datasets overall (due to public backlash on its stories), BERT likely capitalizes on this mention-emotion bias. To improve upon our baselines, we explore pretraining as a means of implicitly incorporating an inductive bias into our BERT model. Our hope is that these pre-training tasks will not only make BERT more robust in the Twitter domain, but also provide useful (albeit abstract) knowledge for the end emotion prediction tasks. For brevity, we chiefly focus on BERT, although our methods can be generalized to other pre-trained models. Setup. We explore, in isolation, supervised and unsupervised pre-training tasks. For the supervised setting, we pre-train on a multi-class emotion task (EMONET ) (Abdul-Mageed and Ungar, 2017) and binary sentiment analysis task (SENTIMENT ) (Go et al., 2009). For the unsupervised setting, we pretrain on dynamic masked language modeling (Liu et al., 2019) on (unlabeled) samples from EMONET , SENTIMENT , and HURRICANEEXT (3.1). For both types of tasks, we further pre-train BERT for a fixed number of epochs, then fine-tune it on a HURRICANEEMO task. We compare these results to NO-PRETRAIN , namely the BERT results verbatim from Table 5. We report mean performance across 10 pre-training  fine-tuning runs. Further training details, including samples sizes for the pre-training tasks, are available in Appendix D. Results. Table 6 shows the pre-training results. Supervised pre-training significantly helps with 3-4 emotions, but degrades overall performance on 2-4 emotions. We posit SENTIMENT aids emotions with highly predictive features. For example, wtf in it's literally the size of texas. wtf is correlated with AGGRESSIVENESS , but no such lexical cues exist in not all heros wear capes <3 thank you stanley homeless #hurricane evacuee grooms lost pets, which is an AWE sample. The unsupervised pre-training results also show a couple trends. First, EMONET largely hurts downstream performance, especially reducing SUBMISSION accuracy by -6%. Second, SENTIMENT (in its unlabeled form) yields no noticeable benefits. This implies sentiment information is much more valuable, but of course, subject to the fact that the emotion task is heavily aligned with the original sentiment task. Third, we obtain encouraging results with HURRICANEEXT pre-training. The gains are most noticeable on AGGRESSIVENESS and LOVE , but this objective adds +1-2% accuracy for tasks on which supervised pre-training suffered. When new disasters emerge, it is likely we may not have emotion annotations, as alluded to previously (2). Nevertheless, these annotations would be valuable for organizations trying to understand the emotional profile of users during a crisis (Fraustino et al., 2012). In this section, we explore ways to leverage supervision from large-scale emotion datasets (e.g., EMONET (Abdul-Mageed and Ungar, 2017)) in providing labels for our hurricane emotion datasets. We frame this problem as unsupervised domain adaptation; EMONET is the labeled source domain and our hurricane datasets are the unlabeled target domain. Below, we elaborate AGR OPT LOV SBM AWE DSP RMR CNT AVG SRC-ONLY 53.3 42.2 43.4 47.1 54.7 49.8 62.5 56.5 51.2 PRETRAIN-SRC 54.8 43.2 45.1 47.8 54.4 50.4 63.3 57.1 52.0 PRETRAIN-TRG 55.0 44.2 46.2 48.0 55.5 49.9 63.7 60.5 52.9 PRETRAIN-JOINT 52.7 44.2 45.5 47.8 54.8 49.9 61.6 56.3 51.6 TRG-ONLY 67.6 75.0 54.0 67.4 68.3 55.7 58.5 66.8 64.1 Table 7: Unsupervised domain adaptation accuracies (abbreviations defined in Table 5). Framework. EMONET was conceived as a multi-class classification task for Plutchik-8 emotions (Abdul-Mageed and Ungar, 2017). In contrast, we introduce binary classification tasks, one for each Plutchik-8 emotion. We split the EMONET multi-class task into 8 binary tasks; this creates a one-to-one alignment between each source and target domain task. We separately perform unsupervised domain adaptation for each binary task. Methods. We use our BERT model (without task-guided pre-training) as the underlying classifier. Following Han and Eisenstein (2019), we chiefly focus on using strategic pre-training techniques that enable effective transfer between disparate domains. The systems for comparison are: (1) SRCONLY : BERT is trained in the source domain and evaluated in the target domain; (2) TRG-ONLY : BERT is trained and evaluated in the target domain. These results are borrowed verbatim from Table 5; (3) PRETRAIN -*: BERT undergoes dynamic masked language modeling pre-training using data from domain *, is trained in the source domain, and finally evaluated in the target domain (Han and Eisenstein, 2019). PRETRAIN-SRC only uses pre-training samples from the source domain, PRETRAIN-TRG only uses samples from the target domain, and PRETRAIN-JOINT uses samples from both the source and target domains. 7 We report mean performance across 10 pre-training  fine-tuning runs. Results. Table 7 shows the unsupervised domain adaptation results. Overall, we do not find a significant increase in performance over the SRCONLY baseline. Pre-training consistently adds +1% in average accuracy, but still leaves a large gap between PRETRAIN-SRC and TRG-ONLY . Re-7 PRETRAIN-JOINT is conceptually similar to ADAPTABERT in Han and Eisenstein (2019), however, we dynamically generate pre-training data (Liu et al., 2019). gardless, we have a few observations. First, we do not see a (relatively) large increase in performance for SUBMISSION , AWE , DISAPPROVAL , and REMORSE . These emotions may need more explicit strategies to enable domain adaptation. This is also supported by our previous results (6), where we also do not see a (relatively) large benefit from task-guided pre-training. Second, PRETRAINJOINT performs worse than both PRETRAIN-SRC and PRETRAIN-TRG . We posit that, for our emotion tasks, pre-training with a mixture of domains yields a noisier training signal compared to a parameter bias towards the target domain. We present HURRICANEEMO , an annotated dataset of perceived emotions spanning 15,000 tweets from multiple hurricanes. Tweets are annotated with fine-grained Plutchik-24 emotions, from which we analyze implicit and explicit emotions and construct Plutchik-8 binary classification tasks. Comprehensive experiments demonstrate our dataset is a challenging benchmark, even for large-scale pre-trained language models. We release our code and datasets as a step towards facilitating research in disaster-centric domains. Thanks to Katrin Erk for reviewing an early version of this manuscript, Yasumasa Onoe for discussions on masked language model pre-training, and the anonymous reviewers for their helpful comments. This work was partially supported by the NSF Grants IIS-1850153, IIS-1912887, and IIS-1903963.
abstain abstain abstain abstain objective result abstain abstain abstain abstain other abstain abstain abstain abstain method abstain method method result method method abstain result abstain method result method abstain abstain abstain other abstain abstain objective method abstain other abstain abstain objective method abstain abstain method method objective result method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain method abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain method abstain method method method method method abstain method abstain abstain method method abstain abstain abstain abstain abstain result result abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain method method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method method abstain method method abstain abstain abstain method abstain abstain method abstain method method method abstain method abstain method method method objective abstain other other