In the field of sentiment analysis, several studies have highlighted that a single sentence may express multiple, sometimes contrasting, sentiments and emotions, each with its own experiencer, target and/or cause. To this end, over the past few years researchers have started to collect and annotate data manually, in order to investigate the capabilities of automatic systems not only to distinguish between emotions, but also to capture their semantic constituents. However, currently available gold datasets are heterogeneous in size, domain, format, splits, emotion categories and role labels, making comparisons across different works difficult and hampering progress in the area. In this paper, we tackle this issue and present a unified evaluation framework focused on Semantic Role Labeling for Emotions (SRL4E), in which we unify several datasets tagged with emotions and semantic roles by using a common labeling scheme. We use SRL4E as a benchmark to evaluate how modern pretrained language models perform and analyze where we currently stand in this task, hoping to provide the tools to facilitate studies in this complex area. Emotion detection  a long-standing open problem in Natural Language Processing (NLP)  is the task of automatically associating one or more emotions with a text. Even though emotional states are highly subjective and often depend on several factors, such as one's past experiences, culture and ed-ucation, the automatic identification, categorization and analysis of emotions in texts has been found to be beneficial in a wide array of downstream tasks, such as hate speech detection (Markov et al., 2021), sarcasm detection (Chauhan et al., 2020), and modeling political discourse (Huguet Cabot et al., 2021), inter alia . In the past decade, Deep Learning techniques have become ubiquitous in the development of automatic systems for an increasing number of NLP tasks, including emotion detection (Chatterjee et al., 2019). However, most of the effective neural-based approaches still require significant amounts of training data in order to learn to perform at their best. For this reason, with a view to bootstrapping the development of neural systems for emotion detection, there have been several efforts to annotate corpora with emotions manually (Bostan and Klinger, 2018). Nevertheless, over the past few years, numerous studies have indicated that a short text, even a single sentence, may contain multiple  at times concurring, at other times contrasting  sentiments and emotions. And not only this, two emotions in the same sentence may be experienced, targeted, and/or caused by different semantic constituents which, similarly to predicate-argument structures in Semantic Role Labeling (SRL), can be linked to form abstract semantic structures. The potential applications in social media analysis, abuse detection, and other actively studied areas in NLP (Rajamanickam et al., 2020) of such automatically-extracted emotion-focused semantic structures have prompted researchers to create datasets aimed at investigating the capabilities of modern systems to parse emotional events (Ober-lnder et al., 2020). Unfortunately, despite the increasing interest in this area, currently available gold datasets feature heterogeneous structures and characteristics, ranging from varying sizes to different domains, file format, splits and, most importantly, non-overlapping emotion categories. We argue that this heterogeneity obstructs, or at least hinders, further progress in this relatively new area of sentiment analysis. In this paper, we take a step towards addressing the above-mentioned issues and introduce a unified framework for Semantic Role Labeling for Emotions (SRL4E). In SRL4E, we unify several gold but heterogeneous datasets that contain anno-4586 tations both for emotions and for their semantic constituents, so as to obtain a new homogeneous dataset that covers diverse domains and that can be used to train, validate and evaluate current and future work in this task. Our contributions can be summarized as follows:  We propose a unified gold benchmark for training and evaluating a system on Semantic Role Labeling for Emotions (SRL4E);  We take advantage of SRL4E to show the inadequacy of training a model on domain-specific data and the benefits of our unified framework;  We show the advantages of bilingual transfer from English to Chinese, and vice versa, in SRL4E. We release SRL4E at https://github.com/ SapienzaNLP/srl4e in the hope that our unified framework will become a stepping stone for the development and evaluation of current and future approaches to Semantic Role Labeling for Emotions. Emotion classification datasets. Currently, there are a wide variety of datasets annotated with emotion classes, ranging across different domains and using different annotation schemes. Among others, we can find datasets on emotional experiences (Scherer and Wallbott, 1994), children's fairy tales (Alm et al., 2005), news headlines (Strapparava and Mihalcea, 2007), blog posts (Aman and Szpakowicz, 2007, 2008), news (Lei et al., 2014), social media posts and reviews (Buechel and Hahn, 2017), dialogs (Li et al., 2017; Chatterjee et al., 2019), Facebook posts (Preotiuc-Pietro et al., 2016), with many focusing on tweets (Mohammad, 2012; Mohammad and Bravo-Marquez, 2017; CrowdFlower, 2016; Liu et al., 2017; Schuff et al., 2017) due to their tendency to have dense emotional content. To meet such a diversity of contents and formats, Bostan and Klinger (2018) created a unified resource for emotion classification comprising many of the aforementioned datasets, while Tafreshi and Diab (2018), instead, added an additional clause-level annotation layer to some existing resources. More recent efforts, such as GoEmotion (Demszky et al., 2020), XED (hman et al., 2020) and CancerEmo (Sosea and Caragea, 2020), provide, respectively, emotion annotations for Reddit comments, multilingual subtitles and blog posts about health problems. Although the above-mentioned corpora have enabled systems to perform emotion detection across different domains, their annotations are sentence-level and, therefore, introduce an oversimplifica-tion: they indicate only the overall sentiment and/or emotion that appears in a given text, neglecting the cases in which a short text, even a single sentence, may express multiple emotions. Furthermore, the aforementioned datasets do not indicate which part of the text elicits an emotion and who experiences, is the target of, or causes that emotion. As a consequence, a system trained on these datasets may produce predictions that are hard to interpret and more difficult to use in real-world applications. To overcome these problems, we rely on resources that not only indicate emotions, but also identify their semantic constituents, namely, emotional CUE s, EXPERIENCER s, TARGET s and STIMULI . Emotion Taxonomy. Among the studies that aim to identify the fundamental emotions, Ekman (1992) proposed a set of six categories: anger , disgust , fear , joy , sadness and surprise ; Plutchik (1980) shared the same set with two additions: anticipation and trust . Instead of relying on discrete categories, Russell (1980) proposed the circumplex model where every emotion can be described by three continuous values: arousal , dominance and valence . More recent studies in psychology use more fine-grained sets of emotions, ranging from 12 (Cowen et al., 2019b) to 28 categories (Cowen and Keltner, 2020), devised depending on the context of the study, e.g., speech prosody and facial expressions. However, the analysis of Demszky et al. (2020) over a fine-grained set of 28 emotions suggests that a large number of categories results in more frequent disagreements on similar classes (such as anger and annoyance , or excitement and joy ) which, in turn, can lead to low inter-annotator agreement and unbalanced distributions among some of these categories. Therefore, we adopt Plutchik's Wheel of Emotions (Plutchik, 2001), which provides clearly distinct and well-defined coarse-grained categories, whose composition can be used to virtually describe all other fine-grained sets. Moreover, some datasets in SRL4E (Moham-mad et al., 2014; Kim and Klinger, 2018; Bostan et al., 2020) already use Plutchik's or Plutchik-4587 based categories. Emotions and SRL. Over the past few years, automatic systems for SRL have achieved impressive performance in identifying and labeling predicate-argument relations (Shi and Lin, 2019; Conia and Navigli, 2020; Blloshmi et al., 2021; Conia et al., 2021), and have long become useful tools in several downstream tasks, from Question Answering (He et al., 2015) to Machine Translation (Marcheggiani et al., 2018). Defined by Mrquez et al. (2008) as the task of answering the question  Who did What to Whom, Where, When and How? , SRL is almost a natural choice for the extraction of the semantic constituents of those events that elicit emotional states. Indeed, emotional CUE s can be seen as particular types of predicates, and their semantic constituents as their arguments. Among the currently available datasets for emotion detection, there are some that also provide this kind of more granular semantic information. In particular Aman and Szpakowicz (2007) and Liew et al. (2016) released corpora that indicate multiple emotions and their corresponding emotion CUE s in each sentence; Ghazi et al. (2015) and Gao et al. (2017) indicate the cause of an emotion, with the latter providing such annotations both in English and in Chinese. Finally, Mohammad et al. (2014), Mohammad et al. (2015), Kim and Klinger (2018) and Bostan et al. (2020) provide annotations for emotion CUE s, EXPERIENCER s, TARGET s and STIMULI , employing, however, different sets of emotions in different domains. This means that the results of a system trained on one of these datasets cannot be compared against the results of another system trained on a different dataset, emphasizing the need for a unified framework to train and evaluate future approaches to this task. This is also evidenced by the success of existing works, e.g. Bostan and Klinger (2018) for sentence-level Emotion Classification and Raganato et al. (2017) for Word Sense Disambiguation. In SRL4E, not only do we aggregate the resources under the same task formulation, but we also manually correct their inconsistencies and unify the different emotion schemes. In this Section, we introduce SRL4E. We first describe the categories of emotions and the format of the semantic roles we adopt to unify the annotation scheme of the original datasets. Next, we provide a short overview of the datasets included in SRL4E. Finally, we give a formal definition of the task. The task of SRL (Gildea and Jurafsky, 2000) is aimed at identifying, given an input sentence, who or what the participants are in an action or event denoted by a predicate. As mentioned in Section 2, this is comparable to answering the question  Who did What to Whom , Where , When and How ? (Mrquez et al., 2008). When it comes to emotions, however, the task does not necessarily revolve around an action, but more precisely around an emotional cue , a word or an expression that acts as a trigger for an emotion. Therefore, it would be more appropriate to reformulate the question as:  Who feels What , towards Whom and Why ?. To answer this question, we first need to define a set of semantic roles, i.e., semantic relations that can exist between an emotion CUE and its semantic constituents. Following previous work (Mohammad et al., 2014; Bostan et al., 2020), we take a subset of semantic roles, namely, EXPERIENCER , TARGET and STIMULUS , from those defined in the Emotion semantic frame of FrameNet (Baker et al., 1998). While the use of thematic roles allows for human-readable labels (Kipper Schuler, 2005; Di Fabio et al., 2019), we also provide their respective definitions in Table 1. 3.2 Choosing a common set of emotions In psychology, the debate on which categories are best suited for describing emotions is still open (Barrett et al., 2018; Cowen and Keltner, 2018; Cowen et al., 2019a). There are numerous studies that try to tackle this problem, and some of the most authoritative were briefly described in Section 2, above. In this work, we adopt Plutchik's Wheel of Emotions (Plutchik, 1980, 2001) to standardize the heterogeneous emotion categories used in the various datasets. Plutchik's Wheel of Emotions is composed of a coarse-grained set of 8 basic emotions: anger , fear , sadness , disgust , surprise , anticipation , trust , and joy . These emotions can be compounded into dyads which express the much wider range of human feelings, with the advantage of maintaining a solid and unambiguous base set. For example, combining anticipation together with joy describes the emotion of optimism , whereas anticipation with sadness describes pessimism . Further compositions are described in Appendix A. SRL4E includes 6 datasets: 4588 Role Definition CUE Trigger word or expression that describes (even implicitly) an emotion. Kim and Klinger (2018) and Bostan et al. (2020) use Plutchik's or Plutchik-based emotions;  Aman and Szpakowicz (2007) and Gao et al. (2017) use Ekman's or Ekman-based emotions, which are a subset of Plutchik's set and can be directly mapped to it;  Mohammad et al. (2014) use 19 emotions, but provide a mapping to Plutchik's emotions;  Liew et al. (2016) use 28 emotions for which we provide a mapping to Plutchik's emotions. We provide a more detailed description of each dataset in Section 3.3. As a further contribution, we produce an alignment of each set of emotions to a sentiment polarity  positive, negative, neutral, or other (used when polarity cannot be inferred based on the emotion category)  to allow SRL4E also to be used to train and evaluate a system on Semantic Role Labeling for Sentiments. In the following, we describe the datasets that we included in SRL4E. For each dataset, we provide general information, including source, domain, format and tagging scheme. We also indicate where we intervened manually to identify and correct errors such as typos, format errors and inconsistencies. Table 2 reports the sizes of the original and converted datasets in SRL4E. Table 3 summarizes which annotations form part of the original corpora and, therefore, which ones are also part of SRL4E. We report the license, availability and link of each resource in Appendix B. Blogs. This dataset, proposed by Aman and Szpakowicz (2007), consists of 5,202 sentences, extracted from 173 online blog posts. Each sentence Resource Original SRL4E % Blogs 5,202 4,855 93.3 Elections 1,385 1,024 73.9 EmoTweet 15,553 15,553 100.0 GNE 5,000 5,000 100.0 NTCIR (ZH) 2,022 1,956 96.7 NTCIR (EN) 1,826 1,796 98.4 REMAN 1,720 1,705 99.1 All 32,708 31,889 97.5 Table 2: Original/new sizes after conversion to SRL4E. is annotated using Ekman's six emotion categories and no emotion , along with intensities. The words or spans that indicate emotions are marked, allowing us to remap them to the CUE in our unified format. The dataset was annotated by two experts. For each sample, we decided to consider only the CUE s that were annotated with the same emotion by both annotators. Where possible, we manually identified and corrected some annotations containing typos. Elections. This dataset, introduced by Mohammad et al. (2014, 2015), includes 1,385 unique tweets related to the 2012 US presidential election and collected using the Twitter API. The tweets were annotated via crowdsourcing using an informative tagging scheme which comprised not only 4589 I stand by Obama 100% he deserves another 4yrs in office. a set of 19 emotions, but also other features such as emotion intensity, valence, purpose, style, CUE , EXPERIENCER , TARGET and STIMULUS . Each sample was annotated by multiple people, i.e., each sample appears more than once with different annotations, one for each annotator. We adjudicated role spans by majority voting, discarding all the tweets with conflicting annotations. EmoTweet. EmoTweet, presented by Liew et al. (2016), is the largest dataset that we include in our unified resource. It comprises 15,553 tweets, collected through the Twitter API using various sampling strategies (e.g., by user, by topic, random, etc.) and annotated via crowdsourcing. The original tagging scheme of this dataset features 28 emotion categories along with valence and arousal. For each emotion, the CUE s are indicated and are easily mappable to our unified format. However, a mapping to Plutchik's emotions is not provided by the authors, so we formulated a conversion scheme based on the similarity of the emotion categories with those from other works that are instead mapped to Plutchik's emotions, such as Demszky et al. (2020). In addition, we also intervened to identify and manually correct some typos in the annotations. GNE. GoodNewsEveryone, proposed by Bostan et al. (2020), is a dataset composed of 5,000 news headlines from 82 sources, annotated via crowdsourcing. It is labeled with writer and reader emotions using a set of emotions derived from Pluthick's classes and is, therefore, easily mappable to the standard Plutchik set. To keep the annotations consistent with those of the other datasets in our unified framework, we considered only the writer's emotions. GNE provides annotations for every semantic role we include in our framework, namely, CUE , EXPERIENCER , TARGET and STIMULUS , making this resource highly valuable for our purposes. Whenever possible, we identified and manually corrected the annotations that contained typos. NTCIR 13 ECA. This dataset was proposed as a part of the NTCIR 13 Emotion Cause Analysis task. It consists of 1,826 unique sentences from English novels and 2,022 unique sentences from Chinese news, annotated using Ekman's classes. Moreover, emotion keywords and causes are annotated, making them suitable to be considered, respectively, as CUE and STIMULUS in our unified format. REMAN. Relational EMotion ANnotation, introduced by Kim and Klinger (2018), is a corpus consisting of 1,720 fictional text excerpts from Project Gutenberg. These documents were annotated using an informative tagging scheme, which included emotion categories based on Plutchik's set, CUE , EXPERIENCER , TARGET , STIMULUS , named entities, events and coreferences, making it another desirable dataset for our unified framework. For some sentences, we automatically identified and manually corrected some typos in order to increase the overall quality of this dataset. Here we provide a more formal definition of the SRL4E task. Unlike the majority of previous work on emotion detection, instead of assigning an emotion to a sentence, we associate each emotion with a CUE . In this way, in each sentence, more than one CUE can be identified and associated with its corresponding emotion category and semantic roles, allowing the coexistence of multiple emotions, EXPERIENCER s, TARGET s and STIMULI in the same sentence. A visual representation of the relationship between CUE , emotion category and roles is shown in Figure 1. To the best of our knowledge, other than SRL4E, Liew et al. (2016) and Kim and Klinger (2018) are the only approaches that leverage CUES to model the presence of multiple emotions in a sentence. for Emotions can be divided into three key steps: CUE identification, emotion classification and role identification. While there are no hard constraints on the order of these steps, we believe that CUE identification should be done first since its output will serve as the input of the other two steps, however, we also believe that our framework could be a step towards the development of joint approaches that solve the three steps at the same time. Cue identification. As we described earlier, the CUE acts similarly to a predicate in SRL. Indeed, the main objective of CUE identification is to recognize where and how many emotions are present in a sentence, and what their trigger words or expressions are. The output of this step consists of a set of CUE s, each corresponding to an emotion in the text, as illustrated in Figure 1. Emotion classification. Traditional approaches in emotion classification take as input a sentence and output the emotion class corresponding to that sentence. In SRL4E, instead, given a pair (sen-tence, CUE ) we want to classify the emotion expressed in the sentence by the indicated input CUE . Note that the result of this approach is not necessarily the same as a sentence-level approach. Role identification. As previously stated, SRL aims at identifying the semantic constituents of an action expressed by a predicate. In SRL4E, instead, we are interested in identifying the actants of an emotional event which is hinted at by the CUE . Therefore a CUE can be considered in the same way as a predicate in SRL and role identification consists in identifying all those spans of text that have a semantic relationship  EXPERIENCER , TARGET , STIMULUS  with the CUE . Emotion classes distribution. Depending on the dataset, the distribution of emotion classes changes drastically, as illustrated in Figure 2. For example, in Elections, which contains random tweets related to an American election campaign, almost 45% of samples are tagged with disgust , as one might expect: this is because many of the tweets in question tend to discredit the opposing party; similarly, the second most used class is trust in the tweets in favor of candidates. Another interesting example is GNE, where the most frequent category is surprise , highlighting the sensationalistic tone typically found in newspaper headlines. It is worth noting that, in contrast to each individual dataset, our unified dataset includes a fairly balanced distribution between categories, where the only category that appears more often is joy (20%), while all the others are between 6% and 10%, approximately. Other statistics. The statistics reported in Table 4 show the heterogeneity of the resources included in our framework, with very different text and role lengths. In fact, datasets containing sentences from similar domains share similar values. For example, REMAN and the English version of NTCIR both come from novels and they have comparable text lengths, from 58 to 59 words on average. Instead, Blogs (from online blog posts), Elections and EmoTweet (from tweets) have much shorter sentences, from 14 to 16 words, approximately. Table 4 also shows that almost all CUE s are very short, usually around 1-2 words; only those datasets involving tweets have a much higher value. In fact, in EmoTweet and Elections, CUE s contain 4 and 8 words on average, respectively, due to their dense emotional content and, therefore, their larger number of trigger expressions. It is interesting to note that all datasets feature a similar average length for STIMULI , regardless of the domain. Borderline examples. SRL4E's formulation is based on the presence of CUE s within sentences, which are seen as the trigger of the emotion in that context. This formulation particularly suits those domains where emotions are expressed explicitly, such as GNE, NTCIR and REMAN. However, handling CUE s becomes non-trivial in some situations, for example in social networks (Elections and EmoTweet) and blog posts (Blogs). In these contexts, language features numerous implicit references and ironic content, where the mere presence of an emoji or a particular punctuation mark completely changes the context. In our task formulation, the presence of a CUE is a fundamental requirement even if it may be difficult to identify, as we want to be able to model multiple, sometimes opposite, emotions in the same sentence. Here is an example:  @user Quieter. My sis, brother in law and habibti are going back to Ireland this afternoon [ ;/ CUE ] Tennis doubles [ sounds fun CUE ] ! [ Enjoy CUE ] ! #Juice! In this case, the sadness emotion is associated only with the first CUE , which is ;/, while the joy emotion is associated with the other two. Even if a CUE is composed only of punctuation marks (or emojis), it may still be the only useful signal for disambiguating the emotion, or for separating the presence of multiple emotions in the same sentence. In this Section, we analyze the benefits that our unified framework can bring to a neural model, based on recent contextualized representations from a pretrained language model. The main roadblock to the development of neural models for Semantic Role Labeling for Emotions the heterogeneity of the emotion labels employed by each currently available dataset. Therefore, we first evaluate the benefits that a unified framework brings in emotion classification. Note that, differently from traditional sentence-level Emotion Detection, here we are interested in assigning an emotion to a given (sentence, CUE ) pair, so as to allow a sentence to be assigned different emotions depending on the CUE considered. Model description. We design a simple neural baseline composed mainly of a BERT-based word representation module and a stack of BiLSTM layers. Given an input sentence w and a pre-identified CUE c , the two are concatenated as an input sequence s = [CLS] w [SEP] c [SEP] and fed into the BERT-based word representation module, obtaining a sequence of word encodings e = BERT ( s ) . These word encodings are further processed by a stack of 2 BiLSTM layers with hidden size 512 to obtain a new sequence of output encodings o = BiLSTM ( e ) . Finally, the output encoding o [CLS] corresponding to the [CLS] token is fed into a linear classifier which outputs the emotions corresponding to the (sentence, CUE ) pair. Each model configuration is trained to minimize a binary cross-entropy loss for emotion classification (more than one emotion can be assigned to a given input), for a total of 20 epochs with Adam and a learning rate of 10  3 , leaving the weights of the underlying language model frozen. Results. Table 5 shows the results of our system on emotion classification. First, our unified framework reveals that a system trained on a single dataset can achieve good results on the test set of the same dataset, i.e., on an in-domain evaluation, but is not able to perform as well on other datasets, i.e., on out-of-domain evaluations. Instead, the same system trained jointly on the datasets of SRL4E is able not only to perform consistently across all the test sets, but also to improve over the same system trained on in-domain data only, demonstrating empirically the effectiveness of employing a unified scheme for emotion classification. This is not a given, since each dataset differs  sometimes significantly  from the others in domain and linguistic register. On average, when using multilingual-BERT as the underlying language model, our unified framework provides an improvement of 11.2% in F1 score over EmoTweet, the second best dataset (64.3% against 53.1%). Moreover, Table 5 shows that our unified framework 4592 Trained on Evaluated on (F1 score) Model BL EL ET GN N/E N/Z RE BL EL ET GN N/E N/Z RE ALL m u ltili ngua l BERT (cid:52)       51.0 13.3 38.6 15.3 24.6 11.1 21.8 29.9  (cid:52)      9.2 40.5 21.7 15.6 8.7 7.9 13.7 17.2   (cid:52)     49.9 32.2 76.7 20.1 48.8 22.8 38.2 53.1    (cid:52)    34.3 25.5 30.3 29.0 29.0 18.3 23.3 28.6     (cid:52)   42.1 10.8 34.2 4.0 30.2 11.9 20.1 26.0      (cid:52)  8.9 5.7 17.5 2.6 21.4 22.8 9.2 13.9       (cid:52) 35.4 7.8 22.1 4.8 16.1 2.8 23.5 17.8 (cid:52) (cid:52) (cid:52) (cid:52) (cid:52) (cid:52) (cid:52) 65.9 40.7 74.6 33.7 78.5 77.8 54.1 64.3 Table 5: F1 scores on emotion classification . allows our system to improve in bilingual emotion classification (77.8% against 22.8% in F1 score on ALL). We now turn to CUE identification, where we aim to find every CUE in an input sentence. We frame this subtask as a BIO-tagging problem and devise a neural model to highlight the benefits of our unified framework in this task. Model description. For CUE identification, we use a similar system architecture to the one we used for emotion classification. However, this time the input of the BERT-based word representation module is just the input sentence, whereas the output is a sequence of BIO tags. Specifically, the output encodings o = o 1 , . . . , o n produced by the last BiLSTM layer are given to a classifier which learns to predict B-cue, I-cue or O. Results. As one can see in Table 6, similarly to what we observed in emotion classification, our unified framework highlights how a model trained on a single dataset is not robust to out-of-domain evaluations. Instead, the same model trained on all the datasets in SRL4E shows consistent results across all the test sets, providing a significant improvement in F1 score over the second best dataset, EmoTweet (56.5% against 47.3% in F1 score on ALL, with an absolute improvement of 9.2%). Model description. For role identification, we use an approach similar to that for CUE identification. Indeed, similarly to CUE identification, we model role identification as a BIO-tagging problem, with the only difference being that we provide the pre-identified CUE in input, i.e., the input sequence is s = [CLS] w [SEP] c [SEP], where w is the input sentence and c is the CUE span. Results. We find that our results on the identification of each role are in line with the results from CUE identification, leading us to draw similar conclusions (see Tables 7, 8 and 9). In general, we see a familiar pattern in which training our baseline model on a single dataset results in good perfor-4593 Trained on Evaluated on (F1 score) Model EL GN N/E N/Z RE EL GN N/E N/Z RE ALL m u ltili ngua l BERT (cid:52)     52.8 62.7 24.5 25.9 14.1 32.2  (cid:52)    42.4 75.8 21.9 22.4 13.8 37.1   (cid:52)   31.6 40.8 50.4 12.9 20.1 30.3    (cid:52)  16.5 16.5 20.1 56.2 15.9 38.5     (cid:52) 9.8 9.0 24.4 3.8 26.4 10.3 (cid:52) (cid:52) (cid:52) (cid:52) (cid:52) 54.5 76.3 52.7 57.8 16.6 62.5 Table 7: F1 scores of our baseline model on STIMULUS identification . mances on that specific dataset, but with significantly lower results on out-of-domain data. Results generally benefit from a unified resource. For instance, emotion classification and STIMULUS identification almost always struggle in out-of-domain evaluations, while they perform better when the model is trained on all the datasets at the same time. The only exception is CUE identification: when our model is trained on all the data in SRL4E, the performance drops when measured on each dataset separately. This is to be expected: while STIMULI follow a similar syntactic pattern across domains, CUE s appear in very different forms (e.g., Twitter usually contains highly informal language with explicit emotions, while news headlines tend to try to describe events objectively, making emotions more implicit). Instead, when the datasets share a similar domain, the model is able to generalize well, even in cross-lingual settings (such as the English and Chinese versions of NTCIR), highlighting once again the advantages of our unified framework. Recently, the study of emotions in NLP has been gaining interest, due to their potential not only for application to downstream tasks, but also for enhancing the interpretability of automatic outputs, especially when emotions are accompanied by information about their semantic constituents, i.e., their experiencers, targets and stimuli. However, recent efforts to provide manually annotated data for emotions and their semantic constituents have been heterogeneous in their annotation scheme, making it difficult to train, evaluate, and compare novel approaches. In this paper, we aimed at addressing these issues and presented a unified framework for the Semantic Role Labeling of Emotions (SRL4E). Our framework collects, cleans, and unifies the annotation schemes of six datasets that provide information about emotions and their semantic roles, making it easy to train and evaluate existing and future systems. We conducted several experiments to demonstrate empirically that our unified scheme is beneficial in each subtask, namely, emotion classification and role (experiencer, target, stimulus) identification, especially in bilingual settings (English-Chinese). With SRL4E, we hope to stimulate future research in this complex area at the intersection of Emotion Detection and Semantic Role Labeling. We release the software to reproduce the benchmark and our experiments at https://github.com/SapienzaNLP/srl4e . The authors gratefully acknowledge the support of the ERC Consolidator Grant MOUSSE No. 726487 and the European Language Grid project No. 825627 (Universal Semantic Annotator, USeA) under the European Union's Horizon 2020 research and innovation programme. This work was supported in part by the MIUR under grant Dipartimenti di Eccellenza 2018-2022 of the Department of Computer Science of Sapienza University of Rome.
abstain abstain abstain method method abstain other abstain abstain abstain abstain abstain abstain abstain objective method objective objective other other other other other other other other other method other other other other other method other other other other other other other other other other abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain method abstain abstain abstain method method abstain abstain abstain method abstain method abstain abstain abstain method abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result result abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain result abstain abstain objective method objective abstain other other other