Multi-label emotion classification is an important task in NLP and is essential to many applications. In this work, we propose a sequence-to-emotion (Seq2Emo) approach, which implicitly models emotion correlations in a bi-directional decoder. Experiments on SemEval'18 and GoEmotions datasets show that our approach outperforms state-of-the-art methods (without using external data). In particular, Seq2Emo outperforms the binary relevance (BR) and classifier chain (CC) approaches in a fair setting. 1 1 Introduction Emotion classification from text (Yadollahi et al., 2017; Sailunaz et al., 2018) plays an important role in affective computing research, and is essential to human-like interactive systems, such as emotional chatbots (Asghar et al., 2018; Zhou et al., 2018; Huang et al., 2018; Ghosal et al., 2019). Early work treats this task as multi-class classification (Scherer and Wallbott, 1994; Mohammad, 2012), where each data instance (e.g., a sentence) is assumed to be labeled with one and only one emotion. More recently, researchers relax such an assumption and treat emotion analysis as multi-label classification (MLC, Mohammad et al., 2018; Demszky et al., 2020). In this case, each data instance may have one or multiple emotion labels. This is a more appropriate setting for emotion analysis, because an utterance may exhibit multiple emotions (e.g., angry and sad, surprise and joy). The binary relevance approach (BR, Godbole and Sarawagi, 2004) is widely applied to multi-label emotion classification. BR predicts a binary indicator for each emotion individually, assuming that the emotions are independent given the input sentence. However, evidence in psychotherapy 1 Our code is available at https://github.com/ chenyangh/Seq2Emo suggests strong correlation among different emotions (Plutchik, 1980). For example, hate may co-occur more often with disgust than joy. An alternative approach to multi-label emotion classification is the classifier chain (CC, Read et al., 2009). CC predicts the label(s) of an input in an autoregressive manner, for example, by a sequence-to-sequence (Seq2Seq) model (Yang et al., 2018). However, Seq2Seq models are known to have the problem of exposure bias (Bengio et al., 2015), i.e., an error at early steps may affect future predictions. In this work, we propose a sequence-to-emotion (Seq2Emo) approach, where we consider emotion correlations implicitly. Similar to CC, we also build a Seq2Seq-like model, but predict a binary indicator of an emotion at each decoding step of Seq2Seq. We do not feed predicted emotions back to the decoder; thus, our model does not suffer from the exposure bias problem. Compared with BR, our Seq2Emo model implicitly considers the correlation of emotions in the hidden states of the decoder, and with an attention mechanism, our Seq2Emo is able to focus on different words in the input sentence that are relevant to the current emotion. We evaluate our model for multi-label emotion classification on SemEval'18 (Mohammad et al., 2018) and GoEmotions (Demszky et al., 2020) benchmark datasets. Experiments show that Seq2Emo achieves state-of-the-art results on both datasets (without using external data). In particular, Seq2Emo outperforms both BR and CC in a fair, controlled comparison. Emotion classification is an activate research area in NLP. It classifies text instances into a set of emotion categories, e.g., angry, sad, happy, and surprise. Well-accepted emotion categorizations include the six basic emotions in Ekman (1984) and the eight primary emotions in Plutchik's wheel of emotions (1980). Early work uses manually constructed emotion lexicons for the emotion classification task (Tokuhisa et al., 2008; Wen and Wan, 2014; Shahraki and Zaiane, 2017). Such lexicon resources include WordNet-Affect (Strapparava and Valitutti, 2004), EmoSenticNet (Poria et al., 2014), and the NRC Emotion Intensity Lexicon (Moham-mad, 2018). Distant supervision (Mintz et al., 2009) has been applied to emotion classification, as researchers find existing labeled datasets are small for training an emotion classifier. For example, Mohammad (2012) finds that social media users often use hashtags to express emotions, and thus certain hashtags can be directly regarded as the noisy label of an utterance. Likewise, Felbo et al. (2017) use emojis as noisy labels for emotion classification. Such distant supervision can also be applied to pretrain emotion-specific embeddings and language models (Tang et al., 2014; Ghosh et al., 2017). In addition, Yu et al. (2018) apply multi-task learning to combine polarity sentiment analysis and multi-label emotion classification with dual attention. Different from the above studies that use extra emotional resources, our work focuses on modeling the correlations among emotions. This improves multi-label emotion classification without using additional data. A similar paper to ours is the Sequence Generation Model (SGM, Yang et al., 2018). SGM accomplishes multi-label classification by an autoregressive Seq2Seq model, and is an adaptation of classifier chains (Read et al., 2009) in the neural network regime. Our paper models emotion correlation implicitly by decoder hidden states and does not suffer from the drawbacks of autoregressive models. Consider a multi-label emotion classification problem. Suppose we have K predefined candidate emotions, and an utterance or a sentence x can be assigned with one or more emotions. We represent the target labels as y = ( y 1 ,    , y K )  { 0 , 1 } K with y i = 1 representing that the i th emotion is on. Our Seq2Emo is a Seq2Seq-like framework, shown as Figure 1. It encodes x with an LSTM, and iteratively performs binary classifications over y i with another LSTM as the decoder. Formally, let a sentence be x = (x 1 ,    , x M ) . We first encode each word x i with GloVe embeddings (Pennington et al., 2014), denoted by GloVe(x i ) . We further use the ELMo contextual embeddings (Peters et al., 2018), which processing the entire sentence x by a pretrained LSTM. The corresponding hidden state is used as the embedding representation of a word x i in its context. This is denoted by ELMo( x ) i . We use a two-layer bi-directional LSTM on the above two embeddings. The forward LSTM, for example, has the form h  Et = LSTM  E ([GloVe(x t ); ELMo( x ) t ] , h  Et  1 ) where the superscript E denotes the encoder. Likewise, the backward LSTM yields the representation h  Et . They are concatenated as h Et = [ h  E ; h  E ] . Here, we use BiLSTM for simplicity, following Sanh et al. (2019) and Huang et al. (2019). Other pretrained models, such as the Tranformer-based BERT (Devlin et al., 2019), may also be adopted. This, however, falls out of the scope of our paper, as we mainly focus on multi-label emotion classification. Empirical results on the GoEmotions dataset shows that, by properly addressing multi-label classification, our model outperforms a Transformer-based model (Table 2). Decoder. In Seq2Emo, an LSTM-based decoder is used to make sequential predictions on every candidate emotion. Suppose a predefined order of emotions is given, e.g., angry, joy, and sad. The decoder will perform a binary classification over these emotions in sequence. The order, in fact, does not affect our model much, as it is the same for all training samples and can be easily learned. In addition, we feed a learnable emotion embedding as input at each step of the decoder. This enhances the decoder by explicitly indicating which emotion is being predicted at a step. Different from a traditional Seq2Seq decoder, we do not feed previous predictions back as input, so as to avoid exposure bias. This also allows Seq2Emo to use a bi-directional LSTM as the decoder, which implicitly model the correlation among different emotions. where e j is the embedding for the j th emotion, and  h  Dj  1 is calculated by the attention mechanism in Luong et al. (2015). Here, the attention mechanism dynamically aligns source words when predicting the specific target emotion at a decoding step. Let   j,i be the attention probability of the j th decoder step over the i th encoder step, computed by s  j,i = ( h  Dj ) (cid:62) W  a h Ei (2)   j,i = exp( s  j,i ) (cid:80) Mi =1 exp( s  j,i ) (3) where M is the number of encoder steps, and s  j,i computes an unnormalized score for each pair of h  Dj and h Ei with a learnable parameter matrix W  a . Then, we compute an attention-weighted sum of encoder hidden states as the context vector c  j : c  j = M (cid:88) i =1   j,i h Ei (4) The context vector is concatenated with the LSTM hidden state as  h  Dj = [ c  j ; h  Dj ] . Likewise, we compute  h  D j for the backward decoder LSTM. They are further concatenated for predicting the emotion in question: P ( y j = 1 | x ) =  ( w (cid:62) j [ h  Dj ;  h  Dj ] + b j ) (5) where  is a sigmoid function; w j and b j are the parameters for predicting the j th emotion. Notice that w j and b j are different at decoding different steps, because we are predicting different emotions. This treatment is similar to the binary relevance approach (BR, Godbole and Sarawagi, 2004). Our Seq2Emo implicitly models the correlations among emotions through the decoder's bidirectional LSTM hidden states, which is more suited to multi-label classification than BR's individual predictions. Our Seq2Emo also differs from the classifier chain approach (CC, Read et al., 2009), which uses softmax to predict the next plausible emotion from all candidates. Thus, CC has to feed the previous predictions as input, and suffers from the exposure bias problem. By contrast, we predict the presence of all the emotions in sequence. Hence, feeding back previous predictions is not necessary, and this prevents the exposure bias. In this sense, our model combines the merits of both BR and CC. Datasets. We conduct experiments on two multi-labeled emotion datasets: SemEval'18 (Affect in Tweets: Task E-c, Mohammad et al., 2018) and GoEmotions (Demszky et al., 2020). Compared with GoEmotions, SemEval'18 has fewer emotion categories, and is smaller in size. Both datasets come with standard train-dev-test splits. Appendix A shows the statistics of these datasets. Metrics. Following Yang et al. (2018) and Mohammad et al. (2018), we use Jaccard Index (Rogers and Tanimoto, 1960), Hamming Loss (Schapire and Singer, 1999), Macroand Micro-averaged F1 scores (Chinchor, 1992) as the evaluation metrics. Among them, Jaccard, Macroand Micro-F1 are different ways of counting correctly predicted labels (the higher, the better); Hamming Loss (HL) counts the misclassifications (the lower, the better). Baselines. On SemEval'18, we compare our system with the top submissions from the SemEval-2018 competition and recent development. NTUA-SLP (Baziotis et al., 2018) uses large amount of external emotion-related data to pretrain an LSTM-based model. TCS Research's system (Meish-eri and Dey, 2018) uses the support vector machine with mannually engineered features: output from LSTM models, emotion lexicons (Mo-hammad and Kiritchenko, 2015), and SentiNeural (Radford et al., 2017). PlusEmo2Vec (Park et al., 2018) combines neural network models, which are pretrained by using emojis as labels (Felbo et al., 2017). Apart from the competition, Yu et al. (2018) propose DATN, which introduces sentiment information through dual-attention. These aforementioned systems are based on the BR approach. SGM (Yang et al., 2018), however, is a CC-based model for multi-label classification. We include it as a baseline by using its publicly released code. 2 Since GoEmotions dataset is fairly recent, we only include the results originally reported by Demszky et al. (2020). Settings. For the encoder, we set the two-layer bi-directional LSTM's dimension to 1200. Given the small number of emotions to embed, we set the dimension of decoder LSTM to 400. The GloVe embedding is 300 dimensional, and the ELMo embedding is 1024 dimensional. We use the Adam optimizer (Kingma and Ba, 2015), where the learning rate is set to 5e-4 initially and decayed with cosine annealing. The batch size is set to 16 for SemEval'18, and set to 32 for GoEmotions for effi-ciency concerns. We perform 5-fold cross-validation on the combined train-dev split for each experiment. Within each fold, we apply early stopping to prevent over-fitting and return the best model based on Jaccard accuracy for testing. We then merge the predicted results over the test set by majority voting. Additionally, we repeat each 5-fold experiment 5 times to further improve reduce noise. Overall performance. Table 1 presents the results on the SemEval'18 dataset. The proposed Seq2Emo outperforms the top submissions of the SemEval-2018 shared task in general. Compared with the median submission, Seq2Emo outperforms over 10% in the Jaccard accuracy. Admittedly, Seq2Emo performs slightly lower (but comparably) with NTUA-SLP and DATN, both introducing extra emotion/sentiment information through transfer learning. Our work, however, focuses on modeling the multi-label classification problem for emotion analysis and achieves high performance. While both NTUA-SLP and DATN are based on the BR approach, we implement additional baselines for fair comparison. In particular, we implement BR and BR-att variants, where the latter uses an attention mechanism when predicting the emotions, similar to our Seq2Emo. In the same spirit, we also implement a CC-based baseline, which is a Seq2Seq model predicting the next emotion among all candidates. For fair comparison, all of the BR, BR-att, and CC variants are trained with the same setting as our Seq2Emo. In this controlled setting, we observe that the proposed Seq2Emo consistently outperform BR, BR-att, and CC on the SemEval'18 dataset in all metrics. For the GoEmotions dataset, we show the results in Table 2. Since it is a very new dataset, we can only find previous reported results from Demszky et al. (2020). In addition, we include BR, BR-att, and CC for fair comparison. Results show that Seq2Emo outperforms other models on most of the metrics, except that Seq2Emo is worse than CC on Jaccard accuracy. This is understandable, as we have quite a few metrics with different datasets. It is worth noting that the model of Demszky et al. (2020) is based on BERT (Devlin et al., 2019). We replicate their approach to obtain all the evaluation metrics. We observe that our replication achieves a similar Macro-F1 to Demszky et al. (2020), and thus our replication is fair. The results show that our Seq2Emo achieves comparable or higher performance than the BERT-based model. We run one-sided t-tests to compare Seq2Emo with the best competing model that does not use additional data, shown in Tables 1 and 2. Results ver-# Model Jaccard  Micro F.  Macro F.  HL  1 BERT (Demszky et al., 2020)   46.00  2 BERT (our implementation)  53.06 58.49 46.23 0.0312 3 BR  52.76 58.21 45.38 0.0312 4 BR-att  53.35 58.53 45.11 0.0310 5 CC  55.61 58.38 43.92 0.0352 6 Seq2Emo (uni)  53.07 58.76 45.30 0.0306 7 Seq2Emo  53.79 59.57 47.28 0.0302 t-test p < 0 . ify that most of the comparisons are statistically significant (although some are more significant than others). The two experiments provide consistent evidence on the effectiveness of our Seq2Emo. Seq2Emo with an uni-directional decoder. One of the virtues of Seq2Emo is that it can use a bi-directional LSTM decoder. To show its effectiveness, we perform experiments on Seq2Emo with an uni-directional decoder, denoted as Seq2Emo (uni). We show the results in Tables 1 and 2 for SemEval'18 and GoEmotions datasets, respectively. We first observe that Seq2Emo performs better than Seq2Emo (uni), which in turn is better than BR-att that predicts emotions individually. This confirms that our Seq2Emo is able to implicitly model the correlation of different emotions, and that a bi-directional decoder is better than a uni-directional one. Order of emotions. Both Seq2Emo and the classifier chain (CC) predict emotions sequentially. The difference is that our Seq2Emo predicts the presence (or not) of an emotion in a predefined order. CC predicts the next salient emotion au-toregressively, it learns the emotion order from the training data. We try different orders, including the original order in the dataset and the ascend-ing/descending order based on emotion frequency. We also try an order where the emotion frequency first increases and then decreases (concave-down), and vice versa (concave-up). We perform experiments on SemEval'18 and report the Jaccard accuracy and the standard deviations in Table 3. The results show that Seq2Emo is the least affected by the order of the emotions, whereas the performance of CC varies largely. This verifies that the emotion order does not affect Seq2Emo much as it can be easily learned. CC is more sensitive to emotion order and has a larger variance, as it suffers from the exposure bias problem. pendix B. Results show that our Seq2Emo can attend to relevant words when predicting the emotion of interest. In this work, we propose Seq2Emo for multi-label emotion classification. Our approach implicitly models the relationship of different emotions in its bi-directional decoder, and is shown to be better than an individual binary relevance (BR) classifier. Our model does not suffer from the exposure bias problem and also outperforms the classifier chain (CC). In general, we achieve state-of-the-art performance for multi-emotion classification on the SemEval'18 and GoEmotions datasets (without using additional emotion labels). We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC) under grant Nos. RGPIN-2020-04465 and RGPIN-2020-04440. Chenyang Huang is supported by the Borealis AI Graduate Fellowship Program. Lili Mou and Osmar Zaane are supported by the Amii Fellow Program and the Canada CIFAR AI Chair Program. This research is also supported in part by Compute Canada ( www.computecanada.ca ).
abstain objective result abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain objective method method method method abstain abstain other other other other other other other other other other method other other other abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain other method method abstain method abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain method method abstain method method abstain method abstain method method method method abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result result result result result abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result objective result result result other other other other other