In this paper, we address three challenges in utterance-level emotion recognition in dialogue systems: (1) the same word can deliver different emotions in different contexts; (2) some emotions are rarely seen in general dialogues; (3) long-range contextual information is hard to be effectively captured. We therefore propose a hierarchical Gated Recurrent Unit (HiGRU) framework with a lower-level GRU to model the word-level inputs and an upper-level GRU to capture the contexts of utterance-level embeddings. Moreover, we promote the framework to two variants, HiGRU with individual features fusion (HiGRU-f) and HiGRU with self-attention and features fusion (HiGRU-sf), so that the word/utterance-level individual inputs and the long-range contextual information can be sufficiently utilized. Experiments on three dialogue emotion datasets, IEMOCAP, Friends, and EmotionPush demonstrate that our proposed HiGRU models attain at least 8.7%, 7.5%, 6.0% improvement over the state-of-the-art methods on each dataset, respectively. Particularly, by utilizing only the textual feature in IEMOCAP, our HiGRU models gain at least 3.8% improvement over the state-of-the-art conversational memory network (CMN) with the trimodal features of text, video, and audio. Emotion recognition is a significant artificial intelligence research topic due to the promising potential of developing empathetic machines for people. Emotion is a universal phenomena across different cultures and mainly consists of six basic types: anger, disgust, fear, happiness, sadness, and surprise (Ekman, 1971, 1992). In this paper, we focus on textual dialogue sys-tems because textual feature dominates the performance over audio and video features (Poria et al., Role Utterance Emotion Rachel Oh okay, I'll fix that to. What's her email address? Neutral Ross Rachel! Anger Rachel All right, I promise. I'll fix this. I swear. I'll-I'llI'll-I'll talk to her. Non-neutral Ross Okay! Anger Rachel Okay. Neutral Nurse This room's available. Neutral Rachel Okay! Joy Rachel Okay wait! Non-neutral Rachel You listen to me! Anger Figure 1: The word okay exhibits different emotions in the American television sitcom, Friends. 2015, 2017). In utterance-level emotion recognition, an utterance (Olson, 1977) is a unit of speech bounded by breathes or pauses and its goal is to tag each utterance in a dialogue with the indicated emotion. In this task, we address three challenges: First, the same word can deliver different emotions in different contexts. For example, in Figure 1, the word okay can deliver three different emotions, anger, neutral, and joy, respectively. Strong emotions like joy and anger may be indicated by the symbols ! or ? along the word. To identify a speaker's emotion precisely, we need to explore the dialogue context sufficiently. Second, some emotions are rarely seen in general dialogues. For example, people are usually calm and present a neutral emotion while only in some particular situations, they express strong emotions, like anger or fear. Thus we need to be sensitive to the minority emotions while relieving the effect of the majority emotions. Third, the long-range contextual information is hard to be effectively captured in an ut-terance/dialogue, especially when the length of an utterance/dialogue in the testing set is longer than those in the training set. To tackle these challenges, we propose a hierarchical Gated Recurrent Unit (HiGRU) framework for the utterance-level emotion recognition in dialogue systems. More specifically, HiGRU is composed by two levels of bidirectional GRUs, a lower-level GRU to model the word sequences of each utterance to produce individual utterance embeddings, and an upper-level GRU to capture the sequential and contextual relationship of utterances. We further promote the proposed HiGRU to two variants, HiGRU with individual features fusion (HiGRU-f), and HiGRU with self-attention and features fusion (HiGRU-sf). In HiGRU-f, the individual inputs, i.e., the word embeddings in the lower-level GRU and the individual utterance embeddings in the upper-level GRU, are concatenated with the hidden states to generate the contextual word/utterance embeddings, respectively. In HiGRU-sf, a self-attention layer is placed on the hidden states from the GRU to learn long-range contextual embeddings, which are concatenated with the original individual embeddings and the hidden states to generate the contextual word/utterance embeddings. Finally, the contextual utterance embedding is sent to a fully-connected (FC) layer to determine the corresponding emotion. To alleviate the effect of data imbalance issue, we follow (Khosla, 2018) to train our models by minimizing a weighted categorical cross-entropy. We summarize our contributions as follows:  We propose a HiGRU framework to better learn both the individual utterance embeddings and the contextual information of utterances, so as to recognize the emotions more precisely. We propose two progressive HiGRU variants, HiGRU-f and HiGRU-sf, to sufficiently incorporate the individual word/utterance-level information and the long-range contextual information respectively. We conduct extensive experiments on three textual dialogue emotion datasets, IEMOCAP, Friends, and EmotionPush. The results demonstrate that our proposed HiGRU models achieve at least 8.7%, 7.5%, 6.0% improvement over state-of-the-art methods on each dataset, respectively. Particularly, by utilizing only the textual feature in IEMOCAP, our proposed HiGRU models gain at least 3.8% improvement over the existing best model, conversational memory network (CMN) with not only the text feature, but also the visual, and audio features. Text-based emotion recognition is a long-standing research topic (Wilson et al., 2004; Yang et al., 2007; Medhat et al., 2014). Nowadays, deep learning technologies have become dominant methods due to the outstanding performance. Some prominent models include recursive autoencoders (RAEs) (Socher et al., 2011), convolutional neural networks (CNNs) (Kim, 2014), and recurrent neural networks (RNNs) (Abdul-Mageed and Ungar, 2017). However, these models treat texts independently thus cannot capture the inter-dependence of utterances in dialogues (Kim, 2014; Lai et al., 2015; Grave et al., 2017; Chen et al., 2016; Yang et al., 2016). To exploit the contextual information of utterances, researchers mainly explore in two directions: (1) extracting contextual information among utterances, or (2) enriching the information embedded in the representations of words and utterances. Contextual Information Extraction. The RNN architecture is a standard way to capture the sequential relationship of data. Poria et al. propose a bidirectional contextual long short-term memory (LSTM) network, termed bcLSTM, to model the context of textual features extracted by CNNs. Hazarika et al. improve bcLSTM by a conversational memory network (CMN) to capture the self and inter-speaker emotional influence, where GRU is utilized to model the self-influence and the attention mechanism is employed to excavate the inter-speaker emotional influence. Though CMN is reported to attain better performance than bcLSTM on IEMOCAP (Hazarika et al., 2018), the memory network is too complicated for small-size dialogue datasets. Representation Enrichment. Multimodal features have been utilized to enrich the representation of utterances (Poria et al., 2015, 2017). Previous work indicate that textual features dominate the performance of recognizing emotions in contrast to visual or audio features (Poria et al., 2015, 2017). Recently, the textual features are mainly extracted by CNNs to learn individual utterance embeddings (Poria et al., 2015, 2017; Zahiri and Choi, 2018; Hazarika et al., 2018). However, CNNs do not capture the contextual information within each utterance well. On the other hand, hierarchical RNNs have been proposed and demonstrated good performance in GRU  "  #  $  % GRU GRUGRUGRU  %  "  #  $  %  "  #  $ ( ) ) Fusion  + ( ) ) ( ) Max-pooling Individual Utterance Embedding  . / GRUGRU  . /  . /     Attention  %0   "  #  $  % ( )  + ( ) Fully-connected Contextual Utterance Embedding  1 2 Attention Softmax  %3  %  % GRU GRU GRUGRUGRU  %  "  #  $  %  "  #  $ GRUGRU  1 2  1 2     %0   %3  %  % Fusion   Figure 2: The architecture of our proposed HiGRU-sf. conventional text classification task (Tang et al., 2015), dialogue act classification (Liu et al., 2017; Kumar et al., 2018), and speaker change detection (Meng et al., 2017). But they are not well explored in the task of utterance-level emotion recognition in dialogue systems. Definition 1 ( Utterance-level Emotion Recogni-tion). Suppose we are given a set of dialogues, D = { D i } Li =1 , where L is the number of dialogues. In each dialogue, D i = { ( u j , s j , c j ) } N i j =1 , is a sequence of N i utterances, where the utterance u j is spoken by the speaker s j  S with a certain emotion c j  C . All speakers compose the set S and the set C consists of all emotions, such as anger, joy, sadness, and neutral. Our goal is to train a model M to tag each new utterance with an emotion label from C as accurately as possible. To solve this task, we propose a hierarchical Gated Recurrent Units (HiGRU) framework and extend two progressive variants, HiGRU with individual features fusion (HiGRU-f) and HiGRU with self-attention and features fusion (HiGRU-sf) (il-lustrated in Figure 2). The vanilla HiGRU consists of two-level GRUs: the lower-level bidirectional GRU is to learn the individual utterance embedding by modeling the word sequence within an utterance and the upper-level bidirectional GRU is to learn the contextual utterance embedding by modeling the utterance sequence within a dialogue. Individual Utterance Embedding. For the j th utterance in D i , u j = { w k } M j k =1 , where M j is the number of words in the utterance u j . The corresponding sequence of individual word embeddings { e ( w k ) } M j k =1 are fed into the lower-level bidirectional GRU (Cho et al., 2014) to learn the individual utterance embedding in two opposite directions:  h k = GRU( e ( w k ) ,  h k  1 ) , (1)  h k = GRU( e ( w k ) ,  h k +1 ) . The two hidden states  h k and  h k are concatenated into hs = [  h k ;  h k ] to produce the contextual word embedding for w k via the tanh activation function on a linear transformation: where W w  R d 1  2 d 1 and b w  R d 1 are the model parameters, d 0 and d 1 are the dimensions of word embeddings and the hidden states of the lower-level GRU, respectively. Contextual Utterance Embedding. For the i th dialogue, D i = { ( u j , s j , c j ) } N i j =1 , the learned individual utterance embeddings, { e ( u j ) } N i j =1 , are fed into the upper-level bidirectional GRU to capture the sequential and contextual relationship of utterances in a dialogue:  H j = GRU( e ( u j ) ,  H j  1 ) , (5)  H j = GRU( e ( u j ) ,  H j +1 ) . Here, the hidden states of the upper-level GRU are represented by H j  R d 2 , to distinguish from those learned in the lower-level GRU denoted by h k . Accordingly, we can obtain the contextual utterance embedding by e c ( u j ) = tanh( W u  Hs + b u ) , (7) where Hs = [  H j ;  H j ] , W u  R d 2  2 d 2 and b u  R d 2 are the model parameters, d 2 is the dimension of the hidden states in the upper-level GRU. Since the emotions are recognized at utterance-level, the learned contextual utterance embedding e c ( u j ) is directly fed to a FC layer followed by a softmax function to determine the corresponding emotion label:  y j = softmax ( W fc  e c ( u j ) + b fc ) , (8) where  y j is the predicted vector over all emotions, and W fc  R |C| d 2 , b fc  R |C| . The vanilla HiGRU contains two main issues: (1) the individual word/utterance embeddings are diluted with the stacking of layers; (2) the upper-level GRU tends to gather more contextual information from the majority emotions, which deteriorates the overall model performance. To resolve these two problems, we propose to fuse individual word/utterance embeddings with the hidden states from GRUs so as to strengthen the information of each word/utterance in its contextual embedding. This variant is named as HiGRU-f, representing HiGRU with individual features fusion. Hence, the lower-level GRU can maintain individual word embeddings and the upper-level GRU can relieve the effect of majority emotions and attain a more precise utterance representation for different emotions. Specifically,    ,1    ,2    ,3 . . .    ,   1,1  1,2  1,3 . . .  1,   2,1  2,2  2,3 . . .  2,   3,1  3,2  3,3 . . .  3,  ... ... ... ... . . .  1  2  3 . . .    Copy  Softmax Matmul  1  2  3 ...     1  2  3 ...     Figure 3: Self-attention over the forward hidden states of GRU. the contextual embeddings are updated as: e c ( w k ) = tanh( W w  hs f + b w ) , (9) e c ( u j ) = tanh( W u  Hs f + b u ) , (10) where W w  R d 1  ( d 0 +2 d 1 ) , W u  R d 2  ( d 1 +2 d 2 ) , hs f = [  h k ; e ( w k );  h k ] , and Hs f = [  H j ; e ( u j );  H j ] . Another challenging issue is to extract the contextual information of long sequences, especially the sequences in the testing set that are longer than those in the training set (Bahdanau et al., 2014). To fully utilize the global contextual information, we place a self-attention layer upon the hidden states of HiGRU and fuse the attention outputs with the individual word/utterance embeddings and the hidden states to learn the contextual word/utterance embeddings. Hence, this variant is termed HiGRU-sf, representing HiGRU with self-attention and features fusion. Particularly, we apply self-attention upon the forward and backward hidden states separately to produce the left context embedding, h lk ( H lj ), and the right context embedding, h rk ( H rj ), respectively. This allows us to gather the unique global contextual information at the current step in two opposite directions and yield the corresponding contextual embeddings computed as follows: e c ( w k ) = tanh( W w  hs sf + b w ) , (11) e c ( u j ) = tanh( W u  Hs sf + b u ) , (12) where W w  R d 1  ( d 0 +4 d 1 ) , W u  R d 2  ( d 1 +4 d 2 ) , hs sf = [ h lk ;  h k ; e ( w k );  h k ; h rk ] , and Hs sf = [ H lj ;  H j ; e ( u j );  H j ; H rj ] . Self-Attention (SA). The self-attention mechanism is an effective non-recurrent architecture to compute the relation between one input to all other inputs and has been successfully applied in various natural language processing applications such as reading comprehension (Hu et al., 2018), and neural machine translation (Vaswani et al., 2017). Figure 3 shows the dot-product SA over the forward hidden states of GRU to learn the left context h lk . Each element in the attention matrix is computed by f (  h k ,  h p ) = (cid:40)  h k (cid:62)  h p , if k, p  M j ,  , otherwise . An attention mask is then applied to waive the inner attention between the sequence inputs and paddings. At each step, the corresponding left context h lk is then computed by the weighted sum of all the forward hidden states: h lk = M j (cid:88) p =1 a kp  h p , a kp = exp( f (  h k ,  h p )) (cid:80) Mj p (cid:48) =1 exp (cid:16) f (  h k ,  h p (cid:48) ) (cid:17) , (14) where a kp is the weight of  h p to be included in h lk . The right context h rk can be computed similarly. Following (Khosla, 2018) which attains the best performance in the EmotionX shared task (Hsu and Ku, 2018), we minimize a weighted categorical cross-entropy on each utterance of all dialogues to optimize the model parameters: where y j is the original one-hot vector of the emotion labels, and y cj and  y cj are the elements of y j and  y j corresponding to the class c . Similar to (Khosla, 2018), we assign the loss weight  ( c j ) inversely proportional to the number of training utterances in the class c j , denoted by I c , i.e., assigning larger loss weights for the minority classes to relieve the data imbalance issue. The difference is that we add a constant  to adjust the smoothness of the distribution. Then, we have: 1  ( c ) = I c (cid:80) |C| c (cid:48) =1 I c (cid:48) . We conduct systematical experiments to demonstrate the advantages of our proposed HiGRU models. The experiments are carried out on three textual dialogue emotion datasets (see the statistics in Table 1): IEMOCAP 1 . It contains approximately 12 hours of audiovisual data, including video, speech, motion capture of face, text transcriptions. Following (Poria et al., 2017; Hazarika et al., 2018): (1) We apply the first four sessions for training and the last session for test; (2) The validation set is extracted from the shuffled training set with the ratio of 80:20; (3) We only evaluate the performance on four emotions: anger, happiness, sadness, neutral, and remove the rest utterances. Friends 2 . The dataset is annotated from the Friends TV Scripts (Hsu and Ku, 2018), where each dialogue in the dataset consists of a scene of multiple speakers. Totally, there are 1,000 dialogues, which are split into 720, 80, and 200 dialogues for training, validation, and testing, respectively. Each utterance in a dialogue is labeled by one of the eight emotions: anger, joy, sadness, neutral, surprise, disgust, fear, and non-neutral. EmotionPush 3 . The dataset consists of private conversations between friends on the Facebook messenger collected by an App called EmotionPush, which is released for the EmotionX shared task (Hsu and Ku, 2018). Totally, there are 1,000 dialogues, which are split into 720, 80, 200 dialogue for training, validation, and testing, respectively. All the utterances are categorized into one of the eight emotions as in the Friends dataset. Following the setup of (Hsu and Ku, 2018), in Friends and EmotionPush, we only evaluate the model performance on four emotions: anger, joy, sadness, and neutral, and we exclude the contribution of the rest emotion classes during training by setting their loss weights to zero. Data Preprocessing. We preprocess the datasets by the following steps: (1) The utterances are split into tokens with each word being made into the lowercase; (2) All non-alphanumerics except ? and ! are removed because these two symbols usually exhibit strong emotions, such as surprise, 1 https://sail.usc.edu/iemocap/ 2 http://doraemon.iis.sinica.edu.tw/ emotionlines 3 http://doraemon.iis.sinica.edu.tw/ emotionlines Dataset #Dialogue (#Utterance) #Emotion Train Val Test Ang Hap/Joy Sad Neu Others IEMOCAP 96 (3,569) 24 (721) 31 (1,208) 1,090 1,627 1,077 1,704 0 Friends 720 (10,561) 80 (1,178) 200 (2,764) 759 1,710 498 6,530 5,006 EmotionPush 720 (10,733) 80 (1,202) 200 (2,807) 140 2,100 514 9,855 2,133 Table 1: Statistics of the textual dialogue datasets. joy and anger; (3) We build a dictionary based on the words and symbols extracted, and follow (Po-ria et al., 2017) to represent the tokens by the publicly available 300-dimensional word2vec 4 vectors trained on 100 billion words from Google News. The tokens not included in the word2vec dictionary are initialized by randomly-generated vectors. To conduct fair comparison, we adopt two metrics as (Hsu and Ku, 2018), the weighted accuracy (WA) and unweighted accuracy (UWA): where p c is the percentage of the class c in the testing testing set, and a c is the corresponding accuracy. Generally, recognizing strong emotions may provide more value than detecting the neutral emotion (Hsu and Ku, 2018). Thus, in Friends and EmotionPush, UWA is a more favorite evaluation metric because WA is heavily compromised with the large proportion of the neutral emotion. Our proposed vanilla HiGRU, HiGRU-f, and HiGRU-sf 5 are compared with the following state-of-the-art baselines: bcLSTM (Poria et al., 2017) : a bidirectional contextual LSTM with multimodal features extracted by CNNs; CMN (Hazarika et al., 2018) : a conversational memory network with multimodal features extracted by CNNs; SA-BiLSTM (Luo et al., 2018) : a self-attentive bidirectional LSTM model, a neat model achieving the second place of EmotionX Challenge (Hsu and Ku, 2018); CNN-DCNN (Khosla, 2018) : a convolutional-deconvolutional autoencoder with more handmade features, the winner of EmotionX Challenge (Hsu and Ku, 2018); bcLSTM  and bcGRU : our implemented bcLSTM and bcGRU with the weighted loss on the textual feature extracted from CNNs. All our implementations are coded on the Pytorch framework. To prevent the models fitting the order of data, we randomly shuffle the training set at the beginning of every epoch. Parameters. For bcLSTM  and bcGRU , the CNN layer follows the setup of (Kim, 2014), i.e., consisting of the kernels of 3, 4, and 5 with 100 feature maps each. The convolution results of each kernel are fed to a max-over-time pooling operation. The dimension of the hidden states of the upper-level bidirectional LSTM or GRU is set to 300. For HiGRU, HiGRU-f, and HiGRU-sf, the dimensions of hidden states are set to 300 for both levels. The final FC layer contains two sub-layers with 100 neurons each. Training. We adopt Adam (Kingma and Ba, 2014) as the optimizer and set an initial learning rate, 1  10  4 for IEMOCAP and 2 . 5  10  4 for Friends and EmotionPush, respectively. An annealing strategy is utilized by decaying the learning rate by half every 20 epochs. Early stopping with a patience of 10 is adopted to terminate training based on the accuracy of the validation set. Specifically, following the best models on each dataset, the parameters are tuned to optimize WA on the validation set of IEMOCAP and to optimize UWA on the validation set of Friends and EmotionPush, respectively. Gradient clipping with a norm of 5 is applied to model parameters. To prevent overfitting, dropout with a rate of 0.5 is applied after the contextual word/utterance embeddings, and the FC layer. Loss weights. For Friends and EmotionPush, as mentioned in Section 4.1, the loss weights are set to zero except the four considered emotions, to ignore the others during training. Besides, the power rate  of loss weights is tested from 0 to 1.5 with Model (Feat) Ang Hap Sad Neu WA UWA bcLSTM (T) 76.07 78.97 76.23 67.44 73.6 74.6 (T+V+A) 77.98 79.31 78.30 69.92 76.1 76.3 CMN (T) --74.1 (T+V+A) 89.88 81.75 77.73 67.32 77.6 79.1 bcLSTM  (T) 75.29 79.40 78.07 76.53 77.7 (1.1) 77.3 (1.4) bcGRU (T) 77.20 80.99 76.26 72.50 76.9 (1.6) 76.7 (1.3) HiGRU (T) 75.41 91.64 79.79 70.74 80.6 (0.5) 79.4 (0.5) HiGRU-f (T) 76.69 88.91 80.25 75.92 81.5 (0.7) 80.4 (0.5) HiGRU-sf (T) 74.78 89.65 80.50 77.58 82.1 (0.4) 80.6 (0.2) Table 2: Experimental results on IEMOCAP. Table 2 and Table 3 report the average results of 10 trials each on the three datasets, where the standard deviations of WA and UWA are recorded by the subscripts in round brackets. The results of bcLSTM, CMN, SA-BiLSTM, and CNN-DCNN are copied directly from the original papers for a fair comparison because we follow the same con-figuration for the corresponding datasets. From the results, we have the following observations: (1) Baselines. Our implemented bcLSTM  and bcGRU, attain comparable performance with the state-of-the-art methods on all three datasets. From the results on IEMOCAP in Table 2, we observe that: (a) By utilizing the textual feature only, bcGRU outperforms bcLSTM and CMN trained on the textual feature significantly, attaining +3.3 and +2.8 gain in terms of WA, respectively. bcLSTM  performs better than bcGRU, and even beats bcLSTM and CMN with the trimodal features in terms of WA. In terms of UWA, CMN performs better than bcLSTM  only when it is equipped with multimodal features. (b) By examining the detailed accuracy in each emotion, bcLSTM  and bcGRU with the textual feature attain much higher accuracy on the neutral emotion than bcLSTM with the only textual feature while maintaining good performance on the other three emotions. The results show that the weighted loss function benefits the training of models. bcGRU trained on the same dataset (F+E) of CNN-DCNN perform better than CNN-DCNN on EmotionPush while attaining comparable performance with CNN-DCNN on Friends. The results show that by utilizing the contextual information with the weighted loss function, bcLSTM  and bcGRU can beat the state-of-the-art method. From Table 2, we observe that: (a) CMN with the trimodal features attains the best performance on the anger emotion while our vanilla HiGRU achieves the best performance on the happiness emotion and gains further improvement on sadness and neutral emotions over CMN. Overall, the vanilla HiGRU achieves at least 8.7% and 3.8% improvement over CMN with the textual feature and the trimodal features in terms of WA, respectively. The results, including those of bcLSTM  and bcGRU, indicate that GRU learns better representations of utterances than CNN in this task. (b) The two variants, HiGRU-f and HiGRU-sf, can further attain +0.9 and +1.5 improvement over HiGRU in terms of WA and +1.0 and +1.2 improvement over HiGRU in terms of UWA, respectively. The results demonstrate that the included individual word/utterance-level features and long-range contextual information in HiGRU-f and HiGRU-sf, are indeed capable of boosting the performance of the vanilla HiGRU. From Table 3, we can see that: (a) In terms of UWA, HiGRU trained and tested on individual sets of Friends and EmotionPush gains at least 7.5% and 6.0% improvement over CNN-DCNN, respectively. Overall, our proposed HiGRU achieves well-balanced performance for the four tested emotions, especially attaining significant better performance on the minority emotions of anger and sadness. (b) Moreover, HiGRU-f and HiGRU-sf further improve HiGRU +1.2 accuracy and +1.7 accuracy on Friends and +0.6 accuracy and +1.8 accuracy on EmotionPush in terms of UWA, respectively. The results again demonstrate the superior power of HiGRU-f and HiGRU-sf. (3) Mixing Training Sets. By examining the results from the last ten rows in Table 3, we conclude that it does not necessarily improve the performance by mixing the two sets of training data. Though the best performance of SA-BiLSTM Model Train Friends (F) EmotionPush (E) Ang Joy Sad Neu WA UWA Ang Joy Sad Neu WA UWA SA-BiLSTM F+E 49.1 68.8 30.6 90.1 -59.6 24.3 70.5 31.0 94.2 -55.0 CNN-DCNN F+E 55.3 71.1 55.3 68.3 -62.5 45.9 76.0 51.7 76.3 -62.5 bcLSTM  F(E) 64.7 69.6 48.0 75.6 72.4 (4.2) 64.4 (1.6) 32.9 69.9 47.1 78.0 74.7 (4.4) 57.0 (2.1) bcGRU F(E) 69.5 65.4 52.9 74.7 71.7 (4.7) 65.6 (1.2) 33.7 71.1 57.2 76.1 73.9 (2.9) 59.5 (1.8) bcLSTM  F+E 54.5 75.6 43.4 73.0 70.5 (4.5) 61.6 (1.6) 52.4 79.1 54.7 73.3 73.4 (3.8) 64.9 (2.1) bcGRU F+E 59.0 78.6 42.3 71.4 70.2 (5.1) 62.8 (1.4) 49.4 74.8 61.9 72.4 72.1 (4.3) 64.6 (1.8) HiGRU F(E) 66.9 73.0 51.8 77.2 74.4 (1.7) 67.2 (0.6) 55.6 78.1 57.4 73.8 73.8 (2.0) 66.3 (1.7) HiGRU-f F(E) 69.1 72.1 60.4 72.1 71.3 (2.9) 68.4 (1.0) 55.9 78.9 60.4 72.4 73.0 (2.2) 66.9 (1.2) HiGRU-sf F(E) 70.7 70.9 57.7 76.2 74.0 (1.4) 68.9 (1.5) 57.5 78.4 64.1 72.5 73.0 (1.6) 68.1 (1.2) HiGRU F+E 55.4 81.2 51.4 64.4 65.8 (4.2) 63.1 (1.5) 50.8 76.9 69.0 75.7 75.3 (1.7) 68.1 (1.2) HiGRU-f F+E 54.9 78.3 55.5 68.7 68.5 (3.0) 64.3 (1.2) 58.3 79.1 69.6 70.0 71.5 (2.5) 69.2 (0.9) HiGRU-sf F+E 56.8 81.4 52.2 68.7 69.0 (2.0) 64.8 (1.3) 57.8 79.3 66.3 77.4 77.1 (1.0) 70.2 (1.1) Table 3: Experimental results on Friends and EmotionPush. and CNN-DCNN is obtained by training on the mixed dataset, the testing results show that our implemented bcLSTM  , bcGRU and our proposed HiGRU models can attain better performance on EmotionPush but yield worse performance on Friends in terms of UWA. By examining the detailed emotions, we speculate that: EmotionPush is a highly imbalanced dataset with over 60% of utterances in the neutral emotion. Introducing EmotionPush into a more balanced dataset, Friends, is equivalent to down-sampling the minority emotions in Friends. This hurts the performance on the minority emotions, anger and sadness. Meanwhile, introducing Friends into EmotionPush corresponds to up-sampling the minority emotions in EmotionPush. The performance of the sadness emotion is significantly boosted and that on the anger emotion is at least unaffected. Model Size. We study how the scale of the utterance encoder affects the performance of our proposed models, especially when our models contain a similar number of parameters as the baseline, say bcGRU. Such a fair condition can be made between our HiGRU-sf and bcGRU if we set d 1 to 150. From the testing results on Friends in Table 4, we can observe that: (1) Under the fair condition, the performance of our HiGRU-sf is not degraded compared to that when d 1 = 300 . HiGRU-sf still outperforms bcGRU by a significant margin. (2) Overall, no matter d 1 is larger or smaller than 150, HiGRU-sf maintains consistently good performance and the difference between HiGRU-sf and HiGRU-f or HiGRU keeps noticeable. These results further demonstrate the superiority of our proposed models over the baseline bcGRU and the motivation of developing the two variants based on the vanilla HiGRU. Successful Cases. We investigate three scenes related to the word okay that expresses three distinct emotions. The first two scenes come from the testing set of Friends and the third one from that of IEMOCAP. We report the predictions made by bcGUR and our HiGRU-sf, respectively, in Table 5. In Scene-1 , okay with period usually exhibits little emotion and both bcGRU and HiGRU-sf correctly classify it as Neu. In Scene-2 , okay with ! expresses strong emotion. However, bcGRU misclassifies it to Ang while HiGRU-sf successfully recognizes it as Joy. Actually, the mistake can be traced back to the first utterance of this scene which is also misclassified as Ang. This indicates that bcGRU tends to capture the wrong atmosphere within the dialogue. As for Scene-3 , okay with period now indicates Sad and is correctly recognized by HiGRU-sf but misclassified as Neu by bcGRU. Note that HiGRU-sf also classifies the third utterance in Scene-3 as Sad which seems to be conflicting Role Utterance Truth bcGRU HiGRU-sf Scene-1 Phoebe Okay. to the ground truth. In fact, our HiGRU-sf captures the blues of this parting situation, where the true label Hap may not be that suitable. These results show that our HiGRU-sf learns from both each utterance and the context, and can make correct predictions of the emotion of each utterance. Failed Cases. At last, we show some examples that both bcGRU and our HiGRU-sf fail in recognizing the right emotions in Table 6, i.e., Scene-4 from Friends and Scene-5 from EmotionPush. In Scene-4 , both bcGRU and HiGRU-sf make wrong predictions for the fifth and the sixth utterances. It should be good news that Ross has his paper published and Rachel is glad to see related reports about it. However, the transcripts do not reveal very strong emotions compared to what the characters might act in the TV show. This kind of scenes may be addressed by incorporating some other features like audio and video. As for Scene-5 , the third and the fifth utterances are classified into wrong emotions. Notice that the emotions indicated from the two utterances are very subtle even for humans. The Speaker-2 did not plan to get up today, but Speaker-1 kept him/her up and it ended up with a really lax day. So, the Speaker-2 feels joyful now. This indicates that even taking into the context into account, the models' capability of understanding subtle emotions is still limited and more exploration is required. We propose a hierarchical Gated Recurrent Unit (HiGRU) framework to tackle the utterance-level emotion recognition in dialogue systems, where the individual utterance embeddings are learned by the lower-level GRU and the contexts of utterances are captured by the upper-level GRU. We promote the HiGRU framework to two variants, HiGRU-f, and HiGRU-sf, and effectively capture the word/utterance-level inputs and the long-range contextual information, respectively. Experimental results demonstrate that our proposed HiGRU models can well handle the data imbalance issue and sufficiently capture the available text information, yielding significant performance boosting on all three tested datasets. In the future, we plan to explore semi-supervised learning methods to address the problem of data scarcity in this task. This work is supported by the Research Grants Council of the Hong Kong Special Administrative Region, China (No. CUHK 14208815 and No. CUHK 14210717 of the General Research Fund, and Project No. UGC/IDS14/16), and Meitu (No. 7010445). We thank the three anonymous reviewers for the insightful suggestions on various aspects of this work.
result objective method objective method abstain abstain method abstain objective abstain abstain objective abstain abstain abstain abstain objective abstain objective abstain abstain abstain method objective objective method objective objective other other other other other other other other other other other other other other other other other other abstain method abstain abstain method method abstain abstain abstain abstain other abstain abstain abstain other abstain other other abstain abstain abstain abstain other other abstain method abstain method method abstain abstain abstain abstain abstain other abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain result result abstain result abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain result abstain abstain abstain abstain abstain result other abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain objective objective other other