The emotional state of a speaker can be influenced by many different factors in dialogues, such as dialogue scene, dialogue topic, and interlocutor stimulus. The currently available data resources to support such multimodal affective analysis in dialogues are however limited in scale and diversity. In this work, we propose a M ulti-modal M ulti-scene M ulti-label E motional D ialogue dataset, M 3 ED , which contains 990 dyadic emotional dialogues from 56 different TV series, a total of 9,082 turns and 24,449 utterances. M 3 ED is annotated with 7 emotion categories (happy, surprise, sad, disgust, anger, fear, and neutral) at utterance level, and encompasses acoustic, visual, and textual modalities. To the best of our knowledge, M 3 ED is the first multimodal emotional dialogue dataset in Chinese. It is valuable for cross-culture emotion analysis and recognition. We apply several state-of-the-art methods on the M 3 ED dataset to verify the validity and quality of the dataset. We also propose a general Multimodal Dialogue-aware Interaction framework, MDI, to model the dialogue context for emotion recognition, which achieves comparable performance to the state-of-the-art methods on the M 3 ED. The full dataset and codes are available 1 . Emotion Recognition in Conversation (ERC) aims to automatically identify and track the emotional status of speakers during a dialogue (Poria et al., 2019b). It is a crucial component to improve natural human-computer interactions and has a wide range of applications in interaction scenarios, including call-center dialogue systems (Danieli et al., 2015), conversational agents (Fragopana-gos and Taylor, 2005) and mental health diagnoses (Ringeval et al., 2018), etc. Different from traditional multimodal emotion recognition on isolated * Corresponding Author 1 https://github.com/AIM3-RUC/RUCM3ED (Sad) !"#$%&'( Xiaoan, I am sorry (Surprise) utterances, multimodal ERC is a more challenging problem, because there are many influencing factors that affect the speakers' emotional state in a dialogue, including the dialogue context from multi-modalities, the scene, the topic, and even the personality of subjects, etc. (Poria et al., 2019b; Scherer, 2005; Koval et al., 2015). It has been proved in recent works (Majumder et al., 2019; Ghosal et al., 2019; Hu et al., 2021; Shen et al., 2020) that contextual information plays an important role in ERC tasks and brings significant improvements over baselines that only consider isolated utterances. DialogueRNN (Majumder et al., 2019) uses recurrent networks to model global and speaker-specific temporal-context information. DialogueGCN (Ghosal et al., 2019) and MMGCN (Hu et al., 2021) use graph-based networks to capture conversational dependencies between utterances in dialogues. DialogXL (Shen et al., 2020) applies a strong pre-trained language model XLNet (Yang et al., 2019) to ERC and proposes a dialog-aware self-attention method for modeling the context information. The IEMOCAP (Busso et al., 2008) and MELD (Poria et al., 2019a) are two multimodal emotional dialogue benchmark datasets, which are widely used in the above-mentioned works and promote research in the affective computing field. However, both of them are limited in size and diversity. The videos in MELD are collected only from the Friends TV series, and the videos in IEMOCAP are recorded in laboratory environments from ten actors performing scripted and spontaneous dialogues. These limitations not only affect the investigation of generalization and robustness of the models, but also limit the exploration of other important influencing factors in dialogues, such as dialogue scene, dialogue topic, emotional influence from interlocutors, and so on. In this work, we construct a large-scale Multimodal Multi-scene and Multi-label Emotional Dialogue dataset, M 3 ED , which consists of 990 emotional dyadic dialogue video clips from 56 different TV series (about 500 episodes), ensuring that there are various dialogue scenes and topics. We also consider the blended annotations of emotions, which are commonly observed in real-life human interactions (Devillers et al., 2005; Vidrascu and Devillers, 2005). M 3 ED contains 24449 utterances in total, which are more than three times larger than IEMOCAP and almost two times larger than MELD. There are rich emotional interaction phenomena in M 3 ED dialogues, for example, 5,396 and 2,696 inter-turn emotion-shift and emotion-inertia scenarios respectively, and 2,879 and 10,891 intra-turn emotion-shift and emotion-inertia scenarios respectively. To the best of our knowledge, M 3 ED is the first large-scale multi-modal emotional dialogue dataset in Chinese, which can promote research of affective computing for the Chinese language. It is also a valuable addition for cross-cultural emotion analysis and recognition. We further perform the sanity check of the dataset quality. Specifically, we evaluate our proposed M 3 ED dataset on several state-of-the-art approaches, including DialogueRNN, DialogueGCN, and MMGCN. The experimental results show that both context information and multiple modalities can help model the speakers' emotional states and significantly improve the recognition performance, in which context information and multiple multiple modalities are two salient factors of a multimodal emotion dialogue dataset. Furthermore, motivated by the masking strategies of self-attention used in DialogXL (Shen et al., 2020), we propose a general Multimodal Dialogue-aware Interaction (MDI) framework which considers multimodal fusion, global-local context modeling, and speaker interactions modeling and achieves state-of-the-art performance. All in all, M 3 ED is a large, diverse, high-quality, and comprehensive multimodal emotional dialogue dataset, which can support more explorations in the related research directions, such as multi-label learning, interpretability of emotional changes in dialogues, cross-culture emotion recognition, etc. The main contributions of this work are as follows: We build a large-scale Multi-modal Multi-scene and Multi-label Emotional Dialogue dataset called M 3 ED, which can support more explorations in the affective computing field.  We perform a comprehensive sanity check of the dataset quality by running several state-of-the-art approaches on M 3 ED and the experimental results prove the validity and quality of the dataset.  We propose a general Multimodal Dialogue-aware Interaction framework, MDI, which involves multimodal fusion, global-local context and speaker interaction modeling, and it achieves comparable performance to other state-of-the-art approaches. Table 1 summarizes some of the most important emotion datasets related to this work. The EmoryNLP (Zahiri and Choi, 2018), EmotionLines (Chen et al., 2018), and DailyDialog (Li et al., 2017) are emotional dialogue datasets in only text modality, which have been widely used in the ERC tasks. The CMU-MOSEI (Zadeh et al., 2018), AFEW (Dhall et al., 2012), MEC (Li et al., 2018), and CH-SIMS (Yu et al., 2020) contain multiple modalities and have been wildly used for multimodal emotion recognition, but they are not conversational and can not support explorations of dialogue emotional analysis. The IEMOCAP (Busso et al., 2008), MSP-IMPROV (Busso et al., 2016) and MELD (Poria et al., 2019a) are the currently available multimodal emotional dialogue datasets. The IEMOCAP and MSP-IMPROV datasets are recorded from ten/twelve actors performing scripted and spontaneous dyadic dialogues, and each utterance is manually labeled with discrete emotion categories. The MELD (Poria et al., 2019a) is a multi-modal multi-party emotional dialogue dataset extended from the text-based EmotionLines dataset (Chen et al., 2018), which is derived only from the Friends TV series. Previous works on ERC focus on modeling context information in a conversation with different frameworks. BC-LSTM (Poria et al., 2017) employs a Bi-directional LSTM to capture temporal-context information in conversations. CMN (Haz-arika et al., 2018b) and ICON (Hazarika et al., 2018a) use distinct GRUs to model the global and speaker-specific temporal-context, and apply memory networks to model speaker emotional states. DialogueRNN (Majumder et al., 2019) uses distinct GRUs to model global and speaker-specific temporal-context, and global emotional states tracking respectively. DialogueGCN (Ghosal et al., 2019) captures conversational dependencies between utterances with a graph-based structure. MMGCN (Hu et al., 2021) further proposes a GCN-based multimodal fusion method for multimodal ERC tasks to improve recognition performance. DialogXL (Shen et al., 2020) first introduces a strong pre-trained language model XLNet for text-based ERC. It also proposes several masking strategies of self-attention to model the global, local, interspeaker, and intra-speaker interactions. In order to build a large-scale, diversified, and high-quality multimodal emotional dialogue dataset, we collect video dialogue clips from different TV series, which can simulate spontaneous emotional behavior in the real-world environment (Dhall et al., 2012; Li et al., 2018; Poria et al., 2019a). Since high-quality conversation video clips are very important, we require the crowd workers to follow the strict selection requirements, including the following major aspects: 1) The required TV series should belong to these categories, such as family, romance, soap opera, and modern opera, which have rich and natural emotional expressions. 2) The workers are required to select 15  25 high-quality emotional dialogue video clips from each TV series. 3) Each dialogue should have at least 3 rounds of interaction and a clear conversation topic. 4) In order to ensure the quality of the visual and acoustic modalities, the workers are required to select two-person dialogue scenes with clear facial expressions and intelligible voices. After the dialogue selection, we randomly check several dialogues for each TV series and filter out the low-quality dialogues or ask the crowd workers to correct the inappropriate start and end timestamps. In order to facilitate the process of emotion annotation, we first require the crowd workers to correct the text content and annotate the speaker info of each utterance. Since the videos of TV series do not have embedded subtitles, we use the OCR-5701 based (Optical Character Recognition) method 2 to automatically generate the text content and the corresponding timestamps. For speaker annotations, the first speaker in the dialogue is annotated as A, and the other speaker is annotated as B. In addition, we annotate the role names, ages and genders of these speakers as well. We annotate each utterance based on Ekman's six basic emotions ( happy, surprise, sad, disgust, anger, and fear ) and an additional emotion label neutral , which is an annotation scheme widely used in previous works (Poria et al., 2019a; Busso et al., 2008). The annotators are asked to sequentially annotate the utterances, after watching the videos. Thus, the textual, acoustic and visual information, and the previous utterances in the dialogue are available for emotional annotation. The annotators are allowed to select more than one emotional label to account for blended emotions (e.g., anger&sad), which are commonly observed in real-life human interactions (Devillers et al., 2005). If none of the seven emotion categories can accurately describe the emotion status of the utterance, a special other category can be annotated. In order to obtain high-quality annotations, we together with several emotional psychology experts design an annotation tutorial with reference to previous guidelines (Ekman, 1992; Campos et al., 2013). We train the annotators and provide them with an examination, and only those who pass the exam can participate in the annotation stage. The vast majority of the dataset is annotated by university students and all the annotators are native Mandarin speakers. We assign three annotators to each dialogue. We apply the majority voting strategy over all the annotations of an utterance to produce its final emotion label. Please note that annotators are allowed to assign more than one emotion label to an utterance, and the importance of these labels is in descending order. We simply assign an importance value to the emotion label of each utterance in descending order, e.g. I ( e ) = 7 for the first emotion label, I ( e ) = 6 for the second emotion label, and so on. If a label is not assigned to the utterance, its importance value I ( e ) = 0 . An emotion label e is assigned as one of the final emotion labels for an utterance, if it is assigned to the utterance by at least two annotators. And its importance value is decided by averaging its importance ranking from all annotators: I ( e ) = (cid:80) 3 k =1 I k ( e ) , where I k ( e ) is its importance value from annotator k . To further ensure annotation quality, we design two strategies to review and revise incorrect annotations. 1) We calculate the annotation agreement between the annotators of each dialogue. For the dialogues with a poor agreement, we require all relevant annotators to review the annotations again and make corrections if necessary. 2) For the utterances (0.5% of all utterances) that don't have a majority annotators' agreement, we ask several high-quality annotators to review them and make a final emotion annotation decision for these utterances. Finally, we analyze the inter-annotators agreement and achieve an overall Fleiss' Kappa (Fleiss et al., 2013) statistic of k = 0 . 59 for a seven-class emotion problem, which is higher than other datasets, such as k = 0 . 43 in MELD, k = 0 . 48 in Table 2 presents several basic statistics of the M 3 ED dataset. It contains 990 dialogues, 9,082 turns, 24,449 utterances derived from 56 different TV series (about 500 episodes), which ensures the scale and diversity of the dataset. We adopt the TV-independent data split manner in order to avoid any TV-dependent bias, which means there is no overlap of TV series across training, validation, and testing sets. The basic statistics are similar across these three data splits. There are rich emotional interactions phenomena in the M 3 ED, for example, 5,396 and 2,696 inter-turn emotion-shift and emotion-inertia scenarios respectively, and 2,879 and 10,891 intra-turn emotion-shift and emotion-inertia scenarios. The emotion shift and emotion inertia are two important factors in dialogues, which are challenging and worthy of exploration (Poria et al., 2019a). As shown in the table, 89% of utterances are assigned with one emotion label, and 11% of utterances are assigned with blended emotions 3 . Table 3 presents the single emotion distribution statistics. The distribution of each emotion cate-3 cate-3 The top 5 most frequent blended emotions are: anger&disgust, anger&sad, sad&anger, disgust&anger and fear&sad gory is similar across train/val/test sets. As shown in Table 4, there are in total 626 different speakers in M 3 ED with balanced gender distribution. Among all the speakers, young and middle-aged speakers account for more than 80%. A dialogue can be defined as a sequence of utterances D = { utt 1 , utt 2 , ..., utt N } , where N is the number of utterances. Each utterance consists of textual ( l ), acoustic ( a ) and visual ( v ) modalities. We denote u At [ a, v, l ] as the utterance-level feature of utterance utt t from speaker A with the textual, acoustic and visual modality respectively. The task aims to predict the emotional state for each utterance in the dialogue based on all existing modalities. Figure 2 illustrates our proposed Multimodal Dialogue-aware Interaction (MDI) framework, which contains three main modules: 1) Multimodal Fusion module aims to generate the utterance-level multimodal representation from different modalities. 2) Dialog-aware Interaction module aims to model the interactions in the dialogue; 3) Interaction Fusion and Classification module fuses the different interaction information from the outputs of the Dialog-aware Interaction module, and then makes the emotional state prediction based on the fused interaction information. Multimodal Fusion Module: Based on the modality-specific feature representations from different modalities, we apply early fusion of these modalities features to produce the multimodal feature representation: u = concat ( u [ a ] , u [ v ] , u [ l ]) . Dialog-aware Interaction Module: In order to adequately capture the contextual information in the dialogue, we propose the Dialog-aware Interaction Module which consists of L dialog-aware interaction blocks (gray block in Figure 2). In each block, we adopt four sub-modules, Global Interaction, Local Interaction, Intra-speaker Interaction and Inter-speaker Interaction, to model the global, local, intra-speaker and inter-speaker interactions in the dialogue respectively. We implement these four types of interactions in one Transformer layer by skillfully changing the masking strategies of self-attention (Shen et al., 2020; Li et al., 2020) as illustrated in Figure 3. Interaction Fusion and Classification: As the Dialog-aware Interaction Module produces different outputs that carry various interaction contextual information, we fuse these outputs via simple addition. Finally, we use one fully connected layer as a classifier to predict the emotional state based on the fused interaction information. We investigate the state-of-the-art features of different modalities including textual, acoustic, and visual features for emotion recognition tasks 4 . 4 More detailed description of the feature extractors can be found in the supplementary material. A.2 Textual Features: We extract the word-level features from a pre-trained RoBERTa model (Yu et al., 2020). Furthermore, to get more efficient emotional features, we extract the finetuned features ([CLS] position) from the finetuned RoBERTa model trained on M 3 ED. We refer to the word-level and finetuned utterance-level textual features as L_Frm, and L_Utt respectively. Acoustic Features: We extract the frame-level features from a pre-trained Wav2Vec2.0 model (Baevski et al., 2020). We extract the finetuned features (the last time step) from the Wav2Vec2.0 model finetuned on M 3 ED. We refer to the frame-level and finetuned utterance-level acoustic features as A_Frm and A_Utt respectively. Visual Features: We first propose a two-stage strategy to detect the speaker's faces 5 . We then extract the face-level features via a pre-trained DenseNet model (Huang et al., 2017) for each utterance based on the detected speaker's faces. DenseNet was trained on two facial expression benchmark corpus, FER+ (Barsoum et al., 2016) and AffectNet (Mollahosseini et al., 2017). We average the face-level features within one utterance to get the averaged utterance-level features. We refer to the face-level, averaged utterance-level visual features as V_Frm, V_Utt respectively. We evaluate several state-of-the-art methods including utterance-level recognition methods and dialog-level recognition methods on the proposed M 3 ED dataset, and they are listed as follows: MultiEnc: A flexible and efficient utterance-level multimodal emotion recognition framework (Zhao et al., 2021) that consists of several modality-specific encoders (LSTM, LSTM and TextCNN for acoustic, visual and textual modalities respectively) and a fusion encoder (several fully-connected layers) for emotion prediction. For the utterance-level modality features, three DNN encoders are used for the three modalities respectively. DialogueRNN: A state-of-the-art RNN-based ERC framework proposed in (Majumder et al., 2019), which captures the global and speaker-specific temporal context information, and global emotional state information via different GRUs. For the multimodal experiments, the early-fusion method that concatenates different modality features as input is adopted in this work. DialogueGCN: A state-of-the-art GCN-based ERC framework proposed in (Ghosal et al., 2019), which models long-distance dependency and speaker interactions via direct edges and different designed relations respectively. For the multimodal experiments, we also adopt the early-fusion method in this work. MMGCN: A state-of-the-art GCN-based multimodal ERC framework proposed in (Hu et al., 2021). For the uni-modal experiments, we only model the fully connected graph. We split the M 3 ED dataset into training, validation, testing sets in a TV-independent manner, which is a more challenging experiment setting. The distribution of the data splits is shown in Table 3. We use the weighted-F1 score (WF1) as the evaluation metrics. We tune the parameters on the validation set and report the performance on the testing set. We run each model three times and report the average performance to alleviate the influence of random parameter initialization. We conduct two sets of experiments, including 1) the utterance-level baseline experiments of emotion recognition on isolated utterances without considering dialogue context, which aims to check the quality of each modality and compare the effectiveness of multimodal information for emotion recognition, and 2) the dialogue-level experiments of emotion recognition in the dialogue, which aims to compare our proposed general MDI framework with the state-of-the-art models in modeling dialogue context for emotion recognition. For the utterance-level experiments, we adopt the Multi-Enc (Section 5.2) framework as the baseline model. For the dialogue-level experiments, we compare to DialogueRNN, DialogueGCN, and MMGCN models. Since different modality features are used in this work, we have tried different hidden sizes (such as 180, 256, and 512) in our experiments. For the experiments on the proposed Multimodal Dialog-aware Interaction framework (Section 4), we use the Adam optimizer with learning rate of 3e-5. We set the dropout as 0.1, the hidden size as 384 in the unimodal experiments and 512 in the multimodal experiments. Table 5 presents the utterance-level baseline results. Among the different unimodal features, the Table 5: Utterance-level baseline performance (WF1) of different features and different modalities. finetuned utterance-level features achieve significant improvement on textual and acoustic modalities. The multimodal information can bring significant performance improvement over unimodal. However, for the multimodal experiments, the finetuned features do not show much improvement over the frame-level features. It is mainly because the finetuned features retain more classification information and lose some modality-specific information, which limits the complementarity between the modalities. In addition, we observe that there is no big gap between the performances on different modalities, which indicates the good quality of different modalities in our M 3 ED dataset. Since the state-of-the-art dialogue-level methods mainly focus on modeling the dialogue context information based on the utterance-level features, we adopt the finetuned utterance-level features (Utt_ft) in the following experiments. Table 6 presents the dialogue-level experiment results. The results show that context information and multiple modalities, the two salient factors of a multimodal emotion dialogue dataset, both bring significant performance improvement, which also proves the validity and quality of the M 3 ED dataset to some extend. Compared to the state-of-the-art models, our proposed general MDI framework achieves superior performance in the textual, acoustic, and visual unimodal experiments. It demonstrates that the four dialogue-aware interaction strategies which consider both the globaland local-context interactions and the intraand inter-speaker interactions have better dialogue modeling ability than only considering part of these interactions, which demon-5705 Table 6: Emotion recognition performance (WF1) in dialogues under the unimodal and multimodal conditions. strates the strong dialogue context modeling ability of MDI. However, MDI does not outperform other models under the multimodal conditions, which may be due to the limited training dataset size and the limited ability of the vanilla multimodal fusion strategy in interaction modeling. In the future, we will explore more effective multimodal fusion module and interaction modeling module within the MDI framework to improve its performance under multimodal conditions. The M 3 ED dataset is a large, diversified, high-quality, and comprehensive multimodal emotional dialogue dataset. Based on the characteristics of the dataset and the analysis from the extensive experiments, we believe that M 3 ED can support a number of related explorations in affective computing field. Based on the experiment results, we think that the finetuned features lack sufficient modality-specific information, which limits the performance under the multimodal conditions. Therefore, it is worth exploring to realize a more efficient multimodal fusion module based on the raw frame-level features and make the above proposed general Multimodal Dialog-aware Interaction (MDI) framework an end-to-end model. According to psychological and behavioral studies, emotional inertia and stimulus (exter-nal/internal) are important factors that affect the speaker's emotional state in dialogues. The emotional inertia and emotional stimulus can explain how one speaker's emotion affects his own or the other speaker's emotion. There are rich emotional interaction phenomena including interand intra-turn emotion shifts in the M 3 ED dataset. Therefore, it can support the exploration of interpretability of emotional changes in a Dialogue. The blended emotions are commonly observed in human real-life dialogues, and multi-label learning can help reveal and model the relevance between different emotions. Therefore, the M 3 ED dataset can support the exploration of multi-label emotion recognition in conversations. Emotional expression varies across different languages and cultures. The M 3 ED dataset in Chinese is a valuable addition to the existing benchmark datasets in other languages. It can promote the research of cross-culture emotion analysis and recognition. In this work, we propose a multi-modal, multi-scene, and multi-label emotional dialogue dataset, M 3 ED, for multimodal emotion recognition in conversations. Compared to MELD, the currently largest multimodal dialogue dataset for emotion recognition, M 3 ED is larger (24,449 vs. 13,708 ut-terances), more diversified (56 different TV series vs. only one TV series Friends), with higher-quality (balanced performance across all three modalities), and containing blended emotions annotation which is not available in MELD. M 3 ED is the first multimodal emotion dialogue dataset in Chinese, which can serve as a valuable addition to the affective computing community and promote the research of cross-culture emotion analysis and recognition. Furthermore, we propose a general Multimodal Dialog-aware Interaction framework, which considers multimodal fusion, temporal-context modeling, and speaker interactions modeling, and achieves the state-of-the-art performance. We also propose several interesting future exploration directions based on the M 3 ED dataset. This work was partially supported by the National Key R&D Program of China (No. 2020AAA0108600), the National Natural Science Foundation of China (No. 62072462), Large-Scale Pre-Training Program 468 of Beijing Academy of Artificial Intelligence (BAAI), A*STAR RIE2020 Advanced Manufacturing and Engineering Domain (AME) Programmatic Grant (No. A1687b0033), NRF Centre for Advanced Robotics Technology Innovation (CARTIN) Project and China Scholarship Council. This work presents M 3 ED, free and open dataset for the research community to study the multimodal emotion recognition in dialogues. Data in M 3 ED are collected from TV series in Chinese. To ensure that crowd workers were fairly compensated, we paid them at an hourly rate of 40 yuan ($6.25 USD) per hour, which is a fair and reasonable hourly wage in Beijing. First, to select high-quality dialogues from 56 TV-series, we recruited 12 Chinese college students (5 males and 7 females). Each student was paid 100 yuan ($15.625 USD) for selecting about 18 dialogues from each TV series. To annotate the emotional status of the selected dialogues, we recruited 14 Chinese college students (6 males and 8 females). Each student was paid 200 yuan ($31.25 USD) for annotating about 18 dialogues from each TV series with emotion labels, text correction, speaker, gender, and age information. If only the emotion labels were annotated, the payment for each TV series was 100 yuan ($15.625 USD). Considering the copy-right issue of TV-series, we will only release the name list of the TV-series and our annotations. To facilitate future comparison research on this dataset, we will provide our extracted visual expression features and acoustic features. We anticipate that the high-quality and rich annotation labels in the dataset will advance research in multimodal emotion recognition.
abstain abstain objective abstain objective abstain result objective abstain abstain abstain other other other other objective abstain objective other other other other method method method objective objective result result other abstain method result result abstain abstain abstain other method abstain method method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method method method method method method method method abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain objective objective other method abstain abstain abstain abstain abstain abstain abstain method method abstain