Modeling Contextual Relationships Among Utterances for Multimodal Sentiment Analysis
Multimodal sentiment analysis is a developing area of research, which involves identification of emotions and sentiments in videos. Current research considers utterances as independent entities, i.e., ignores the inter-dependencies and relations among utterances of a video. In this paper, we propose an LSTM based model which enables these utterances to capture contextual information from its surroundings in the same video, thus aiding the classification process. Our model shows 5 − 10% improvement over the state of the art and high robustness to generalizability.
1 Introduction
Emotion recognition and sentiment analysis have become a new trend in social media, helping users to automatically extract the opinions expressed in user-generated content, especially videos. Thanks to the high availability of computers and smartphones, and the rapid rise of social media, consumers tend to record their reviews and opinions about products or films and upload them on social media platforms, such as YouTube or Facebook. Such videos often contain comparisons, which can aid prospective buyers make an informed decision.
The primary advantage of analyzing videos over text is the surplus of behavioral cues present in vocal and visual modalities. The vocal modulations and facial expressions in the visual data, along with textual data, provide important cues to better identify affective states of the opinion holder. Thus, a combination of text and video data helps to create a better emotion and sentiment analysis model (Poria et al., 2017).
Recently, a number of approaches to multimodal sentiment analysis, producing interesting results, have been proposed (Pérez-Rosas et al., 2013; Wollmer et al., 2013; Poria et al., 2015). However, there are major issues that remain unaddressed, such as the role of speaker-dependent versus speaker-independent models, the impact of each modality across the dataset, and generalization ability of a multimodal sentiment classifier. Leaving these issues unaddressed has presented difficulties in effective comparison of different multimodal sentiment analysis methods.
An utterance is a unit of speech bound by breathes or pauses. Utterance-level sentiment analysis focuses on tagging every utterance of a video with a sentiment label (instead of assigning a unique label to the whole video). In particular, utterance-level sentiment analysis is useful to understand the sentiment dynamics of different aspects of the topics covered by the speaker throughout his/her speech. The true meaning of an utterance is relative to its surrounding utterances.
In this paper, we consider such surrounding utterances to be the context, as the consideration of temporal relation and dependency among utterances is key in human-human communication. For example, the MOSI dataset (Zadeh et al., 2016) contains a video, in which a girl reviews the movie ‘Green Hornet’. At one point, she says “The Green Hornet did something similar”. Normally, doing something similar, i.e., monotonous or repetitive might be perceived as negative. However, the nearby utterances “It engages the audience more”, “they took a new spin on it”, “and I just loved it” indicate a positive context.
In this paper, we discard the oversimplifying hypothesis on the independence of utterances and develop a framework based on long short-term memory (LSTM) to extract utterance features that also consider surrounding utterances.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
Our model enables consecutive utterances to share information, thus providing contextual information in the classification process. Experimental results show that the proposed framework has outperformed the state of the art on benchmark datasets by 5−10%. The paper is organized as follows: Section 2 provides a brief literature review on multimodal sentiment analysis; Section 3 describes the proposed method in detail; experimental results and discussion are shown in Section 4; finally, Section 5 concludes the paper.
2 Related Work
Text-based sentiment analysis systems can be broadly categorized into knowledge-based and statistics-based systems (Cambria, 2016). While the use of knowledge bases was initially more popular for the identification of emotions and polarity in text, sentiment analysis researchers have recently been using statistics-based approaches, with a special focus on supervised statistical methods (Pang et al., 2002; Socher et al., 2013).
In 1970, Ekman (Ekman, 1974) carried out extensive studies on facial expressions which showed that universal facial expressions are able to provide sufficient clues to detect emotions. Recent studies on speech-based emotion analysis (Datcu and Rothkrantz, 2008) have focused on identifying relevant acoustic features, such as fundamental frequency (pitch), intensity of utterance, bandwidth, and duration.
As for fusing audio and visual modalities for emotion recognition, two of the early works were done by De Silva et al. (De Silva et al., 1997) and Chen et al. (Chen et al., 1998). Both works showed that a bimodal system yielded a higher accuracy than any unimodal system. More recent research on audio-visual fusion for emotion recognition has been conducted at either feature level (Kessous et al., 2010) or decision level (Schuller, 2011).
While there are many research papers on audiovisual fusion for emotion recognition, only a few have been devoted to multimodal emotion or sentiment analysis using textual clues along with visual and audio modalities. Wollmer et al. (Wollmer et al., 2013) and Rozgic et al. (Rozgic et al., 2012a,b) fused information from audio, visual, and textual modalities to extract emotion and sentiment. Metallinou et al. (Metallinou et al., 2008) and Eyben et al. (Eyben et al., 2010a) fused audio and textual modalities for emotion recognition.
Both approaches relied on a feature-level fusion. Wu et al. (Wu and Liang, 2011) fused audio and textual clues at decision level.
3 Method
In this work, we propose a LSTM network that takes as input all utterances in a video and extracts contextual unimodal and multimodal features by modeling the dependencies among the input utterances. Below, we propose an overview of the method -
1. Context-Independent Unimodal UtteranceLevel Feature Extraction
First, the unimodal features are extracted without considering the contextual information of the utterances (Section 3.1). Table 1 presents the feature extraction methods used for each modality.
2. Contextual Unimodal and Multimodal Classification
The context-independent unimodal features (from Step 1) are then fed into a LSTM network (termed contextual LSTM) that allows consecutive utterances in a video to share semantic information in the feature extraction process (which provides context-dependent unimodal and multimodal classification of the utterances). We experimentally show that this proposed framework improves the performance of utterance-level sentiment classification over traditional frameworks.
Videos, comprising of its constituent utterances, serve as the input. We represent the dataset as U:
U = ⎡⎢⎢⎢⎢⎢⎣ u1,1 u1,2 u1,3 ... u1,L1 u2,1 u2,2 u2,3 ... u2,L2 . . . ... . uM,1 uM,2 uM,3 ... uM,LM ⎤⎥⎥⎥⎥⎥⎦ .
Here, ui,j denotes the jth utterance of the ith video and L = [L1, L2, ..., LM ] represents the number of utterances per video in the dataset set.
3.1 Extracting Context-Independent Unimodal Features
Initially, the unimodal features are extracted from each utterance separately, i.e., we do not consider the contextual relation and dependency among the utterances (Table 1). Below, we explain the textual, audio, and visual feature extraction methods.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
3.1.1 text-CNN: Textual Features Extraction
For feature extraction from textual data, we use a convolutional neural network (CNN).
The idea behind convolution is to take the dot product of a vector of k weights, wk, known as kernel vector, with each k-gram in the sentence s(t) to obtain another sequence of features c(t) = (c1(t), c2(t), . . . , cL(t)):
cj = wTk ⋅ xi∶i+k−1. We then apply a max pooling operation over the feature map and take the maximum value ĉ(t) = max{c(t)} as the feature corresponding to this particular kernel vector. We use varying kernel vectors and window sizes to obtain multiple features.
The process of extracting textual features is as follows -
First, we represent each sentence as the concatenation of vectors of the constituent words. These vectors are the publicly available 300-dimensional word2vec vectors trained on 100 billion words from Google News (Mikolov et al., 2013). The convolution kernels are thus applied to these word vectors instead of individual words. Each sentence is wrapped to a window of 50 words which serves as the input to the CNN.
The CNN has two convolutional layers - the first layer having a kernel size of 3 and 4, with 50 feature maps each and a kernel size 2 with 100 feature maps for the second. The convolution layers are interleaved with pooling layers of dimension 2. We use ReLU as the activation function. The convolution of the CNN over the sentence learns abstract representations of the phrases equipped with implicit semantic information, which with each successive layer spans over increasing number of words and ultimately the entire sentence.
Modality Model
Text text-CNN: Deep Convolutional NeuralNetwork with word embeddings
Video 3d-CNN: 3-dimensional CNNs employed onutterances of the videos Audio openSMILE: Extracts low level audiodescriptors from the audio modality
Table 1: Methods for extracting context independent baseline features from different modalities.
3.1.2 Audio Feature Extraction
Audio features are extracted in 30 Hz frame-rate; we use a sliding window of 100 ms. To compute the features, we use the open-source software
openSMILE (Eyben et al., 2010b) which automatically extracts pitch and voice intensity. Voice normalization is performed and voice intensity is thresholded to identify samples with and without voice. Z-standardization is used to perform voice normalization.
The features extracted by openSMILE consist of several low-level descriptors (LLD) and their statistical functionals. Some of the functionals are amplitude mean, arithmetic mean, root quadratic mean, etc. Taking into account all functionals of each LLD, we obtained 6373 features.
3.1.3 Visual Feature Extraction
We use 3D-CNN to obtain visual features from the video. We hypothesize that 3D-CNN will not only be able to learn relevant features from each frame, but will also be able to learn the changes among given number of consecutive frames.
In the past, 3D-CNN has been successfully applied to object classification on 3D data (Ji et al., 2013). Its ability to achieve state-of-the-art results motivated us to use it.
Let vid ∈ Rc×f×h×w be a video, where c = number of channels in an image (in our case c = 3, since we consider only RGB images), f = number of frames, h = height of the frames, and w = width of the frames. Again, we consider the 3D convolutional filter filt ∈ Rfm×c×fl×fh×fw, where fm = number of feature maps, c = number of channels, fd = number of frames (in other words depth of the filter), fh = height of the filter, and fw = width of the filter. Similar to 2D-CNN, filt slides across video vid and generates output convout ∈ Rfm×c×(f−fd+1)×(h−fh+1)×(w−fw+1). Next, we apply max pooling to convout to select only relevant features. The pooling will be applied only to the last three dimensions of the array convout.
In our experiments, we obtained best results with 32 feature maps (fm) with the filter-size of 5 × 5 × 5 (or fd × fh × fw). In other words, the dimension of the filter is 32 × 3 × 5 × 5 × 5 (or fm × c × fd × fh × fw). Subsequently, we apply max pooling on the output of convolution operation, with window-size being 3×3×3. This is followed by a dense layer of size 300 and softmax. The activations of this dense layer are finally used as the video features for each utterance.
3.2 Context-Dependent Feature Extraction
We hypothesize that, within a video, there is a high probability of utterance relatedness with respect
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
to their sentimental and emotional clues. Since most videos tend to be about a single topic, the utterances within each video are correlated, e.g., due to the development of the speaker’s idea, coreferences, etc. This calls for a model which takes into account such inter-dependencies and the effect these might have on the current utterance. To capture this flow of informational triggers across utterances, we use a LSTM-based recurrent network scheme (Gers, 2001).
3.2.1 Long Short-Term Memory
LSTM is a kind of recurrent neural network (RNN), an extension of conventional feed-forward neural network. Specifically, LSTM cells are capable of modeling long-range dependencies, which other traditional RNNs fail to do given the vanishing gradient issue. Each LSTM cell consists of an input gate i, an output gate o, and a forget gate f , which enables it to remember the error during the error propagation. Current research (Zhou et al., 2016) indicates the benefit of using such networks to incorporate contextual information in the classification process.
In our case, the LSTM network serves the purpose of context-dependent feature extraction by modeling relations among utterances. We term our architecture ‘contextual LSTM’. We propose several architectural variants of it later in the paper.
LSTM
sc-L ST M
Utterance 1 Utterance 2 Utterance !
Context Sensitive Utterance features
Dense Layer
Softmax Output
LSTM LSTM
Context Independent Utterance features
Figure 1: Contextual LSTM network: input features are passed through an unidirectional LSTM layer, followed by a dense layer and then a softmax layer. Categorical cross entropy loss is taken for training. The dense layer activations serve as the output features.
3.2.2 Contextual LSTM Architecture
Let unimodal features have dimension k, each utterance is thus represented by a feature vector xi,t ∈ Rk, where t represents the tth utterance of the video i. For a video, we collect the vectors for all the utterances in it, to get Xi = [xi,1,xi,2, ...,xi,Li] ∈ RLi×k, where Li represents the number of utterances in the video. This matrix Xi serves as the input to the LSTM. Figure 1 demonstrates the functioning of this LSTM module.
In the procedure getLstmFeatures(Xi) of Algorithm 1, each of these utterance xi,t is passed through a LSTM cell using the equations mentioned in line 32 to 37. The output of the LSTM cell hi,t is then fed into a dense layer and finally into a softmax layer (line 38 to 39). The activations of the dense layer zi,t are used as the contextdependent features of contextual LSTM.
3.2.3 Training
The training of the LSTM network is performed using categorical cross entropy on each utterance’s softmax output per video, i.e.,
loss = 1 N
N
∑ n=1
C
∑ c=1 yn,c log2( ˆyn,c),
where N = total number of utterances in a video, yn,c = original output of class c, and ˆyn,c = predicted output.
A dropout layer between the LSTM cell and dense layer is introduced to check overfitting. As the videos do not have same the number of utterances, padding is introduced to serve as neutral utterances. To avoid the proliferation of noise within the network, masking is done on these padded utterances to eliminate their effect in the network. Parameter tuning is done on the train set by splitting it into train and validation components with 80/20% split. RMSprop has been used as the optimizer which is known to resolve Adagrad’s radically diminishing learning rates (Duchi et al., 2011). After feeding the train set to the network, the test set is passed through it to generate their context-dependent features.
Different Network Architectures We consider the following variants of the contextual LSTM architecture in our experiments -
sc-LSTM This variant of the contextual LSTM architecture consists of unidirectional
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
LSTM cells. As this is the simple variant of the contextual LSTM, we termed it as simple contextual LSTM (sc-LSTM)
h-LSTM We also test on an architecture where the dense layer after the LSTM cell is omitted. Thus, the output of the LSTM cell hi,t provides our context-dependent features and the softmax layer provides the classification. We call this architecture hidden LSTM (h-LSTM).
bc-LSTM Bi-directional LSTMs are two unidirectional LSTMs stacked together having opposite directions. Thus, an utterance can get information from other utterances occurring before and after itself in the video. We replaced the regular LSTM with a bi-directional LSTM and named the resulting architecture as bi-directional contextual LSTM (bc-LSTM). The training process of this architecture is similar to sc-LSTM.
uni-SVM In this setting, we first obtain the unimodal features as explained in Section 3.1, concatenate them and then send to a SVM for the final classification. It should be noted that using a gated recurrent unit (GRU) instead of LSTM did not improve the performance.
3.3 Fusion of Modalities
We accomplish multimodal fusion in two different ways as explained below -
3.3.1 Non-hierarchical Framework
In non-hierarchical framework, we concatenate context-independent unimodal features (from Section 3.1) and feed that into the contextual LSTM networks, i.e., sc-LSTM, bc-LSTM, and h-LSTM.
3.3.2 Hierarchical Framework
Contextual unimodal features, taken as input, can further improve performance of the multimodal fusion framework explained in Section 3.3.1. To accomplish this, we propose a hierarchical deep network which comprises of two levels –
Level-1: context-independent unimodal features (from 3.1) are fed to the proposed LSTM network (Section 3.2.2) to get context-sensitive unimodal feature representations for each utterance. Individual LSTM networks are used for each modality.
Level-2: consists of a contextual LSTM network similar to Level-1 but independent in training and computation. Output from each LSTM network in Level-1 are concatenated and fed into
this LSTM network, thus providing an inherent fusion scheme - the prime objective of this level (Fig 2). The performance of the second level banks on the quality of the features from the previous level, with better features aiding the fusion process. Algorithm 1 describes the overall computation for utterance classification. For the hierarchical framework, we train Level 1 and Level 2 successively but separately.
Weight Bias Wi,Wf ,Wc,Wo ∈ Rd×k bi, bf , bc, bo ∈ Rd Pi, Pf , Pc, PoVo ∈ Rd×d bz ∈ Rm
Wz ∈ Rm×d bsft ∈ Rc Wsft ∈ Rc×m
Table 2: Summary of notations used in Algorithm 1. Note: d - dimension of hidden unit. k - dimension of input vectors to LSTM layer . c - number of classes.
4 Experimental Results
4.1 Dataset details
Most of the research in multimodal sentiment analysis is performed on datasets with speaker overlap in train and test splits.
Because each individual has a unique way of expressing emotions and sentiments, finding generic, person-independent features for sentimental analysis is very tricky. In real-world applications, the
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
592
593
594
595
596
597
598
599
Algorithm 1 Proposed Architecture 1: procedure TRAINARCHITECTURE( U, V) 2: Train context-independent models with U 3: for i:[1,M] do ▷ extract baseline features 4: for j:[1,Li] do 5: xi,j ← TextFeatures(ui,j) 6: x ′
i,j ← V ideoFeatures(ui,j) 7: x”i,j ← AudioFeatures(ui,j) 8: Unimodal: 9: Train LSTM at Level-1 with X,X ′ andX”.
10: for i:[1,M] do ▷ unimodal features 11: Zi ← getLSTMFeatures(Xi) 12: Z ′ i ← getLSTMFeatures(X ′
i) 13: Z”i ← getLSTMFeatures(X”i ) 14: Multimodal: 15: for i:[1,M] do 16: for j:[1,Li] do 17: if Non-hierarchical fusion then 18: x∗i,j ← (xi,j ∣∣x ′
i,j ∣∣x”i,j) ▷ concatenation
19: else 20: if Hierarchical fusion then 21: x∗i,j ← (zi,j ∣∣z ′
i,j ∣∣z”i,j) ▷ concatenation 22: Train LSTM at Level-2 with X∗. 23: for i:[1,M] do ▷ multimodal features 24: Z∗i ← getLSTMFeatures(X∗i ) 25: testArchitecture( V) 26: return Z∗
27: procedure TESTARCHITECTURE( V) 28: Similar to training phase. V is passed through the
learnt models to get the features and classification outputs. Table 2 shows the trainable parameters.
29: procedure GETLSTMFEATURES(Xi) ▷ for ith video 30: Zi ← φ 31: for t:[1,Li] do ▷ Table 2 provides notation 32: it ← σ(Wixi,t + Pi.ht−1 + bi) 33: C̃t ← tanh(Wcxi,t + Pcht−1 + bc) 34: ft ← σ(Wfxt + Pfht−1 + bf) 35: Ct ← it ∗ C̃t + ft ∗Ct−1 36: ot ← σ(Woxt + Poht−1 + VoCt + bo) 37: ht ← ot ∗ tanh(Ct) ▷ output of lstm cell 38: zt ← ReLU(Wzht + bz) ▷ dense layer 39: prediction← softmax(Wsftzt + bsft) 40: Zi ← Zi ∪ zt 41: return Zi
model should be robust to person variance but it is very difficult to come up with a generalized model from the behavior of a limited number of individuals To this end, we perform person-independent experiments to emulate unseen conditions. Our train/test splits of the datasets are completely disjoint with respect to speakers.
While testing, our models have to classify emotions and sentiments from utterances by speakers they have never seen before.
IEMOCAP: The IEMOCAP contains the acts of 10 speakers in a two way conversation segmented into utterances. The database contains the following categorical labels: anger, happiness, sadness, neutral, excitement, frustration, fear, surprise, and other, but we take only the first four so as to compare with the state of the art (Rozgic et al., 2012b) and other authors. Videos by the first 8 speakers are considered in the train set. The train/test split details are provided in table 3.
MOSI: The MOSI dataset is a dataset rich in sentimental expressions where 93 persons review topics in English. It contains positive and negative classes as its sentiment labels. The train/validation set comprises of the first 62 individuals in the dataset.
MOUD: This dataset contains product review videos provided by around 55 persons. The reviews are in Spanish (we use Google Translate API1 to get the english transcripts). The utterances are labeled to be either positive, negative or neutral. However, we drop the neutral label to maintain consistency with previous work. The first 59 videos are considered in the train/val set.
Table 3 provides information regarding train/test split of all the datasets. In these splits it is ensured that 1) No two utterances from the train and test splits belong to the same video. 2) The train/test splits have no speaker overlap. This provides the speaker-independent setting.
Table 3 also provides cross dataset split details where the complete datasets of MOSI and MOUD are used for training and testing respectively. The proposed model being used on reviews from different languages allows us to analyze its robustness and generalizability.
It should be noted that the datasets’ individual configuration and splits are same throughout all the experiments (i.e., context-independent unimodal feature extraction, LSTM-based context-
1http://translate.google.com
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
dependent unimodal and multimodal feature extraction and classification).
4.2 Performance of Different Models and Comparisons
In this section, we present unimodal and multimodal sentiment analysis performance of different LSTM network variants as explained in Section 3.2.3 and comparison with the state of the art.
Hierarchical vs Non-hierarchical Fusion Framework - As expected, trained contextual unimodal features help the hierarchical fusion framework to outperform the non-hierarchical framework. Table 4 demonstrates this by comparing both hierarchical and non-hierarchical framework using the bc-LSTM network. Due to this fact, we provide all further analysis and results using the hierarchical framework. Nonhierarchical model outperforms the performance of the baseline uni-SVM. This further leads us to conclude that it is the context-sensitive learning paradigm which plays the key role in improving performance over the baseline.
Comparison among Network Variants - It is to be noted that both sc-LSTM and bc-LSTM perform quite well on the multimodal emotion recognition and sentiment analysis datasets. Since, bcLSTM has access to both the preceding and following information of the utterance sequence, it performs consistently better on all the datasets over sc-LSTM.
The usefulness of the dense layer in improving the performance is prominent from the experimental results as shown in Table 4. The performance improvement is in the range of 0.3% to 1.5% on MOSI and MOUD datasets. On the IEMOCAP dataset, the performance improvement of bc-LSTM and sc-LSTM over h-LSTM is in the range of 1% to 5%.
Comparison with the Baseline and state of the art - Every LSTM network variant has outperformed the baseline uni-SVM on all the datasets by the margin of 2% to 5%(see Table 4). These results prove our initial hypothesis that modeling the contextual dependencies among utterances, which uni-SVM cannot do, improves the classification. The higher performance improvement on the IEMOCAP dataset indicates the necessity of modeling long-range dependencies among the utterances as continuous emotion recognition
is a multiclass sequential problem where a person doesnt frequently change emotions (Wöllmer et al., 2008).
We have implemented and compared with the current state-of-the-art approach proposed by Poria et al. (Poria et al., 2015). In their method, they extracted features from each modality and fed to a multiple kernel learning (MKL) classifier. However, they did not conduct the experiment in speaker-independent manner and also did not consider the contextual relation among the utterances. Experimental results in Table 5 shows that the proposed method has outperformed Poria et al. (Poria et al., 2015) by a significant margin. For the emotion recognition task, we have compared our method with the current state of the art (Rozgic et al., 2012b), who extracted features in a similar fashion to (Poria et al., 2015) did. However, for fusion they used SVM trees.
4.3 Importance of the Modalities
As expected, in all kinds of experiments, bimodal and trimodal models have outperformed unimodal models. Overall, audio modality has performed better than visual on all the datasets. On MOSI and IEMOCAP datasets, textual classifier achieves the best performance over other unimodal classifiers. On IEMOCAP dataset, the unimodal and multimodal classifiers obtained poor performance to classify neutral utterances. Textual modality, combined with non-textual modes boosts the performance in IEMOCAP by a large margin. However, the margin is less in the other datasets.
On the MOUD dataset, textual modality performs worse than audio modality due to the noise introduced in translating Spanish utterances to English. Using Spanish word vectors2 in text-CNN results in an improvement of 10% . Nonetheless, we report results using these translated utterances as opposed to utterances trained on Spanish word vectors, in order to make fair comparison with (Poria et al., 2015).
4.4 Generalization of the Models
To test the generalizability of the models, we have trained our framework on complete MOSI dataset and tested on MOUD dataset (Table 6). The performance was poor for audio and textual modality as the MOUD dataset is in Spanish while the model is trained on MOSI dataset which is in En-
2http://crscardellino.me/SBWCE
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Modality MOSI MOUD IEMOCAP hierarchical (%)
no n-
hi er
(% ) hierarchical (%)
no n-
hi er
(% ) hierarchical (%)
no n-
hi er
(% )
un i-
SV M
hL
ST M
sc -L
ST M
bc -L
ST M
un i-
SV M
hL
ST M
sc -L
ST M
bc -L
ST M
un i-
SV M
hL
ST M
sc -L
ST M
bc -L
ST M
T 75.5 77.4 77.6 78.1 49.5 50.1 51.3 52.1 65.5 68.9 71.4 73.6 V 53.1 55.2 55.6 55.8 46.3 48.0 48.2 48.5 47.0 52.0 52.6 53.2 A 58.5 59.6 59.9 60.3 51.5 56.3 57.5 59.9 52.9 54.4 55.2 57.1
T + V 76.7 78.9 79.9 80.2 78.5 50.2 50.6 51.3 52.2 50.9 68.5 70.3 72.3 75.4 73.2 T + A 75.8 78.3 78.8 79.3 78.2 53.1 56.9 57.4 60.4 55.5 70.1 74.1 75.2 75.6 74.5 V + A 58.6 61.5 61.8 62.1 60.3 62.8 62.9 64.4 65.3 64.2 67.6 67.8 68.2 68.9 67.3 T + V + A 77.9 78.1 78.6 80.3 78.1 66.1 66.4 67.3 68.1 67.0 72.5 73.3 74.2 76.1 73.5
Table 4: Comparison of models mentioned in Section 3.2.3. The table reports the accuracy of classification. Note: non-hier ← Non-hierarchical bc-lstm. For remaining fusion hierarchical fusion framework is used (Section 3.3.2)
Modality Sentiment (%) Emotion on IEMOCAP (%)MOSI MOUD angry happy sad neutral T 78.12 52.17 76.07 78.97 76.23 67.44 V 55.80 48.58 53.15 58.15 55.49 51.26 A 60.31 59.99 58.37 60.45 61.35 52.31
T + V 80.22 52.23 77.24 78.99 78.35 68.15 T + A 79.33 60.39 77.15 79.10 78.10 69.14 V + A 62.17 65.36 68.21 71.97 70.35 62.37 A + V + T 80.30 68.11 77.98 79.31 78.30 69.92 State-of 73.551 63.251 73.10 2 72.402 61.902 58.102-the-art 1by (Poria et al., 2015),2by (Rozgic et al., 2012b)
Table 5: Accuracy % on textual (T), visual (V), audio (A) modality and comparison with the state of the art. For fusion, hierarchical fusion framework was used (Section 3.3.2)
Modality MOSI→MOUDuni-SVM h-LSTM sc-LSTM bc-LSTM T 46.5% 46.5% 46.6% 46.9% V 43.3% 45.5% 48.3% 49.6% A 42.9% 46.0% 46.4% 47.2%
T + V 49.8% 49.8% 49.8% 49.8% T + A 50.4% 50.9% 51.1% 51.3% V + A 46.0% 47.1% 49.3% 49.6% T + V + A 51.1% 52.2% 52.5% 52.7%
Table 6: Cross Dataset comparison. The table reports the accuracy of classification.
glish language. However, notably visual modality performs better than other two modalities in this experiment which signifies that in cross-lingual scenarios facial expressions carry more generalized, robust information than audio and textual modalities. We could not carry out the similar experiment for emotion recognition as no other utterance-level dataset apart from the IEMOCAP was available at the time of our experiments.
4.5 Qualitative Analysis
In some cases the predictions of the proposed method are wrong given the difficulty in recognizing the face and noisy audio signal in the utterances. Also, cases where the sentiment is very weak and non contextual, the proposed approach shows some bias towards its surrounding utter-
ances which further leads to wrong predictions. This can be solved by developing a context aware attention mechanism. In order to have a better understanding on roles of modalities for overall classification, we also have done some qualitative analysis. For example, this utterance - ”who doesn’t have any presence or greatness at all.”, was classified as positive by the audio classifier (“doesn’t” was spoken normally by the speaker, but “presence and greatness at all” was spoken with enthusiasm). However, textual modality caught the negation induced by “doesn’t” and classified correctly. In another utterance “amazing special effects” as there was no jest of enthusiasm in speaker’s voice and face audio-visual classifier failed to identify the positivity of this utterance. On the other textual classifier correctly detected the polarity as positive.
On the other hand, textual classifier classified this sentence - “that like to see comic book characters treated responsibly” as positive, possibly because of the presence of positive phrases such as “like to see”, “responsibly”. However, the high pitch of anger in the person’s voice and the frowning face helps identify this to be a negative utterance.
5 Conclusion
Contextual relationship among the utterances is mostly ignored in the literature. In this paper, we developed a LSTM-based network to extract contextual features from the utterances of a video for multimodal sentiment analysis. The proposed method has outperformed the state of the art and showed significant performance improvement over the baseline. As a part of the future work, we plan to propose LSTM attention model to determine importance of the utterances and contribution of modalities in the sentiment classification.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
Dear Authors  thanks for replying to our review comments, which clarifies some detail questions. I appreciate your promise to publish the code, which will be very helpful to other researchers.   Based on this, i increased my overall score to 4.   Strengths: - well-written - extensive experiments - good results  - Weaknesses: - nothing ground-breaking, application of existing technologies - code not available - results are as could be expected  - General Discussion: - why didn't you use established audio features such as MFCCs?  - Minor Details: - L155 and other places: a LSTM to an LSTM - L160, L216 and other Places: why are there hyphens (-) after the text? - L205: explanation of convolution is not clear - Table1 should appear earlier, on page 2 already cited - L263: is 3D-CNN a standard approach in video processing? alternatives? - L375, 378: the ^ should probably positioned above the y - L380:  to check overfitting  to did you mean  to avoid ? - L403, 408..: put names in     or write them italic, to make it easier to recognize them - L420: a SVM to an SVM - L448: Output ... are to wrong numerus, either  Outputs , or use  is   - L489: superflous whitespace after  layer  - L516, 519:  concatenation  should not be in a new line - L567: why don't you know the exact number of persons? - L626: remove comma after Since - L651: doesnt to does not  - L777: insert  hand, the  after other - References: need some cleanup: L823 superflous whitespace, L831 Munich, L860 what is ACL(1)?, L888 superflous ), L894 Volume, L951 superflous new lines, L956 indent Linguistics properly