Modeling Contextual Relationships Among Utterances for Multimodal Sentiment Analysis
Multimodal sentiment analysis is a developing area of research, which involves identification of emotions and sentiments in videos. Current research considers utterances as independent entities, i.e., ignores the inter-dependencies and relations among utterances of a video. In this paper, we propose an LSTM based model which enables these utterances to capture contextual information from its surroundings in the same video, thus aiding the classification process. Our model shows 5 − 10% improvement over the state of the art and high robustness to generalizability.
1 Introduction
Emotion recognition and sentiment analysis have become a new trend in social media, helping users to automatically extract the opinions expressed in user-generated content, especially videos. Thanks to the high availability of computers and smartphones, and the rapid rise of social media, consumers tend to record their reviews and opinions about products or films and upload them on social media platforms, such as YouTube or Facebook. Such videos often contain comparisons, which can aid prospective buyers make an informed decision.
The primary advantage of analyzing videos over text is the surplus of behavioral cues present in vocal and visual modalities. The vocal modulations and facial expressions in the visual data, along with textual data, provide important cues to better identify affective states of the opinion holder. Thus, a combination of text and video data helps to create a better emotion and sentiment analysis model (Poria et al., 2017).
Recently, a number of approaches to multimodal sentiment analysis, producing interesting results, have been proposed (Pérez-Rosas et al., 2013; Wollmer et al., 2013; Poria et al., 2015). However, there are major issues that remain unaddressed, such as the role of speaker-dependent versus speaker-independent models, the impact of each modality across the dataset, and generalization ability of a multimodal sentiment classifier. Leaving these issues unaddressed has presented difficulties in effective comparison of different multimodal sentiment analysis methods.
An utterance is a unit of speech bound by breathes or pauses. Utterance-level sentiment analysis focuses on tagging every utterance of a video with a sentiment label (instead of assigning a unique label to the whole video). In particular, utterance-level sentiment analysis is useful to understand the sentiment dynamics of different aspects of the topics covered by the speaker throughout his/her speech. The true meaning of an utterance is relative to its surrounding utterances.
In this paper, we consider such surrounding utterances to be the context, as the consideration of temporal relation and dependency among utterances is key in human-human communication. For example, the MOSI dataset (Zadeh et al., 2016) contains a video, in which a girl reviews the movie ‘Green Hornet’. At one point, she says “The Green Hornet did something similar”. Normally, doing something similar, i.e., monotonous or repetitive might be perceived as negative. However, the nearby utterances “It engages the audience more”, “they took a new spin on it”, “and I just loved it” indicate a positive context.
In this paper, we discard the oversimplifying hypothesis on the independence of utterances and develop a framework based on long short-term memory (LSTM) to extract utterance features that also consider surrounding utterances.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
Our model enables consecutive utterances to share information, thus providing contextual information in the classification process. Experimental results show that the proposed framework has outperformed the state of the art on benchmark datasets by 5−10%. The paper is organized as follows: Section 2 provides a brief literature review on multimodal sentiment analysis; Section 3 describes the proposed method in detail; experimental results and discussion are shown in Section 4; finally, Section 5 concludes the paper.
2 Related Work
Text-based sentiment analysis systems can be broadly categorized into knowledge-based and statistics-based systems (Cambria, 2016). While the use of knowledge bases was initially more popular for the identification of emotions and polarity in text, sentiment analysis researchers have recently been using statistics-based approaches, with a special focus on supervised statistical methods (Pang et al., 2002; Socher et al., 2013).
In 1970, Ekman (Ekman, 1974) carried out extensive studies on facial expressions which showed that universal facial expressions are able to provide sufficient clues to detect emotions. Recent studies on speech-based emotion analysis (Datcu and Rothkrantz, 2008) have focused on identifying relevant acoustic features, such as fundamental frequency (pitch), intensity of utterance, bandwidth, and duration.
As for fusing audio and visual modalities for emotion recognition, two of the early works were done by De Silva et al. (De Silva et al., 1997) and Chen et al. (Chen et al., 1998). Both works showed that a bimodal system yielded a higher accuracy than any unimodal system. More recent research on audio-visual fusion for emotion recognition has been conducted at either feature level (Kessous et al., 2010) or decision level (Schuller, 2011).
While there are many research papers on audiovisual fusion for emotion recognition, only a few have been devoted to multimodal emotion or sentiment analysis using textual clues along with visual and audio modalities. Wollmer et al. (Wollmer et al., 2013) and Rozgic et al. (Rozgic et al., 2012a,b) fused information from audio, visual, and textual modalities to extract emotion and sentiment. Metallinou et al. (Metallinou et al., 2008) and Eyben et al. (Eyben et al., 2010a) fused audio and textual modalities for emotion recognition.
Both approaches relied on a feature-level fusion. Wu et al. (Wu and Liang, 2011) fused audio and textual clues at decision level.
3 Method
In this work, we propose a LSTM network that takes as input all utterances in a video and extracts contextual unimodal and multimodal features by modeling the dependencies among the input utterances. Below, we propose an overview of the method -
1. Context-Independent Unimodal UtteranceLevel Feature Extraction
First, the unimodal features are extracted without considering the contextual information of the utterances (Section 3.1). Table 1 presents the feature extraction methods used for each modality.
2. Contextual Unimodal and Multimodal Classification
The context-independent unimodal features (from Step 1) are then fed into a LSTM network (termed contextual LSTM) that allows consecutive utterances in a video to share semantic information in the feature extraction process (which provides context-dependent unimodal and multimodal classification of the utterances). We experimentally show that this proposed framework improves the performance of utterance-level sentiment classification over traditional frameworks.
Videos, comprising of its constituent utterances, serve as the input. We represent the dataset as U:
U = ⎡⎢⎢⎢⎢⎢⎣ u1,1 u1,2 u1,3 ... u1,L1 u2,1 u2,2 u2,3 ... u2,L2 . . . ... . uM,1 uM,2 uM,3 ... uM,LM ⎤⎥⎥⎥⎥⎥⎦ .
Here, ui,j denotes the jth utterance of the ith video and L = [L1, L2, ..., LM ] represents the number of utterances per video in the dataset set.
3.1 Extracting Context-Independent Unimodal Features
Initially, the unimodal features are extracted from each utterance separately, i.e., we do not consider the contextual relation and dependency among the utterances (Table 1). Below, we explain the textual, audio, and visual feature extraction methods.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
3.1.1 text-CNN: Textual Features Extraction
For feature extraction from textual data, we use a convolutional neural network (CNN).
The idea behind convolution is to take the dot product of a vector of k weights, wk, known as kernel vector, with each k-gram in the sentence s(t) to obtain another sequence of features c(t) = (c1(t), c2(t), . . . , cL(t)):
cj = wTk ⋅ xi∶i+k−1. We then apply a max pooling operation over the feature map and take the maximum value ĉ(t) = max{c(t)} as the feature corresponding to this particular kernel vector. We use varying kernel vectors and window sizes to obtain multiple features.
The process of extracting textual features is as follows -
First, we represent each sentence as the concatenation of vectors of the constituent words. These vectors are the publicly available 300-dimensional word2vec vectors trained on 100 billion words from Google News (Mikolov et al., 2013). The convolution kernels are thus applied to these word vectors instead of individual words. Each sentence is wrapped to a window of 50 words which serves as the input to the CNN.
The CNN has two convolutional layers - the first layer having a kernel size of 3 and 4, with 50 feature maps each and a kernel size 2 with 100 feature maps for the second. The convolution layers are interleaved with pooling layers of dimension 2. We use ReLU as the activation function. The convolution of the CNN over the sentence learns abstract representations of the phrases equipped with implicit semantic information, which with each successive layer spans over increasing number of words and ultimately the entire sentence.
Modality Model
Text text-CNN: Deep Convolutional NeuralNetwork with word embeddings
Video 3d-CNN: 3-dimensional CNNs employed onutterances of the videos Audio openSMILE: Extracts low level audiodescriptors from the audio modality
Table 1: Methods for extracting context independent baseline features from different modalities.
3.1.2 Audio Feature Extraction
Audio features are extracted in 30 Hz frame-rate; we use a sliding window of 100 ms. To compute the features, we use the open-source software
openSMILE (Eyben et al., 2010b) which automatically extracts pitch and voice intensity. Voice normalization is performed and voice intensity is thresholded to identify samples with and without voice. Z-standardization is used to perform voice normalization.
The features extracted by openSMILE consist of several low-level descriptors (LLD) and their statistical functionals. Some of the functionals are amplitude mean, arithmetic mean, root quadratic mean, etc. Taking into account all functionals of each LLD, we obtained 6373 features.
3.1.3 Visual Feature Extraction
We use 3D-CNN to obtain visual features from the video. We hypothesize that 3D-CNN will not only be able to learn relevant features from each frame, but will also be able to learn the changes among given number of consecutive frames.
In the past, 3D-CNN has been successfully applied to object classification on 3D data (Ji et al., 2013). Its ability to achieve state-of-the-art results motivated us to use it.
Let vid ∈ Rc×f×h×w be a video, where c = number of channels in an image (in our case c = 3, since we consider only RGB images), f = number of frames, h = height of the frames, and w = width of the frames. Again, we consider the 3D convolutional filter filt ∈ Rfm×c×fl×fh×fw, where fm = number of feature maps, c = number of channels, fd = number of frames (in other words depth of the filter), fh = height of the filter, and fw = width of the filter. Similar to 2D-CNN, filt slides across video vid and generates output convout ∈ Rfm×c×(f−fd+1)×(h−fh+1)×(w−fw+1). Next, we apply max pooling to convout to select only relevant features. The pooling will be applied only to the last three dimensions of the array convout.
In our experiments, we obtained best results with 32 feature maps (fm) with the filter-size of 5 × 5 × 5 (or fd × fh × fw). In other words, the dimension of the filter is 32 × 3 × 5 × 5 × 5 (or fm × c × fd × fh × fw). Subsequently, we apply max pooling on the output of convolution operation, with window-size being 3×3×3. This is followed by a dense layer of size 300 and softmax. The activations of this dense layer are finally used as the video features for each utterance.
3.2 Context-Dependent Feature Extraction
We hypothesize that, within a video, there is a high probability of utterance relatedness with respect
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
to their sentimental and emotional clues. Since most videos tend to be about a single topic, the utterances within each video are correlated, e.g., due to the development of the speaker’s idea, coreferences, etc. This calls for a model which takes into account such inter-dependencies and the effect these might have on the current utterance. To capture this flow of informational triggers across utterances, we use a LSTM-based recurrent network scheme (Gers, 2001).
3.2.1 Long Short-Term Memory
LSTM is a kind of recurrent neural network (RNN), an extension of conventional feed-forward neural network. Specifically, LSTM cells are capable of modeling long-range dependencies, which other traditional RNNs fail to do given the vanishing gradient issue. Each LSTM cell consists of an input gate i, an output gate o, and a forget gate f , which enables it to remember the error during the error propagation. Current research (Zhou et al., 2016) indicates the benefit of using such networks to incorporate contextual information in the classification process.
In our case, the LSTM network serves the purpose of context-dependent feature extraction by modeling relations among utterances. We term our architecture ‘contextual LSTM’. We propose several architectural variants of it later in the paper.
LSTM
sc-L ST M
Utterance 1 Utterance 2 Utterance !
Context Sensitive Utterance features
Dense Layer
Softmax Output
LSTM LSTM
Context Independent Utterance features
Figure 1: Contextual LSTM network: input features are passed through an unidirectional LSTM layer, followed by a dense layer and then a softmax layer. Categorical cross entropy loss is taken for training. The dense layer activations serve as the output features.
3.2.2 Contextual LSTM Architecture
Let unimodal features have dimension k, each utterance is thus represented by a feature vector xi,t ∈ Rk, where t represents the tth utterance of the video i. For a video, we collect the vectors for all the utterances in it, to get Xi = [xi,1,xi,2, ...,xi,Li] ∈ RLi×k, where Li represents the number of utterances in the video. This matrix Xi serves as the input to the LSTM. Figure 1 demonstrates the functioning of this LSTM module.
In the procedure getLstmFeatures(Xi) of Algorithm 1, each of these utterance xi,t is passed through a LSTM cell using the equations mentioned in line 32 to 37. The output of the LSTM cell hi,t is then fed into a dense layer and finally into a softmax layer (line 38 to 39). The activations of the dense layer zi,t are used as the contextdependent features of contextual LSTM.
3.2.3 Training
The training of the LSTM network is performed using categorical cross entropy on each utterance’s softmax output per video, i.e.,
loss = 1 N
N
∑ n=1
C
∑ c=1 yn,c log2( ˆyn,c),
where N = total number of utterances in a video, yn,c = original output of class c, and ˆyn,c = predicted output.
A dropout layer between the LSTM cell and dense layer is introduced to check overfitting. As the videos do not have same the number of utterances, padding is introduced to serve as neutral utterances. To avoid the proliferation of noise within the network, masking is done on these padded utterances to eliminate their effect in the network. Parameter tuning is done on the train set by splitting it into train and validation components with 80/20% split. RMSprop has been used as the optimizer which is known to resolve Adagrad’s radically diminishing learning rates (Duchi et al., 2011). After feeding the train set to the network, the test set is passed through it to generate their context-dependent features.
Different Network Architectures We consider the following variants of the contextual LSTM architecture in our experiments -
sc-LSTM This variant of the contextual LSTM architecture consists of unidirectional
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
LSTM cells. As this is the simple variant of the contextual LSTM, we termed it as simple contextual LSTM (sc-LSTM)
h-LSTM We also test on an architecture where the dense layer after the LSTM cell is omitted. Thus, the output of the LSTM cell hi,t provides our context-dependent features and the softmax layer provides the classification. We call this architecture hidden LSTM (h-LSTM).
bc-LSTM Bi-directional LSTMs are two unidirectional LSTMs stacked together having opposite directions. Thus, an utterance can get information from other utterances occurring before and after itself in the video. We replaced the regular LSTM with a bi-directional LSTM and named the resulting architecture as bi-directional contextual LSTM (bc-LSTM). The training process of this architecture is similar to sc-LSTM.
uni-SVM In this setting, we first obtain the unimodal features as explained in Section 3.1, concatenate them and then send to a SVM for the final classification. It should be noted that using a gated recurrent unit (GRU) instead of LSTM did not improve the performance.
3.3 Fusion of Modalities
We accomplish multimodal fusion in two different ways as explained below -
3.3.1 Non-hierarchical Framework
In non-hierarchical framework, we concatenate context-independent unimodal features (from Section 3.1) and feed that into the contextual LSTM networks, i.e., sc-LSTM, bc-LSTM, and h-LSTM.
3.3.2 Hierarchical Framework
Contextual unimodal features, taken as input, can further improve performance of the multimodal fusion framework explained in Section 3.3.1. To accomplish this, we propose a hierarchical deep network which comprises of two levels –
Level-1: context-independent unimodal features (from 3.1) are fed to the proposed LSTM network (Section 3.2.2) to get context-sensitive unimodal feature representations for each utterance. Individual LSTM networks are used for each modality.
Level-2: consists of a contextual LSTM network similar to Level-1 but independent in training and computation. Output from each LSTM network in Level-1 are concatenated and fed into
this LSTM network, thus providing an inherent fusion scheme - the prime objective of this level (Fig 2). The performance of the second level banks on the quality of the features from the previous level, with better features aiding the fusion process. Algorithm 1 describes the overall computation for utterance classification. For the hierarchical framework, we train Level 1 and Level 2 successively but separately.
Weight Bias Wi,Wf ,Wc,Wo ∈ Rd×k bi, bf , bc, bo ∈ Rd Pi, Pf , Pc, PoVo ∈ Rd×d bz ∈ Rm
Wz ∈ Rm×d bsft ∈ Rc Wsft ∈ Rc×m
Table 2: Summary of notations used in Algorithm 1. Note: d - dimension of hidden unit. k - dimension of input vectors to LSTM layer . c - number of classes.
4 Experimental Results
4.1 Dataset details
Most of the research in multimodal sentiment analysis is performed on datasets with speaker overlap in train and test splits.
Because each individual has a unique way of expressing emotions and sentiments, finding generic, person-independent features for sentimental analysis is very tricky. In real-world applications, the
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
592
593
594
595
596
597
598
599
Algorithm 1 Proposed Architecture 1: procedure TRAINARCHITECTURE( U, V) 2: Train context-independent models with U 3: for i:[1,M] do ▷ extract baseline features 4: for j:[1,Li] do 5: xi,j ← TextFeatures(ui,j) 6: x ′
i,j ← V ideoFeatures(ui,j) 7: x”i,j ← AudioFeatures(ui,j) 8: Unimodal: 9: Train LSTM at Level-1 with X,X ′ andX”.
10: for i:[1,M] do ▷ unimodal features 11: Zi ← getLSTMFeatures(Xi) 12: Z ′ i ← getLSTMFeatures(X ′
i) 13: Z”i ← getLSTMFeatures(X”i ) 14: Multimodal: 15: for i:[1,M] do 16: for j:[1,Li] do 17: if Non-hierarchical fusion then 18: x∗i,j ← (xi,j ∣∣x ′
i,j ∣∣x”i,j) ▷ concatenation
19: else 20: if Hierarchical fusion then 21: x∗i,j ← (zi,j ∣∣z ′
i,j ∣∣z”i,j) ▷ concatenation 22: Train LSTM at Level-2 with X∗. 23: for i:[1,M] do ▷ multimodal features 24: Z∗i ← getLSTMFeatures(X∗i ) 25: testArchitecture( V) 26: return Z∗
27: procedure TESTARCHITECTURE( V) 28: Similar to training phase. V is passed through the
learnt models to get the features and classification outputs. Table 2 shows the trainable parameters.
29: procedure GETLSTMFEATURES(Xi) ▷ for ith video 30: Zi ← φ 31: for t:[1,Li] do ▷ Table 2 provides notation 32: it ← σ(Wixi,t + Pi.ht−1 + bi) 33: C̃t ← tanh(Wcxi,t + Pcht−1 + bc) 34: ft ← σ(Wfxt + Pfht−1 + bf) 35: Ct ← it ∗ C̃t + ft ∗Ct−1 36: ot ← σ(Woxt + Poht−1 + VoCt + bo) 37: ht ← ot ∗ tanh(Ct) ▷ output of lstm cell 38: zt ← ReLU(Wzht + bz) ▷ dense layer 39: prediction← softmax(Wsftzt + bsft) 40: Zi ← Zi ∪ zt 41: return Zi
model should be robust to person variance but it is very difficult to come up with a generalized model from the behavior of a limited number of individuals To this end, we perform person-independent experiments to emulate unseen conditions. Our train/test splits of the datasets are completely disjoint with respect to speakers.
While testing, our models have to classify emotions and sentiments from utterances by speakers they have never seen before.
IEMOCAP: The IEMOCAP contains the acts of 10 speakers in a two way conversation segmented into utterances. The database contains the following categorical labels: anger, happiness, sadness, neutral, excitement, frustration, fear, surprise, and other, but we take only the first four so as to compare with the state of the art (Rozgic et al., 2012b) and other authors. Videos by the first 8 speakers are considered in the train set. The train/test split details are provided in table 3.
MOSI: The MOSI dataset is a dataset rich in sentimental expressions where 93 persons review topics in English. It contains positive and negative classes as its sentiment labels. The train/validation set comprises of the first 62 individuals in the dataset.
MOUD: This dataset contains product review videos provided by around 55 persons. The reviews are in Spanish (we use Google Translate API1 to get the english transcripts). The utterances are labeled to be either positive, negative or neutral. However, we drop the neutral label to maintain consistency with previous work. The first 59 videos are considered in the train/val set.
Table 3 provides information regarding train/test split of all the datasets. In these splits it is ensured that 1) No two utterances from the train and test splits belong to the same video. 2) The train/test splits have no speaker overlap. This provides the speaker-independent setting.
Table 3 also provides cross dataset split details where the complete datasets of MOSI and MOUD are used for training and testing respectively. The proposed model being used on reviews from different languages allows us to analyze its robustness and generalizability.
It should be noted that the datasets’ individual configuration and splits are same throughout all the experiments (i.e., context-independent unimodal feature extraction, LSTM-based context-
1http://translate.google.com
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
dependent unimodal and multimodal feature extraction and classification).
4.2 Performance of Different Models and Comparisons
In this section, we present unimodal and multimodal sentiment analysis performance of different LSTM network variants as explained in Section 3.2.3 and comparison with the state of the art.
Hierarchical vs Non-hierarchical Fusion Framework - As expected, trained contextual unimodal features help the hierarchical fusion framework to outperform the non-hierarchical framework. Table 4 demonstrates this by comparing both hierarchical and non-hierarchical framework using the bc-LSTM network. Due to this fact, we provide all further analysis and results using the hierarchical framework. Nonhierarchical model outperforms the performance of the baseline uni-SVM. This further leads us to conclude that it is the context-sensitive learning paradigm which plays the key role in improving performance over the baseline.
Comparison among Network Variants - It is to be noted that both sc-LSTM and bc-LSTM perform quite well on the multimodal emotion recognition and sentiment analysis datasets. Since, bcLSTM has access to both the preceding and following information of the utterance sequence, it performs consistently better on all the datasets over sc-LSTM.
The usefulness of the dense layer in improving the performance is prominent from the experimental results as shown in Table 4. The performance improvement is in the range of 0.3% to 1.5% on MOSI and MOUD datasets. On the IEMOCAP dataset, the performance improvement of bc-LSTM and sc-LSTM over h-LSTM is in the range of 1% to 5%.
Comparison with the Baseline and state of the art - Every LSTM network variant has outperformed the baseline uni-SVM on all the datasets by the margin of 2% to 5%(see Table 4). These results prove our initial hypothesis that modeling the contextual dependencies among utterances, which uni-SVM cannot do, improves the classification. The higher performance improvement on the IEMOCAP dataset indicates the necessity of modeling long-range dependencies among the utterances as continuous emotion recognition
is a multiclass sequential problem where a person doesnt frequently change emotions (Wöllmer et al., 2008).
We have implemented and compared with the current state-of-the-art approach proposed by Poria et al. (Poria et al., 2015). In their method, they extracted features from each modality and fed to a multiple kernel learning (MKL) classifier. However, they did not conduct the experiment in speaker-independent manner and also did not consider the contextual relation among the utterances. Experimental results in Table 5 shows that the proposed method has outperformed Poria et al. (Poria et al., 2015) by a significant margin. For the emotion recognition task, we have compared our method with the current state of the art (Rozgic et al., 2012b), who extracted features in a similar fashion to (Poria et al., 2015) did. However, for fusion they used SVM trees.
4.3 Importance of the Modalities
As expected, in all kinds of experiments, bimodal and trimodal models have outperformed unimodal models. Overall, audio modality has performed better than visual on all the datasets. On MOSI and IEMOCAP datasets, textual classifier achieves the best performance over other unimodal classifiers. On IEMOCAP dataset, the unimodal and multimodal classifiers obtained poor performance to classify neutral utterances. Textual modality, combined with non-textual modes boosts the performance in IEMOCAP by a large margin. However, the margin is less in the other datasets.
On the MOUD dataset, textual modality performs worse than audio modality due to the noise introduced in translating Spanish utterances to English. Using Spanish word vectors2 in text-CNN results in an improvement of 10% . Nonetheless, we report results using these translated utterances as opposed to utterances trained on Spanish word vectors, in order to make fair comparison with (Poria et al., 2015).
4.4 Generalization of the Models
To test the generalizability of the models, we have trained our framework on complete MOSI dataset and tested on MOUD dataset (Table 6). The performance was poor for audio and textual modality as the MOUD dataset is in Spanish while the model is trained on MOSI dataset which is in En-
2http://crscardellino.me/SBWCE
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Modality MOSI MOUD IEMOCAP hierarchical (%)
no n-
hi er
(% ) hierarchical (%)
no n-
hi er
(% ) hierarchical (%)
no n-
hi er
(% )
un i-
SV M
hL
ST M
sc -L
ST M
bc -L
ST M
un i-
SV M
hL
ST M
sc -L
ST M
bc -L
ST M
un i-
SV M
hL
ST M
sc -L
ST M
bc -L
ST M
T 75.5 77.4 77.6 78.1 49.5 50.1 51.3 52.1 65.5 68.9 71.4 73.6 V 53.1 55.2 55.6 55.8 46.3 48.0 48.2 48.5 47.0 52.0 52.6 53.2 A 58.5 59.6 59.9 60.3 51.5 56.3 57.5 59.9 52.9 54.4 55.2 57.1
T + V 76.7 78.9 79.9 80.2 78.5 50.2 50.6 51.3 52.2 50.9 68.5 70.3 72.3 75.4 73.2 T + A 75.8 78.3 78.8 79.3 78.2 53.1 56.9 57.4 60.4 55.5 70.1 74.1 75.2 75.6 74.5 V + A 58.6 61.5 61.8 62.1 60.3 62.8 62.9 64.4 65.3 64.2 67.6 67.8 68.2 68.9 67.3 T + V + A 77.9 78.1 78.6 80.3 78.1 66.1 66.4 67.3 68.1 67.0 72.5 73.3 74.2 76.1 73.5
Table 4: Comparison of models mentioned in Section 3.2.3. The table reports the accuracy of classification. Note: non-hier ← Non-hierarchical bc-lstm. For remaining fusion hierarchical fusion framework is used (Section 3.3.2)
Modality Sentiment (%) Emotion on IEMOCAP (%)MOSI MOUD angry happy sad neutral T 78.12 52.17 76.07 78.97 76.23 67.44 V 55.80 48.58 53.15 58.15 55.49 51.26 A 60.31 59.99 58.37 60.45 61.35 52.31
T + V 80.22 52.23 77.24 78.99 78.35 68.15 T + A 79.33 60.39 77.15 79.10 78.10 69.14 V + A 62.17 65.36 68.21 71.97 70.35 62.37 A + V + T 80.30 68.11 77.98 79.31 78.30 69.92 State-of 73.551 63.251 73.10 2 72.402 61.902 58.102-the-art 1by (Poria et al., 2015),2by (Rozgic et al., 2012b)
Table 5: Accuracy % on textual (T), visual (V), audio (A) modality and comparison with the state of the art. For fusion, hierarchical fusion framework was used (Section 3.3.2)
Modality MOSI→MOUDuni-SVM h-LSTM sc-LSTM bc-LSTM T 46.5% 46.5% 46.6% 46.9% V 43.3% 45.5% 48.3% 49.6% A 42.9% 46.0% 46.4% 47.2%
T + V 49.8% 49.8% 49.8% 49.8% T + A 50.4% 50.9% 51.1% 51.3% V + A 46.0% 47.1% 49.3% 49.6% T + V + A 51.1% 52.2% 52.5% 52.7%
Table 6: Cross Dataset comparison. The table reports the accuracy of classification.
glish language. However, notably visual modality performs better than other two modalities in this experiment which signifies that in cross-lingual scenarios facial expressions carry more generalized, robust information than audio and textual modalities. We could not carry out the similar experiment for emotion recognition as no other utterance-level dataset apart from the IEMOCAP was available at the time of our experiments.
4.5 Qualitative Analysis
In some cases the predictions of the proposed method are wrong given the difficulty in recognizing the face and noisy audio signal in the utterances. Also, cases where the sentiment is very weak and non contextual, the proposed approach shows some bias towards its surrounding utter-
ances which further leads to wrong predictions. This can be solved by developing a context aware attention mechanism. In order to have a better understanding on roles of modalities for overall classification, we also have done some qualitative analysis. For example, this utterance - ”who doesn’t have any presence or greatness at all.”, was classified as positive by the audio classifier (“doesn’t” was spoken normally by the speaker, but “presence and greatness at all” was spoken with enthusiasm). However, textual modality caught the negation induced by “doesn’t” and classified correctly. In another utterance “amazing special effects” as there was no jest of enthusiasm in speaker’s voice and face audio-visual classifier failed to identify the positivity of this utterance. On the other textual classifier correctly detected the polarity as positive.
On the other hand, textual classifier classified this sentence - “that like to see comic book characters treated responsibly” as positive, possibly because of the presence of positive phrases such as “like to see”, “responsibly”. However, the high pitch of anger in the person’s voice and the frowning face helps identify this to be a negative utterance.
5 Conclusion
Contextual relationship among the utterances is mostly ignored in the literature. In this paper, we developed a LSTM-based network to extract contextual features from the utterances of a video for multimodal sentiment analysis. The proposed method has outperformed the state of the art and showed significant performance improvement over the baseline. As a part of the future work, we plan to propose LSTM attention model to determine importance of the utterances and contribution of modalities in the sentiment classification.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

Dear Authors  thanks for replying to our review comments, which clarifies some detail questions. I appreciate your promise to publish the code, which will be very helpful to other researchers.   Based on this, i increased my overall score to 4.   Strengths: - well-written - extensive experiments - good results  - Weaknesses: - nothing ground-breaking, application of existing technologies - code not available - results are as could be expected  - General Discussion: - why didn't you use established audio features such as MFCCs?  - Minor Details: - L155 and other places: a LSTM to an LSTM - L160, L216 and other Places: why are there hyphens (-) after the text? - L205: explanation of convolution is not clear - Table1 should appear earlier, on page 2 already cited - L263: is 3D-CNN a standard approach in video processing? alternatives? - L375, 378: the ^ should probably positioned above the y - L380:  to check overfitting  to did you mean  to avoid ? - L403, 408..: put names in     or write them italic, to make it easier to recognize them - L420: a SVM to an SVM - L448: Output ... are to wrong numerus, either  Outputs , or use  is   - L489: superflous whitespace after  layer  - L516, 519:  concatenation  should not be in a new line - L567: why don't you know the exact number of persons? - L626: remove comma after Since - L651: doesnt to does not  - L777: insert  hand, the  after other - References: need some cleanup: L823 superflous whitespace, L831 Munich, L860 what is ACL(1)?, L888 superflous ), L894 Volume, L951 superflous new lines, L956 indent Linguistics properly


Robust Incremental Neural Semantic Graph Parsing
Parsing sentences to linguisticallyexpressive semantic representations is a key goal of Natural Language Processing. Yet statistical parsing has focussed almost exclusively on bilexical dependencies or domain-specific logical forms. We propose a neural encoder-decoder transition-based parser which is the first full-coverage semantic graph parser for Minimal Recursion Semantics (MRS). The model architecture uses stack-based embedding features, predicting graphs jointly with unlexicalized predicates and their token alignments. Our parser is more accurate than attention-based baselines on MRS, and on an additional Abstract Meaning Representation (AMR) benchmark, and GPU batch processing makes it an order of magnitude faster than a high-precision grammar-based parser. Further, the 86.69% Smatch score of our MRS parser is higher than the upper-bound on AMR parsing, making MRS an attractive choice as a semantic representation.
1 000 001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 025 026 027 028 029 030 031 032 033 034 035 036 037 038 039 040 041 042 043 044 045 046 047 048 049
050
051
052
053
054
055
056
057
058
059
060
061
062
063
064
065
066
067
068
069
070
071
072
073
074
075
076
077
078
079
080
081
082
083
084
085
086
087
088
089
090
091
092
093
094
095
096
097
098
099
Robust Incremental Neural Semantic Graph Parsing
Anonymous ACL submission
Abstract
Parsing sentences to linguisticallyexpressive semantic representations is a key goal of Natural Language Processing. Yet statistical parsing has focussed almost exclusively on bilexical dependencies or domain-specific logical forms. We propose a neural encoder-decoder transition-based parser which is the first full-coverage semantic graph parser for Minimal Recursion Semantics (MRS). The model architecture uses stack-based embedding features, predicting graphs jointly with unlexicalized predicates and their token alignments. Our parser is more accurate than attention-based baselines on MRS, and on an additional Abstract Meaning Representation (AMR) benchmark, and GPU batch processing makes it an order of magnitude faster than a high-precision grammar-based parser. Further, the 86.69% Smatch score of our MRS parser is higher than the upper-bound on AMR parsing, making MRS an attractive choice as a semantic representation.
1 Introduction
An important goal of Natural Language Understanding (NLU) is to parse sentences to structured, interpretable meaning representations that can be used for query execution, inference and reasoning. While it has recently been shown that end-to-end models outperform traditional pipeline approaches using syntactic or semantic parsers on many NLU tasks, those parses were frequently relatively shallow, e.g. restricted to projective bilexical dependencies.
In this paper we focus on the robust parsing of linguistically deep semantic representations. The main representation that we use is Minimal Recursion Semantics (MRS) (Copestake et al., 1995, 2005), which serves as the semantic representation of the English Resource Grammar (ERG) (Flickinger, 2000). The only previous approach to parsing and disambiguating full MRS structures (as opposed to bilexical semantic graphs derived from, but simplifying MRS) were based on the ERG (Toutanova et al., 2005); this approach has high precision but incomplete coverage.
Our main contribution is to develop a fast and robust parser for full MRS-based semantic graphs. We exploit the power of global conditioning enabled by deep learning to predict linguistically deep graphs incrementally. The model does not have access to the underlying ERG or syntactic structures from which the MRS analyses were originally derived. We develop parsers for two graph-based conversions of MRS, Elementary Dependency Structure (EDS) (Oepen and Lønning, 2006) and Dependency MRS (DMRS) (Copestake, 2009), of which the latter is inter-convertible with MRS.
Abstract Meaning Representation (AMR) (Banarescu et al., 2013) is a graph-based semantic representation with similar goals to that of MRS. Apart from differences in the choice of which linguistic phenomena are annotated, MRS is a compositional representation explicitly coupled with the syntactic structure of the sentence, while AMR does not assume compositionality or alignment with the sentence structure. AMR parsing has recently received a lot of attention, but the size of the available training data is still relatively small, and inter-annotator agreement has been shown to be relatively low, placing on upper bound of 83% F1 on the expected parser performance. We apply our model to AMR parsing by introducing struc-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
ture (alignments and distinguishing between lexical and non-lexical concepts) that is present explicitly in MRS but not in AMR.
Parsers based on RNNs have achieved state-ofthe-art performance for dependency parsing (Dyer et al., 2015; Kiperwasser and Goldberg, 2016) and constituency parsing (Vinyals et al., 2015b; Dyer et al., 2016; Cross and Huang, 2016b). However, one of the main advantages of deep learning is the ability to make predictions conditioned on the unbounded contexts encoded with RNNs; this enables us to predict more complex structures than have been possible previously, without increasing algorithmic complexity. One of the main reasons for the prevalence of dependency parsing, including semantic dependency parsing (Ivanova et al., 2013), is that it can be performed with efficient and well-understood algorithms. Therefore deep learning gives us the opportunity to perform robust, linguistically deep parsing.
Our parser is a transition-based model for parsing semantic graphs. However, instead of generating arcs over an ordered, fixed set of nodes (the words in the sentence), we generate the nodes, including their labels and alignments to the input tokens, jointly with the transition actions. We use a variant of the arc-eager transition-system that is able to parse graphs and non-planar dependencies. The sentence is encoded with a bidirectional RNN. The transition sequence, seen as a graph linearization, can be predicted with any encoder-decoder model, but we show that using hard attention, predicting the alignment with a pointer network and conditioning explicitly on stack-based features improves performance. In order to deal with data sparsity candidate lemmas are predicted as a preprocessing step, so that the RNN decoder predicts unlexicalized predicates.
We evaluate our parser on DMRS, EDS and AMR graphs. We show that our model architecture improves performance from 79.68% to 84.16% F1 over an attention-based encoderdecoder baseline. Although the model is less accurate that a high-precision grammar-based parser on a test set of sentences parsable by that grammar, our model is an order of magnitude faster due to incremental prediction and a GPU batch processing implementation of the transition system. On AMR parsing our model obtains 60.11% Smatch, an improvement of 8% over an existing neural AMR parser.
2 Deep Meaning Representations
We define a common framework for semantic graphs, in which we can place both MRSbased graph representations (DMRS and EDS) and AMR. In this framework sentence meaning is represented with rooted, labelled, connected, directed graphs. An example graph is visualized in Figure 1. Node labels are referred to as predicates (concepts in AMR) and edge labels as arguments (AMR relations). In addition, a special class of node modifiers, constants, are used to denote the string values of named entities and numbers (including date and time expressions). Every node is aligned to a token or a continuous span of tokens in the sentence the graph corresponds to.
Minimal Recursion Semantics (MRS) is a framework for computational semantics that can be used for parsing or generation (Copestake et al., 2005). The main units of MRS are elementary predications (EPs). An EP consists of a relation (referred to as a predicate), usually corresponding to a single lexeme, and its arguments. Quantification is expressed by relations, not logical operations like ∃ or ∀. One of the distinguishing characteristics of MRS its support for scope underspecification; multiple scope-resolved logical representations can be derived from one MRS structure. MRS was designed to be integrated
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
with feature-based grammars, and has been implemented in English within the framework of Headdriven Phrase Structure Grammar (HPSG) (Pollard and Sag, 1994) in the English Resource Grammar (ERG) (Flickinger, 2000).
MRS can be converted without loss to variablefree dependency graphs, called Dependency MRS (DMRS) (Copestake, 2009; Copestake et al., 2016). A similar graph-based conversion is Elementary Dependency Structures (EDS) (Oepen and Lønning, 2006), which drops the scopeunderspecification machinery, primarily simplifying edge labels. Figure 1 illustrates an EDS graph.
MRS makes an explicit the distinction between lexical and non-lexical predicates (lexical predicates are prefixed by an underscore). Lexical predicates consist of a lemma followed by a coarse part-of-speech tag and an optional sense label. Predicates absent from the ERG lexicon are represented by their surface forms, POS tags and an unknown sense label. All predicates are annotated with an alignment to the character-span of the (untokenized) input sentence. We convert the character-level spans given by MRS to token-level spans. Lexical predicates usually align with the span of the token(s) they represent, while nonlexical predicates can span longer segments. In full MRS predicates are annotated with a set of morphosyntactic features consisting of attributevalue pairs, but we do not currently model these features.
AMR (Banarescu et al., 2013) graphs can be represented in the same framework, despite a number of linguistic differences with MRS. However, information annotated explicitly in MRS is considered as latent in AMR. This include alignments, as well as distinguishing between lexical and nonlexical concepts. AMR predicates are based on PropBank (Palmer et al., 2005), annotated as lemmas plus sense labels, but they form only a subset of concepts. Other concepts are either English words or special keywords, and can correspond to overt lexemes in some cases but not others.
3 Incremental Graph Parsing
We parse sentences to their meaning representations by incrementally predicting semantic graphs together with their alignments. Let e = e1, e2, . . . , eI be a tokenized English sentence, t = t1, t2, . . . , tJ a sequential representation of its graph derivation and a = a1, a2, . . . , aJ an align-
:root( <1> _v_1 :ARG1( <0> named_CARG :BV-of( <0> proper_q ) )
:ARG2 <5> _v_1 :ARG1( <3> _n_1
:BV-of ( <2> _q ) ) :ARG2( <5> pron
:BV-of ( <5> pronoun_q ) ) )
Figure 2: A top-down linearization of the EDS graph in Figure 1, using unlexicalized predicates.
ment sequence consisting of integers in the range 1, . . . , I . We model the conditional distribution p(t,a|e) which decomposes as
J∏ j=1 p(aj |(a, t)1:j−1, e)p(tj |a1:j , t1:j−1, e).
We also predict the end-of-span alignments as a seperate sequence a(e).
3.1 Top-down linearization
We now consider how to linearize the semantic graphs, before defining the neural models to parameterize the parser in section 4. The first approach is to linearize a graph as the pre-order traversal of its spanning tree, starting at a designated root node (see Figure 2). Variants of this approach has been proposed for neural constituency parsing (Vinyals et al., 2015b), logical form prediction (Dong and Lapata, 2016; Jia and Liang, 2016) and AMR parsing (Barzdins and Gosko, 2016; Peng et al., 2017).
In the linearization, labels of edges whose direction are reversed in the spanning tree are marked by adding -of. Edges not included in the spanning tree, referred to as reentrancies, are represented with special with edges whose dependents are dummy nodes pointing back to the original nodes. Our potentially lossy representation represents the edge by repeating the label and alignment of the dependen node, which is recovered heuristically. The alignment does not influence the order of the nodes in this linearization.
3.2 Arc-eager parsing
Figure 1 shows that the semantic graphs we work with can also be interpreted as dependency graphs, as nodes are aligned to sentence tokens. An approach to predicting dependency graphs incrementally that has been used extensively is transition-
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
based parsing (Nivre, 2008). We apply a variant of the arc-eager transition system that has been proposed for graph (as opposed to tree) parsing (Titov et al., 2009; Gómez-Rodrı́guez and Nivre, 2010) to derive a transition-based parser for deep semantic graphs. In dependency parsing the sentence tokens also act as nodes in the graph, but here we need to generate the nodes incrementally as the transitionsystem proceeds, conditioning the generation on the given sentence. Damonte et al. (2016) proposed an arc-eager AMR parser, but the definition of their system is more narrowly defined for AMR graphs.
The transition system consists of a stack of graph nodes being processed and a buffer, holding a single node at a time. The transition actions are shift, left-arc, right-arc and reduce. An example transition sequence is given in Figure 3, together with the stack and buffer after each step. The shift transition moves the element on the buffer to the top of the stack, and generates a predicate and its alignment as the next node on the buffer. Left-arc and right-arc actions add labeled arcs between the buffer and stack top (for DMRS a transition for undirected arcs is included), but do not change the state of the stack or buffer. Finally, reduce pops the top element from the stack, and predicts its end-ofspan alignment if included in the representation. To predict non-planar arcs, we add another transition, which we call cross-arc, which first predicts the stack index of a node which is not on top of the stack, adding an arc between the head of the buffer and that node. Another special transition designates the buffer node as the root.
To derive an oracle for this transition system, it is necessary to determine the order in which the nodes are generated. We consider two different orderings. The first is that of an in-order traversal of the spanning tree, where the node order is determined by the alignment. This leads to a linearization where the only non-planar arcs are reentrancies. The second is to make the ordering nondecreasing with respect to the alignments, while for nodes with the same alignment following the in-order ordering. In an arc-eager oracle arcs are added greedily, while a reduce action can either be performed as soon as the stack top node has been connected to all its dependents, or delayed until it has to reduce to allow the correct parse tree to be formed. In our model the oracle delays reduce, where possible, until the end alignment of
the stack top node spans the node on the buffer. As the span end alignments often cover phrases that they head (e.g. for quantifiers) this gives a natural interpretation to predicting the span end together with the reduce action.
3.3 Lemma prediction
Lexical predicates in MRS consist of a lemma followed by a sense label. Therefore if we can predict the alignment of a graph node as well as the lemma of the aligned word, the prediction required by the decoder can be simplified to only predicting the sense label. We extract a dictionary mapping words to lemmas from the ERG lexicon, which map words to their possible predicates. In combination with a lemmatizer, this can be used to predict a candidate lemma for each token. The same approach is applied to predict constants, together with additional normalizations, e.g. to map numbers to digit strings.
We use the Stanford CoreNLP toolkit (Manning et al., 2014) to tokenize and lemmatize sentences, and tag tokens with the Stanford Named Entity Recognizer (Finkel et al., 2005). For AMR parsing there is not annotated distinction between lexical and non-lexical tokens, so we automaticallyobtained alignments and the graph structure to classifiy concepts as lexical or non-lexical. The lexicon is restricted to Propbank (Palmer et al., 2005) predicates, so for other concepts we extract a lexicon from the training data.
4 Encoder-Decoder Models
4.1 Sentence encoder
The sentence e is encoded with a bidirectional RNN. We use a standard LSTM architecture without peephole connections (Jozefowicz et al., 2015). For every token e we embed its word, POS tag and named entity (NE) tag as vectors xw, xt and xn, respectively.
The embeddings are concatenated and passed through a linear transformation,
g(e) =W (x)[xw;xt;xn] + b x
such that g(e) has the same dimension as the LSTM. We don’t embed word lemmas separately for the unlexicalized representations. Each input position i is represented by a hidden state hi, which is the concatenation of its forward and backward LSTM state vectors.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Action Stack Buffer Arc added init(0, named CARG) [ ] (0, 0, named CARG) - sh(0, proper q) [(0, 0, named CARG)] (1, 0, proper q) - la(BV) [(0, 0, named CARG)] (1, 0, proper q) (1, BV, 0) sh(1, v 1) [(0, 0, named CARG), (1, 0, proper q)] (2, 1, v 1) - re [(0, 0, named CARG)] (2, 1, v 1) - la(ARG1) [(0, 0, named CARG)] (2, 1, v 1) (2, ARG1, 0)
Figure 3: First part of the transition sequence for parsing the graph in Figure 1. The transitions are shift (sh), reduce (re), left arc (la) and right arc (ra). The action taken at each step is given, along with the state of the stack and buffer after the action is applied, and any arcs added. Shift transitions generate the alignment and predicate of the next graph node. Items on the stack and buffer have the form (node index, alignment, predicate label), and arcs are of the form (head index, argument label, dependent index).
4.2 Hard attention decoder
We model the alignment of graph nodes to sentence tokens, a, as a random variable. For the arceager model, aj corresponds to the alignment of the node of the buffer after action tj is executed. The distribution of tj is over all transitions and predicate predictions (for shifts), predicted with a single softmax.
Let sj be the RNN decoder hidden state at output position j. We initialize s0 with the final state of the backward encoder LSTM. The alignment is predicted with a pointer network (Vinyals et al., 2015a).
The logits are computed with an MLP scoring the decoder hidden state against each of the encoder hidden states (for i = 1, . . . , I),
uij = v T tanh(W (1)hi +W (2)sj).
The alignment distribution is then estimated by
p(aj = i|a1:j−1, t1:j−1, e) = softmax(uij).
To predict the next transition ti, the output vector is conditioned on the encoder state vector haj , corresponding to the alignment:
oj =W (3)sj +W (4)haj vj = R (d)oj + b (d),
where R(d) and b(d) are the output representation matrix and bias vector, respectively.
The transition distribution is then given by
p(tj |a1:j , t1:j−1, e) = softmax(vj).
Let v(t) be the embedding of decoder symbol t. The RNN state at the next time-step is computed
as
dj+1 =W (5)v(tj) +W (6)haj
sj+1 = RNN(dj+1, sj).
The end-of-span alignment a(e)j for MRS-based graphs is predicted with another pointer network. The end alignment of a token is predicted only when a node is reduced from the stack, therefore this alignment is not observed at each time-step; it is also not fed back into the model.
The hard attention approach, based on supervised alignments, can be contrasted to soft attention, which learns to attend over the input without supervision. The attention is computed as with hard attention, as αij = softmax(u i j). However instead of making a hard selection, a weighted average over the encoder vectors is computed as qj = ∑i=I i=1 α i jhi. This vector is used instead of haj for prediction and feeding to the next timestep.
4.3 Stack-based model
We extend the hard attention model to include features based on the transition system stack. Elements on the stack can be represented by the encoder biLSTM representations corresponding to the tokens they are aligned to. We include vectors for the top of the stack and the buffer, the latter which is predicted by the hard attention. This approach is similar to the features proposed by Kiperwasser and Goldberg (2016) and Cross and Huang (2016a) for dependency parsing. The layer that computes the output vector is extended to
oj =W (3)sj +W (4)haj +W (7)hst0
where st0 is the sentence alignment index of the top element on the buffer. The input layer to the
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
next RNN time-step is similarly extended to
dj+1 =W (5)v(tj) +W (6)hbuf +W (8)hst0 ,
where buf is the buffer alignment after tj is executed.
Our implementation of the stack-based model enables batch processing in static computation graphs, similar to Bowman et al. (2016). We maintain a stack of alignment indexes for each element in the batch, which is updated inside the computation graph after each parsing action.
We perform greedy decoding. For the stackbased model we ensure that if the stack is empty, the next transition predicted has to be shift. For the other models we ensure that the output is wellformed during post-processing by robustly skipping over out-of-place symbols or inserting missing ones.
5 Experiments
5.1 Data
DeepBank (Flickinger et al., 2012) is HPSG and MRS annotation of the Penn Treebank Wall Street Journal (WSJ) corpus text, developed following an approach known as dynamic treebanking (Oepen et al., 2004), that couples treebank annotation with grammar development, in the case of the ERG. This approach has been shown to lead to high inter-annotator agreement: 0.94 against 0.71 for AMR (Bender et al., 2015). Parses are only provided for sentences for which the ERG has an analysis acceptable to the annotator – this means that we cannot evaluate parsing accuracy for sentences which the ERG cannot parse (approximately 15% of the original corpus).
We use Deepbank version 1.1, corresponding to ERG 12141, following the suggested split of sections 0 to 19 as training data data, 20 for development and 21 for testing. The gold-annotated training data consists of 35,315 sentences. We use the pyDelphin library2 and software provided with the ERG to extract DMRS and EDS graphs.
For AMR parsing we use LDC2015E86, the dataset released for the SemEval 2016 AMR parsing Shared Task (May, 2016). This data includes newswire, weblog and discussion forum text. The training set has 16,144 sentences. We obtain align-
1http://svn.delph-in.net/erg/tags/ 1214/
2https://github.com/delph-in/pydelphin
ments using the rule-based JAMR aligner (Flanigan et al., 2014).
5.2 Evaluation
Dridan and Oepen (2011) proposed an evaluation metric called Elementary Dependency Matching (EDM) for MRS graphs. EDM computes the F1scores of tuples of predicates and arguments. A predicate tuples consist of the label and character span of a predicate, while an argument tuple consist of the character spans of the head and dependent of the relation, together with the argument label. In order to tolerate tokenization differences with respect to punctuation, we allow span pairs whose ends differ by 1 character to be matched.
EDM can be contrasted with the Smatch metric (Cai and Knight, 2013) for evaluating AMR graphs. This evaluation does not rely on sentence alignments; instead it performs inference over graph alignments to estimate the maximum F1-score obtainable from a 1-1 matching between the predicted and gold graph nodes.
5.3 Model Setup
Our models are implemented in TensorFlow (Abadi et al., 2015). For training we use Adam (Kingma and Ba, 2015) with learning rate 0.01 and batch-size 64. Gradients norms are clipped to 5.0 (Pascanu et al., 2013). We use single-layer LSTMs with dropout of 0.3 (tuned on the development set) on input and output connections. We use encoder and decoder embeddings of size 256, and POS and NE tag embeddings of size 32, For DMRS and EDS graphs the hidden units size is set to 256, for AMR it is 128. This configuration, found using grid search and heuristic search within the range of models that fit into a single GPU, gave the best performance on the development set under multiple graph linearizations. Encoder word embeddings are initialized (in the first 100 dimensions) with pre-trained order-sensitive embeddings (Ling et al., 2015). Singletons in the encoder input is replaced with an unknown word symbol with probability 0.5 for each iteration.
5.4 MRS parsing results
We compare different linearizations and model architectures for parsing DMRS on the development data, showing that our approach is more accurate than baseline neural approaches. We report
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Model EDM EDM-Pred EDM-Arg TD lex 81.44 85.20 76.87 TD unlex 81.72 85.59 77.04 AE lex 81.35 85.79 76.02 AE delex 82.56 86.76 77.54
Table 1: DMRS development set results for attention-based encoder-decoder models with alignments encoded in the linearization, for topdown (TD) and arc-eager (AE) linearizations, and lexicalized and unlexicalized predicate prediction.
Model EDM EDM-Pred EDM-Arg TD soft 81.53 85.32 76.94 TD hard 82.75 86.37 78.37 AE hard 84.65 87.77 80.85 AE stack 85.28 88.38 81.51
Table 2: DMRS development set results of pointer-augmented encoder-decoder models with hard and soft attention architectures.
EDM F1 scores, as well as EDM scores for predicate (EDM-Pred) and argument (EDM-Arg) prediction.
First we report results using a vanilla encoderdecoder model (Table 1). We compare the topdown and arc-eager linearizations, as well as the effect of delexicalizing the predicates (factorizing lemmas out of of the predicate labels and predicting them separately.) In both cases constants are predicted with a dictionary lookup based on the predicted spans; for predicates not in the lexicon an unknown-label is predicted and the words and POS tags are recovered during post-processing.
The arc-eager unlexicalized linearization gives the best performance, down linearization, even though the model has to learn to model the transition stack inside the recurrent states without any supervision of the semantics of transition actions. The unlexicalized models are more accurate, mostly due to their ability to generalize to sparse or unseen predicates occurring in the lexicon. For the arc-eager representation, the oracle EDM is 99% for the lexicalized representation and 98.06% for the delexicalized representation. The remaining errors are due to discrepancies between the tokenization used by our system and the ERG tokenization. The unlexicalized models are also faster to train, as the decoder’s output vocabulary is much smaller, reducing the expense of computing softmaxes over large vocabularies.
Next we consider models that predict the alignments with pointer networks, contrasting soft and hard attention models (Table 2). The results show that the arc-eager models performs better than those based on top-down representation. For the arc-eager model we use hard attention, due to the natural interpretation of the alignment prediction corresponding to the transition system. The arc-eager stack-based architecture improves further over the model that purely relies on the hard attention.
We compared the effect of different orderings of the predicates for the arc-eager model: The monotone ordering (with non-decreasing alignments) performs 0.44 EDM better that the in-order ordering, despite having to parse more non-planar dependencies. We also compare the predicate prediction against a hard attention model that only predicts predicates (in monotone order) together with their start spans. This model obtains 91.36% F1 on predicates together with their start spans with the delexicalized model, compared to 88.22% for lexicalized predicates and 91.65% for the full parsing model.
We present test set results for various metrics in Table 3. We compare the performance of our neural baseline and stack-based decoders with the ACE3 ERG-based parser. Another approach to robust MRS parsing has previously been proposed (Zhang et al., 2014), but no comparable results or implementation is available.
Despite the promising performance of the model there is still a gap between the accuracy of our parser and ACE. One reason for this is that the test set sentences will arguably be easier for ACE to parse as their choice was restricted by the grammar that ACE uses. EDM metrics excluding endspan prediction (-Start) show that our parser has relatively more difficulty in parsing end-span predictions than the grammar-based parser.
3http://sweaglesw.org/linguistics/ace/
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Model ArcEager ACE EDM 85.48 89.58 Smatch 86.50 93.52 EDM-Pred 88.14 91.82 EDM-Arg 82.20 86.92
Table 4: EDS parsing test set results.
Model Concept F1 Smatch TD no point 65.55 57.95 TD soft 66.44 59.36 TD soft delex 67.07 59.88 AE hard delex 72.86 59.83 AE stack delex 73.69 61.21
Table 5: Development set results for AMR parsing. All the models are pointer-based, except where indicated otherwise.
We also evaluate the speed of our model compared with ACE. For the unbatched version of our model, the stack-based parser parses 41.63 tokens per second, while the baseline with linearized spans parses 31.65 tokens per second. The batched (stack-based) implementation parses 529.42 tokens per second, using a batch size of 128. In comparison, the setting of ACE for which we reported accuracies parses 7.47 tokens per second. By restricting the memory usage of ACE, which restricts its coverage, we see that ACE can parse 11.07 tokens per second at 87.7% coverage, and 15.11 tokens per second at 77.8% coverage.
Finally we report results for parsing EDS (Table 4) The EDS parsing task is slightly simpler than DMRS, due to the absence of label and structural annotations that allows for the recovery of MRS, including scope underspecification. These additional labels are harder to predict without access to a grammar.
5.5 AMR parsing
We now apply the same approach to AMR parsing. Results on the development set are given in Table 5. The arc-eager-based models again give better performance, mainly due to improved concept prediction accuracy. However, concept prediction remains a weakness of the model; Damonte et al. (2016) reports that state-of-the-art AMR parsers score between 79% and 83% on concept prediction. In contrast to DMRS parsing, the in-order linearization gives better results than the monotone one. We hypothesize that this is due to the
non-compositional nature of AMR. We show that even our neural baseline outperforms existing results for neural AMR parsing. The arc-eager model does not perform better than the top-down linearization, which we hypothesize is due to noise in the automatically-obtained alignments.
We report test set results on LDC2015E86 test set (Table 6). Our best neural model outperforms the baseline JAMR parser (Flanigan et al., 2014), but still lags behind the performance of state-ofthe-art AMR parsers such as CAMR (Wang et al., 2016). However these models make extensive use of external resources, including syntax trees and semantic role labelling. We see that our attentionbased encoder-decoder model already performs better than previous sequence-to-sequence AMR parsers (Barzdins and Gosko, 2016; Peng et al., 2017), and the arc-eager model boost accuracy further. Our model also outperforms a Synchronous Hyperedge Replacement Grammar model (Peng and Gildea, 2016) which is comparable as it does not make extensive use of external resources.
6 Conclusion
In this paper we advance the state of parsing by employing deep learning techniques to parse sentence to linguistically expressive semantic representations that have not previously been parsed in an end-to-end fashion. We presented a robust, wide-coverage parser for MRS that is faster than existing parsers and amenable to batch processing. We believe that there are many future avenues to explore to further increase the accuracy of such parsers, including different training objectives, more structured architectures and semisupervised learning.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths: The paper proposes an end-to-end neural model for semantic graph parsing, based on a well-designed transition system.  The work is interesting, learning semantic representations of DMRS, which is capable of resolving semantics such as scope underspecification. This work shows a new scheme for computational semantics, benefiting from an end-to-end transition-based incremental framework, which resolves the parsing with low cost.  - Weaknesses:   My major concern is that the paper only gives a very common introduction for the definition of DMRS and EP, and the example even makes me a little confused because I cannot see anything special for DMRS. The description can be a little more detailed, I think. However, upon the space limitation, it is understandable. The same problem exists for the transition system of the parsing model. If I do not have any background of MRS and EP, I can hardly learn something from the paper, just seeing that this paper is very good.  - General Discussion:   Overall, this paper is very interesting to me. I like the DMRS for semantic parsing very much and like the paper very much. Hope that the open-source codes and datasets can make this line of research being a hot topic.