Human communication is multimodal in nature; it is through multiple modalities such as language, voice, and facial expressions, that opinions and emotions are expressed. Data in this domain exhibits complex multi-relational and temporal interactions. Learning from this data is a fundamentally challenging research problem. In this paper, we propose Modal-Temporal Attention Graph (MTAG). MTAG is an interpretable graph-based neural model that provides a suitable framework for analyzing multimodal sequential data. We first introduce a procedure to convert unaligned multimodal sequence data into a graph with heterogeneous nodes and edges that captures the rich interactions across modalities and through time. Then, a novel graph fusion operation, called MTAG fusion, along with a dynamic pruning and read-out technique, is designed to efficiently process this modal-temporal graph and capture various interactions. By learning to focus only on the important interactions within the graph, MTAG achieves state-of-the-art performance on multimodal sentiment analysis and emotion recognition benchmarks, while utilizing significantly fewer model parameters. 1 1 Introduction With recent advances in machine learning research, analysis of multimodal sequential data has become increasingly prominent. At the core of modeling this form of data, there are the fundamental research challenges of fusion and alignment . Fusion is the process of blending information from multiple modalities. It is usually preceded by alignment, which is the process of finding temporal relations between the modalities. An important research area that exhibits this form of data is multimodal language analysis, where sequential modalities of  Equal contribution 1 Code is available at https://github.com/ jedyang97/MTAG . language, vision, and acoustic are present. These three modalities carry the communicative information and interact with each other through time; e.g. positive word at the beginning of an utterance may be the cause of a smile at the end. When analyzing such multimodal sequential data, it is crucial to build models that perform both fusion and alignment accurately and efficiently by a) aligning arbitrarily distributed asynchronous modalities in an interpretable manner, b) efficiently accounting for short and long-range dependencies, c) explicitly modeling the inter-modal interactions between it was a very good movie (emphasis) (neutral) = Positional Embedding  Node Construction  MTAG Graph Fusion and Pruning Text Audio Video FFNVFFNVFFNVFFNTFFNTFFNTFFNTFFNTFFNTFFNAFFNAMTAG Prune MTAG Prune Remove Isolated Node  " Layers  Multimodal & Temporal Edge Construction Figure 2: The 3-stage MTAG framework: Node Construction, Edge Construction and Fusion+Pruning. [Node Construction] Each modality's features are first passed through a distinct Feed-Forward-Network to be mapped into the same embedding size. Then, a positional embedding is added to each transformed feature based on its position in its own modality, so that temporal information are encoded. The features are now nodes in the graph. [Edge Construction] We then apply an algorithm to construct edges among these nodes by appropriately indexing each edge with a modal type and a temporal type. [Fusion+Pruning] Finally, we pass the graph into the MTAG module to learn interactions across modality and time. The output graph with updated node embeddings and pruned edges can be passed to downstream modules, e.g. a Multi-layer Perceptron, to complete specific tasks such as regression or classification. the modalities while simultaneously accounting for intra-modal dynamics. In this paper, we propose MTAG (Modal-Temporal Attention Graph). MTAG is capable of both fusion and alignment of asynchronously distributed multimodal sequential data. Modalities do not need to be pre-aligned, nor do they need to follow similar sampling rate. MTAG can capture interactions of various types across any number of modalities all at once, comparing to previous methods that model bi-modal interactions at a time (Tsai et al., 2019a). At its core, MTAG utilizes an efficient trimodal-temporal graph fusion operation. Coupled with our proposed dynamic pruning technique, MTAG learns a parameter-efficient and interpretable graph. In our experiments, we use two unaligned multimodal emotion recognition and sentiment analysis benchmarks: IEMOCAP (Busso et al., 2008) and CMU-MOSI (Zadeh et al., 2016). The proposed MTAG model achieves state-of-the-art performance with far fewer parameters. Subsequently, we visualize the learned relations between modalities and explore the underlying dynamics of multimodal language data. Our model incorporates all three modalities in both alignment and fusion, a fact that is also substantiated in our ablation studies. 2 Related Works Human Multimodal Language Analysis Analyzing human multimodal language involves learning from data across multiple heterogeneous sources that are often asynchronous, i.e. language, visual, and acoustic modalities that each uses a different sampling rate. Earlier works assumed multimodal sequences are aligned based on word boundaries (Lazaridou et al., 2015; Ngiam et al., 2011; Gu et al., 2018; Dumpala et al., 2019; Pham et al., 2019) and applied fusion methods for aligned sequences. To date, modeling unaligned multimodal language sequences remains understudied, except for (Tsai et al., 2019a; Khare et al., 2020; Zheng et al., 2020), which used cross-modal Transformers to model unaligned multimodal language sequences. However, the cross-modal Transformer module is a bi-modal operation that only account for two modalities' input at a time. In Tsai et al. (2019a), the authors used multiple cross-modal Transformers and applies late fusion to obtain trimodal features, resulting in a large amount of parameters needed to retain original modality information. Other works that also used cross-modal Transformer architecture for include Yang et al. (2020); Siriwardhana et al. (2020). In contrast to the existing works, our proposed graph method, with very small amount of model parameters, can aggregate information from multiple (more than 2) modalities at early stage by building edges between the corresponding modalities, allowing richer and more complex representation of the interactions to be learned. Graph Neural Networks Graph Neural Network (GNN) was introduced in (Gori et al., 2005; Scarselli et al., 2008) with an attempt to extend deep neural networks to handle graph-structured data. Since then, there has been an increasing research interest on generalizing deep neural net-work's operations such as convolution (Kipf and Welling, 2016; Schlichtkrull et al., 2017; Hamilton et al., 2017), recurrence (Nicolicioiu et al., 2019), and attention (Velickovic et al., 2018) to graph. Recently, several heterogeneous GNN methods (Wang et al., 2019a; Wei et al., 2019; Shi et al., 2016) have been proposed. The heterogeneous nodes referred in these works consist of uni-modal views of multiple data generating sources (such as movie metadata node, audience metadata node, etc.), whereas in our case the graph nodes represent multimodal views of a single data generating source (visual, acoustic, textual nodes from a single speaking person). In the NLP domain, multimodal GNN methods (Khademi, 2020; Yin et al., 2020) on tasks such as Visual Question Answering and Machine Translation. However, these settings still differ from ours because they focused on static images and short text which, unlike the multimodal video data in our case, do not exhibit long-term temporal dependencies across modalities. Based on these findings, we discovered there has been little research using graph-based methods for modeling unaligned, multimodal language sequences, which includes video, audio and text. In this paper, we demonstrate our proposed MTAG method can effectively model such unaligned, multimodal sequential data. 3 MTAG In this section, we describe our proposed framework: Modal Temporal Attention Graph (MTAG) for unaligned multimodal language sequences. We describe how we formulate the multimodal data into a graph G ( V , E ) , and the MTAG fusion operation that operates on G . In essence, our graph formulation by design alleviates the need for any hard alignments, and combined with MTAG fusion, allows nodes from one modality to interact Notation Explanation v i Node i e ij Edge from v i to v j N i Neighbor nodes incident into v i x i Initial node feature for v i x (cid:48) i Transformed node feature for v i  i Node type for v i  ij Edge modality type for e ij  ij Edge temporal type for e ij M  i Node type specific transformation matrix a  ij , ij Edge type specific learnable attention vector  i,j Raw attention score of node pair ( v i , v j )  i,j Attention weight of node pair ( v i , v j ), normalized over N i z i Node output feature for v i k Prune percentage h Index of multi-head attention head H Number of total attention heads Table 1: Terminologies used in this paper. freely with nodes from all other modalities at the same time, breaking the limitation of only modeling pairwise modality interactions in previous works. Figure 2 gives a high-level overview of the framework. 3.1 Node Construction As illustrated in Figure 2, each modality's input feature vectors are first passed through a modality-specific Feed-Forward-Network. This allows feature embeddings from different modalities to be transformed into the same dimension. A positional embedding (details in Appendix A) is then added (separately for each modality) to each embedding to encode temporal information. The output of this operation becomes a node v i in the graph. Each node is marked with a modality identifier  i , where  i  { Audio, Video, Text } in our case. 3.2 Edge Construction In this section, we describe our design of modality edges and temporal edges. For a given node of a particular modality, its interactions with nodes from different modalities should be considered differently. For example, given a Video node, its interaction with an Audio node should be different from that with a Text node. In addition, the temporal order of the nodes also plays a key role in multimodal analysis (Poria et al., 2017). For example, a transition from a frown to a smile (   ) may imply a positive sentiment, whereas a transition from a smile to a frown (   ) may imply a negative sentiment. Therefore, interactions between nodes that appear in different temporal orders should also be considered differently. In GNNs, the edges define how node features are aggregated within a graph. In order to encapsulate the diverse types of node interactions, we assign edge types to each edge so that information can be aggregated differently on different types of edges. By indexing edges with edge types, different modal and temporal interactions between nodes can be addressed separately. Multimodal Edges. As we make no assumption about prior alignment of the modalities, the graph is initialized to be a fully connected graph. We use e ij to represent an edge from v i to v j . We assign e ij with a modality type identifier  ij = (  i   j ) . For example, an edge pointing from a Video node to a Text node will be marked with type  ij = ( Video  Text ) . Temporal Edges. In addition to  ij , we also assign a temporal label  ij to each e ij . Depending on the temporal order of v i and v j connected by e ij , we determine the value of  ij to be either of { past, present, future } . For nodes from the same modality, the temporal orders can be easily determined by comparing their order of occurrences. To determine the temporal orders for nodes across different modalities, we first roughly align the two modalities with our pseudo-alignment. Then the temporal order can be simply read out. Pseudo-Alignment. As mentioned above, it is simple to determine the temporal edge types for nodes in a single modality. However, there is no clear definition of earlier" or later" across two modalities, due to the unaligned nature of our input sequences. To this end, we introduce the pseudo-alignment heuristic that coarsely defines the past, present and future connections between nodes across two modalities. Given a node v i from one modality  i , our pseudo-alignment first determines a set of nodes V i,present in the other modality that can be aligned to v i and considered as present". All nodes in the other modality that exists after V i,present are considered future" V i,future , and all those before are considered V i,past . Once the coarse temporal order is established, the cross-modal temporal edge types can be easily determined. Figure 3 shows an example of such pseudo-alignment, and more details regarding the calculations can be found in Appendix A.2. Time FutureEdgePresentEdgePastEdge Vision Node TextNode PastNodes PresentNodes FutureNodes Figure 3: An example of the pseudo-alignment between two unaligned sequences. We first align the longer sequence to the shorter one as uniformly as possible. Then the aligned nodes from the longer sequence becomes the V i,present for node v i in the shorter sequence. V i,past and V i,future can then be determined accordingly. 3.3 Fusion and Pruning 3.3.1 MTAG Fusion With our formulation of the graph, we design the MTAG fusion operation that can digest our graph data with various node and edge types, and thus model the modal-temporal interactions. An algorithm of our method is shown in Algorithm 1 and a visual illustration is given in Figure 4. Specifi-cally, for each neighbor node v j that has an edge incident into a center node v i , we compute a raw attention score  [ h ] ,i,j based on that edge's modality and temporal type:  [ h ] ,i,j = LeakyRelu ( a  ji , ji [ h ]  [ x (cid:48) i (cid:107) x (cid:48) j ]) (1) where [ || ] denotes the concatenation of two column vectors into one long column vector. The [ h ] index is used to distinguish which multi-head attention head is being used. Note that a  ji , ji [ h ] depends on both the modality and temporal edge types of e ji . This results in 27 edge types (9 types of modality interaction  3 types of temporal interaction). We normalize the raw attention scores over all neighbor nodes v j with Softmax so that the normalized attention weight sums to 1 to preserve the scale of the node features in the graph.  [ h ] ,i,j = exp(  [ h ] ,i,j ) (cid:80) k  N i exp(  [ h ] ,i,k ) (2) Then, we perform node feature aggregation for each node v i following: z i = H concat h =1 ( (cid:88) j  N i  [ h ] ,i,j x (cid:48) j ) (3) Algorithm 1: MTAG with edge pruning 1 Feature transformation x (cid:48) i  M  i x i ,  i 2 for h = 1 ...H do 3 for j  N i  i do 4 calculate raw attention score using modalityand temporal-edge-type specific parameters:  [ h ] ,i,j = LeakyRelu ( a  ji , ji [ h ]  [ x (cid:48) i (cid:107) x (cid:48) j ]) 5 normalize raw attention scores over N i  i to get attention weight  [ h ] ,i,j 6 calculate node output feature z i = H concat h =1 ( (cid:80) j  N i  i  [ h ] ,i,j x (cid:48) j ) 7 calculate average attention weight across all heads  i,j = 1 H (cid:80) Hh =1 (  [ h ] ,i,j ) 8 sort  i,j and delete the edges with the smallest k % average attention weight from N i  i , obtaining N (cid:48) i 9 return z i , N (cid:48) i  i where N i defines the neighbors of v i and hyperparameter H is the number of total attention heads. z i now becomes the new node embedding for node v i . After aggregation, v i transformed from a node with unimodal information into a node encoding the diverse modal-temporal interactions between v i and its neighbors (illustrated by the mixing of colors of the nodes in Figure 2). We desgined the operation to have H multi-head attention heads because the heterogeneous input data of the multimodal graph could be of different scales, making the variance of the data high. Adding multi-head attention could help stabilize the behavior of the operation. 3.3.2 Dynamic Edge Pruning Our graph formulation models interactions for all 27 edge types. This design results in a very large number of edges in the graph, making the computation graph difficult to fit into GPU memories. More importantly, when there are so many edges, it is hard to avoid some of these edges from inducing spurious correlations and distracting the model from focusing on the truly important interactions (Lee et al., 2019; Knyazev et al., 2019). To address these challenges, we propose to dynamically prune edges as the model learns the graph. Specifically, after each layer of MTAG, we have the attention weight  [ h ] ,i,j for each attention head h and for i 3 4 5 # ["]$&,()*+*,-# ["]&&,()*+*,-1 2 # ["]$&,(1+-Figure 4: Visualization of the MTAG operation around a single node. The text on each edge indicates which attention vector is used for that edge. Purple triangle represents a video node, green circle represents a text node and blue square represents an audio node. each edge e ij . We take the average of the attention weights over the attention heads:  ij = 1 HH (cid:88) h =1 (  [ h ] ,i,j ) (4) Then, we sort  ij and delete k % edges with the smallest attention weights, where k is a hyperparameter. These deleted edges will no longer be calculated in the next MTAG fusion layer. Our ablation study in Section 5.2 empirically verifies the effectiveness of this approach by comparing to no pruning and random pruning. At the end of the MTAG fusion process, we need to read out information scattered in the nodes into a single vector so that we can pass it through a classification head. Recall that the pruning process drops edges in the graph. If all edges incident into a node have been dropped, then it means that node was not updated based on its neighbors. In that case, we simply ignore that node in the readout process. We readout the graph by averaging all the surviving nodes' output features into one vector. This vector is then passed to a 3-layer Multi-Layer-Perceptron (MLP) to make the final prediction. We empirically evaluate MTAG model on two datasets: IEMOCAP (Busso et al., 2008) and CMU-MOSI (Zadeh et al., 2016); both are well-known datasets used by prior works (Liang et al., 2018; Pham et al., 2019; Tsai et al., 2019b,a) to benchmark multimodal emotion recognition and sentiment analysis. IEMOCAP IEMOCAP is a multimodal emotion recognition dataset consisting of 10K videos. The task we chose is the 4-way multilabel emotion classification, classifying into happy, sad, angry and neutral. For train split, the positive/negative label ratio for each emotion is 954:1763, 338:2379, 690:2027 and 735:1982. For the test split, the ratio is 383:555, 135:803, 193:745 and 227:711. Due to this unbalanced distribution of the the labels, we use F1 score as a better metric for comparison. CMU-MOSI CMU Multimodal Opinion Sentiment Intensity is a multimodal sentiment analysis dataset with 2,199 movie review video clips. Each video clip is labeled with real-valued sentiment score within [  3 , +3] , with +3 being a very positive sentiment and  3 a very negative one. Following previous works (Tsai et al., 2019a), we report five metrics: 7-class classification accuracy (Acc 7 ), binary classification accuracy (Acc 2 , posi-tive/negative sentiments), F1 score, Mean Absolute Error (MAE) and the correlation of the model's prediction with human. We follow prior works (Tsai et al., 2019a) to evaluate on both of the above unaligned datasets, where original audio and video features are used, resulting in variable sequence lengths across modalities. For both datasets, the multimodal features are extracted from the textual (GloVe word embeddings Model \ Metirc Acc 7  Acc 2  F1  MAE  Corr  (Unaligned) CMU-MOSI Sentiment CTC+EF-LSTM 31.0 73.6 74.5 1.078 0.542 LF-LSTM 33.7 77.6 77.8 0.988 0.624 CTC+MCTN 32.7 75.9 76.4 0.991 0.613 CTC+RAVEN 31.7 72.7 73.1 1.076 0.544 MulT 39.1 81.1 81.0 0.889 0.686 MTAG (ours) 38.9 82.3 82.1 0.866 0.722 Table 3: Results on unaligned CMU-MOSI. (Pennington et al., 2014)), visual (Facet (iMotions, 2017)), and acoustic (COVAREP (Degottex et al., 2014)) data modalities. For basleine evaluations, we use Early Fusion LSTM (EF-LSTM) and Late Fusion LSTM (LF-LSTM) (Tsai et al., 2019a) as baselines. In addition, we compare our model against similar methods as in previous works (Tsai et al., 2019a), which combine a Connectionist Temporal Classification (CTC) loss (Graves et al., 2006) with the preexisting methods such as EF-LSTM, MCTN (Pham et al., 2019), RAVEN (Wang et al., 2019b). Shown in Table 2 and Table 3, MTAG substantially out-performs previous methods on unaligned IEMOCAP benchmark and CMU-MOSI benchmark on most of the metrics. MTAG also achieves on-par performance on the Acc 7 metric on CMU-MOSI benchmark. With an extremely small number of parameters, our model is able to learn better alignment and fusion for multimodal sentiment analysis task. Details regarding our model and hyper-parameters can be found in the Appendix A. Parameter Efficiency (MTAG vs MulT) We discover that MTAG is a highly parameter-efficient model. A comparison of model parameters between MTAG and MulT (Tsai et al., 2019a) (previ-ous state of the art) is shown in Table 4. The hyperparameter used for this comparison can be found in the Appendix. With only a fraction ( 6 . 25% ) of MulT's parameters, MTAG is able to achieve on-par, and in most cases superior performance on the two datasets. This demonstrates the parameter efficiency of our method. Qualitative Analysis The attention weights on the graph edges forms a natural way to interpret our model. We visualize the edges to probe what MTAG has learned. The following case study is a randomly selected video clip from the CMU-MOSI validation set. We observe the phenomena shown below is a general trend. In Figure 5, we show an example of the asymmetric bi-modal relations between vision and text. We observe that our model picks on meaningful relations between words such as  I really enjoyed " and facial expressions such as raising eyebrow, highlighted in the red dashed boxes in Figure 5a. Our model can also learn long-range correlation between  I really enjoyed " and head nodding. Interestingly, we discover that strong relations that are not detected by vision-to-text edges can be recovered by the text-to-vision edges. This advocates the design of the multi-type edges, which allows the model to learn different relations independently that can complement one another. Figure 1 gives a holistic view of the attention weights among all three modalities. We observe a pattern where almost all edges involve the text modality. A possible explanation for this observation is that the text is the dominant modality with respect to the sentiment analysis task. This hypothesis is verified by the ablation study in Sec. 5.3. Meanwhile, there appears to be very small amount of edges connecting directly between vision and audio, indicating that there might be little meaningful correlation between them. This resonates with our ablation studies in Table 5, where vision and audio combined produce the lowest bi-modal performance. Under such circumstance, our MTAG learns to kill direct audio-vision relations and instead fuse their information indirectly using the text modality as a proxy, whereas previous methods such as MulT keeps audio-vision attentions alive along the way, introducing possible spurious relations that could distract model learning. We conduct ablation study using unalgined CMU-MOSI dataset. MTAG Full Model implements multimodal temporal edge types, adopts TopK Ablation Acc 2  F1  MAE  Edge Types No Edge Types 82.4 82.5 0.937 Multimodal Edges Only 85.6 85.7 0.859 Temporal Edges Only 85.2 85.2 0.887 Pruning Random Pruning Keep 80% 75.5 74.5 1.080 No Pruning 84.7 84.7 0.908 Modalities Language Only 81.5 81.4 0.911 Vision Only 57.0 57.1 1.41 Audio Only 58.1 58.1 1.37 Vision, Audio 62.0 59.2 1.360 Language, Audio 85.9 85.7 0.915 Language, Vision 86.6 86.6 0.896 Full Model, All Modalities 87.0 87.0 0.859 Table 5: Ablation on unaligned CMU-MOSI validation set. edge pruning that keeps edges with top 80% edge weights, and includes all three modalities as its input. Table 5 shows the performance. We present research questions (RQs) as follows and discuss how ablation studies address them. We first study the effect of edge types on our model performance. As we incrementally add in multimodal and temporal edge types, our model's performance continues to increase. The model with 27 edge types performs the best under all metrics. By dedicating one attention vector a  ji , ji to each edge, MTAG can model each complex relation individually, without having one relation interfering another. As shown in Figure 5 and Table 5, such design enhances multimodal fusion and alignment, helps maintain long-range dependencies in multimodal sequences, and yields better results. We compare our TopK edge pruning to no pruning and random pruning to demonstrates it effectiveness. We find that TopK pruning exceeds both no pruning and random pruning models in every aspect. It is clear that, by selectively keeping the top 80% most important edges, our model learns more IR e a ll y E n j oy e d D o i ng T h e m F o r T h e P h ill y DM ov i e C l ub S o IT hough t O h W h y N o t U m Emphasizes on " I really enjoyed " and raises eyebrow Neutral facial Expression Nods head while raising voice on " Oh why not?! " Video Order Text Order meaningful representations than randomly keeping 80%. Our model also beats the one where no pruning is applied, which attests to our assumption and observation from previous work (Lee et al., 2019; Knyazev et al., 2019) that spurious correlations do exist and can distract model from focusing on important interactions. Therefore, by pruning away the spurious relations, the model learned a better representation of the interactions, while using significantly fewer computation resources. Lastly, we study the impact of different modality combinations used in our model. As shown in Table 5, we find that adding a modality consistently brings performance gains to our model. Through the addition of individual modalities, we find that adding the text modality gives the most significant performance gain, indicating that text may be the most dominant modality for our task. This can also be qualitative confirmed by seeing the concentrated edge weights around text modality in Figure 1. This observation also conforms with the observations seen in prior works (Tsai et al., 2019a; Pham et al., 2019). On the contrary, adding audio only brings marginal performance gain. Overall, this ablation study demonstrates that all modalities are beneficial for our model to learn better multimodal representations. In this paper, we presented the Modal-Temporal Attention Graph (MTAG). We showed that MTAG is an interpretable model that is capable of both fusion and alignment. It achieves similar to SOTA performance on two publicly available datasets for emotion recognition and sentiment analysis while utilizing substantially lower number of parameters than a transformer-based model such as MulT. We thank Jianing Qian, Xiaochuang Han and Haop-ing Bai at CMU and the anonymous reviewers at NAACL for providing helpful discussions and feedbacks. This material is based upon work partially supported by BMW of North America, the National Science Foundation and National Institutes of Health. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of BMW of North America, National Science Foundation or National Institutes of Health, and no official endorsement should be inferred.
abstain abstain abstain objective abstain objective abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain other other abstain abstain abstain abstain method abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result other abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result result result abstain abstain abstain result abstain abstain abstain abstain abstain method other abstain abstain method abstain method abstain abstain abstain method method method abstain method abstain method abstain abstain abstain abstain abstain method result abstain other other other