Jin 1 , Andrzej Cichocki 2 3 , Qibin Zhao 4 , Wanzeng Kong Machine Collaborative Intelligence of Zhejiang Province, and Technology, Hangzhou Dianzi University, China Science and Technology, Moscow, Russia Institute, Polish Academy of Science, Warsaw, Poland Advanced Intelligence Project, RIKEN hdutangjiajia, Jxuanyu599 } @163.com } @hdu.edu.cn } @riken.jp { a.cichocki, Abstract Multimodal sentiment analysis is the challenging research area that attends to the fusion of multiple heterogeneous modalities. The main challenge is the occurrence of some missing modalities during the multimodal fusion procedure. However, the existing techniques require all modalities as input, thus are sensitive to missing modalities at predicting time. In this work, the coupled-translation fusion network (CTFN) is firstly proposed to model bi-direction interplay via couple learning, ensuring the robustness in respect to missing modalities. Specifically, the cyclic consistency constraint is presented to improve the translation performance, allowing us directly to discard decoder and only embraces encoder of Transformer. This could contribute to a much lighter model. Due to the couple learning, CTFN is able to conduct bi-direction cross-modality intercorrelation parallelly. Based on CTFN, a hierarchical architecture is further established to exploit multiple bi-direction translations, leading to double multimodal fusing embeddings compared with traditional translation methods. Moreover, the convolution block is utilized to further highlight explicit interactions among those translations. For evaluation, CTFN was verified on two multimodal benchmarks with extensive ablation studies. The experiments demonstrate that the proposed framework achieves state-of-the-art or often competitive performance. Additionally, CTFN still maintains robustness when considering missing modality. Sentiment analysis has witnessed many significant advances in the artificial intelligence community, in which text (Yadollahi et al., 2017), visual (Kahou et al., 2016), and acoustic (Luo et al., 2019) modalities are primarily employed to the related research  Equal contribution respectively, allowing to exploit the human emotional characteristic and intention effectively (Deng et al., 2018). Intuitively, due to the consistency and complementarity among different sources, the joint representation attend to reason about multimodal messages, which are capable of boosting the performance of the specific task (Pan et al., 2016; Gebru et al., 2017; Al Hanai et al., 2018). Multimodal fusion procedure is to incorporate multiple knowledge for predicting a precise and proper outcome (Baltrusaitis et al., 2018). Historically, the existing fusion has been done generally by leveraging the model-agnostic process, considering the early fusion, late fusion, and hybrid fusion technique (Poria et al., 2017a). Among those, early fusion focussed on the concatenation of the unimodal presentation (D'mello and Kory, 2015). On the contrast, late fusion performs the integration at the decision level, by voting among all the model results (Shutova et al., 2016). As to the hybrid fusion, the output comes from the combination of the early fusion and unimodal prediction (Lan et al., 2014). Nevertheless, multimodal sentiment sequences often consists of unaligned properties, and the traditional fusion manners are failed to take the heterogeneity and misalignment into account carefully, which raises a question on investigating the more sophisticated models and estimating emotional information. (Tsai et al., 2020; Niu et al., 2017). Recently, Transformer-based multimodal fusion framework has been developed to address the above issues with the help of multi-head attention mechanism (Rahman et al., 2020; Le et al., 2019; Tsai et al., 2019). By introducing the standard Transformer network (Vaswani et al., 2017) as the basis, Tsai et al. (Tsai et al., 2019) captured the integrations directly from unaligned multimodal streams in an end-to-end fashion, latently adapted streams from one modality to another with the cross-modal 5302 (cid:1787)(cid:1815)(cid:1804)(cid:1801)(cid:1812)(cid:1809)(cid:1820)(cid:1825) (cid:2778) Cycle ConsistencyConstraint Cycle ConsistencyConstraint (cid:1779)(cid:1814)(cid:1803)(cid:1815)(cid:1804)(cid:1805)(cid:1818) Fusion Embedding (cid:1779)(cid:1814)(cid:1803)(cid:1815)(cid:1804)(cid:1805)(cid:1818) Fusion Embedding (cid:1787)(cid:1815)(cid:1804)(cid:1801)(cid:1812)(cid:1809)(cid:1820)(cid:1825) (cid:2779) attention module, regardless of the need for alignment. Furthermore, Wang et al. (Wang et al., 2020) proposed a parallel Transformer unit, allowing to explore the correlation between multimodal knowledge effectively. However, the decoder component of standard Transformer is employed to improve the translation performance, which may lead to some redundancy. Moreover, the explicit interaction among cross-modality translations were not considered. Essentially, compared to our CTFN, their architecture require access to all modalities as inputs for exploring multimodal interplay with the sequential fusion strategy, thus are rather sensitive in the case of multiple missing modalities. In this paper, CTFN is proposed to model bidirectional interplay based on coupled learning, ensuring the robustness in respect to missing modalities. Specifically, the cyclic consistency constraint is proposed to improve the translation performance, allowing us directly to discard decoder and only embrace encoder of Transformer. This could contribute to a much lighter model. Thanks to the couple learning, CTFN is able to conduct bi-direction cross-modality intercorrelation parallelly. Take CTFN as a basis, a hierarchical architecture is established to exploit modality-guidance translation. Then, the convolution fusion block is presented to further explore the explicit correlation among the above translations. Importantly, based on the parallel fusion strategy, our CTFN model still provides flexibility and robustness when considering only one input modality. For evaluation, CTFN was verified on two multimodal sentiment benchmarks, CMU-MOSI (Zadeh et al., 2016) and MELD (Poria et al., 2019). The experiments demonstrate that CTFN could achieve the state-of-the-art or even better performance compared to the baseline models. We also provide several extended ablation studies, to investigate intrinsic properties of the proposed model. The off-the-shelf multimodal sentiment fusion architecture comprises two leading groups: translation-based and non-translation based model. Non-translation based: Recently, RNN-based models, considering GRU and LSTM, have received significant advances in exploiting the context-aware information across the data (Yang et al., 2016; Agarwal et al., 2019). bc  LST M (Poria et al., 2017b) and GME  LST M (Chung et al., 2014) presented a LSTM-based model to retrieve contextual information, where the unimodal features are concatenated into a unit one as the input information. Similarly, MELD  base (Po-ria et al., 2019) leveraged the concatenation of audio and textual features on the input layer, and employed GRU to model sentimental context. In contrast, CHF usion (Majumder et al., 2018) employed the RNN-based hierarchical structure to draw fine-grained local correlations among the modalities, and the empirical evidence illustrates superior advances compared to the simple concatenation of unimodal presentation. On the basis of RNN, MMMU  BA (Ghosal et al., 2018) further employed multimodal attention block to absorb the contribution of all the neighboring utterances, which demonstrates that the attention mechanism 5303 Reconstruction Error (cid:1865)(cid:1861)(cid:1866)(cid:513) (cid:2180) (cid:2204) (cid:3398)(cid:1794)(cid:1818)(cid:1801)(cid:1814) (cid:1775)(cid:1372)(cid:1796) (cid:4666)(cid:2180) (cid:3028) (cid:1314)(cid:4667) (cid:513) BiGRU & Dense Layer 1 Layer L Layer L Layer 1 BiGRU & Dense   (cid:2180) (cid:3028) (cid:1794)(cid:1818)(cid:1801)(cid:1814) (cid:1775)(cid:1372)(cid:1796) (cid:4666)(cid:2180) (cid:3028) (cid:1314)(cid:4667) (cid:2180) (cid:3049) (cid:1794)(cid:1818)(cid:1801)(cid:1814) (cid:1796)(cid:1372)(cid:1775) (cid:4666)(cid:2180) (cid:3049) (cid:1314)(cid:4667) (cid:2180) (cid:3028) (cid:1314) (cid:2180) (cid:3049) (cid:1314) (cid:1794)(cid:1818)(cid:1801)(cid:1814) (cid:1775)(cid:1372)(cid:1796) (cid:1794)(cid:1818)(cid:1801)(cid:1814) (cid:1796)(cid:1372)(cid:1775) Layer (cid:2191)(cid:4666)(cid:2235)(cid:1372) (cid:2236)(cid:4667) LayerNorm Multi-head LayerNorm Position Feed-forward (cid:2173) (cid:2235) (cid:2167) (cid:2235) (cid:2178) (cid:2235) Addition Addition (cid:2180) (cid:2235) (cid:2171)(cid:2203)(cid:2202)(cid:2198)(cid:2203)(cid:2202) (cid:2235)(cid:1372)(cid:2236) Reconstruction Error (cid:1865)(cid:1861)(cid:1866)(cid:513) (cid:2180) (cid:2183) (cid:3398)(cid:1794)(cid:1818)(cid:1801)(cid:1814) (cid:1796)(cid:1372)(cid:1775) (cid:4666)(cid:2180) (cid:3049) (cid:1314)(cid:4667) (cid:513) Discriminator (cid:1865)(cid:1861)(cid:1866)(cid:513) (cid:2180) (cid:2183) (cid:3398)(cid:2180) (cid:2183) (cid:1314) (cid:513) Discriminator (cid:1865)(cid:1861)(cid:1866)(cid:513) (cid:2180) (cid:2204) (cid:3398)(cid:2180) (cid:2204) (cid:1314) (cid:513) Figure 2: CTFN: X a and X v refer to the features of modality audio and video respectively. can utilize the neighborhood contribution for integrating the contextual information. However, all these methods are suitable for the low-level presentation within the single modality with a non-translation manner, which may be easily sensitive to the noisy terms and missing information in the sources. Translation-based model: Inspired by the re-cent success of sequence to sequence (Seq2Seq) models (Lin et al., 2019; ? ) in machine translation, (Pham et al., 2019) and (Pham et al., 2018) presented multimodal fusion model via the essential insight that translates from a source modality to a target modality, which is able to capture much more robust associations across multiple modalities. MCT N model incorporated a cyclic translation module to retrieve the robust joint representation between modalities in a sequential manner, e.g., the language information firstly associated with the visual modality, and latently translated into the acoustic modality. Compared with the MCT N , Seq 2 Seq 2 Sent introduced a hierarchical fusion model using the Seq2Seq methods. For the first layer, the joint representation of a modality pair is treated as an input sequence for the next Seq2Seq layer in an attempt to decode the third modality. Inspired by the success of the Transformer-based model, Tsai et al. introduced a directional cross-modality attention module to extend the standard Transformer network. Follow the basic idea of Tsai et al. , Wang et al. provided a novel multimodal fusion cell which is comprised of two standard Transformers, embracing the association with a modality pair during the forward and backward translation implicitly. However, all existing models adopt sequential multimodal fusion architecture, which requires all modalities as input, therefore they can be sensitive to the case of multiple missing modalities. Moreover, the explicit interactions among cross-modality translations were not considered. In this section, we firstly present CTFN (Figure 2), which is capable of exploring bi-direction cross-modality translation via couple learning. On the basis of CTFN, a hierarchical architecture is established to exploit multiple bi-direction translations, leading to double multimodal fusing embeddings (Figure 4). Then, the convolutional fusion block (Figure 3) is applied to further highlight explicit correlation among cross-modality translations. The two benchmarks consist of three modalities, audio, video and textual modality. Specifically, the above utterance-level modalities are denoted as X a  RT a  d a , X v  RT v  d v and X t  RT t  d t , respectively. The number of utterances is presented as T i ( i  { a, v, t } ) , and d i ( i  { a, v, t } ) stands for the dimension of the unimodality features. For simplicity, we consider two unimodality presentation X a and X v explored from audio (A) and video (V), respectively. In the primal process of CTFN, we focus on learning a directional translator T ran A  V ( X a , X v ) for translating the modality audio to video. Then, the dual process aims to learn an inverse directional translator T ran V  A ( X v , X a ) , allowing for the translation from modality video to audio. Inspired by the suc-Tran (cid:2157)(cid:1374)(cid:2176) Tran (cid:2157) (cid:1374) (cid:2178) Concatenate (cid:1839) (cid:3028)(cid:3047) (cid:1839) (cid:3028)(cid:3049) Modality T Modality V Modality AT e m p o r a l D o m a i n Convolutional Fusion Block Feature Domain Figure 3: Multimodal convolutional fusion block: M at  RT  F at and M av  RT  F av refer to the cross-modality translations, where T and F  are the size of time and feature domain respectively. cess of Transformer in Natural Language Processing, the encoder of Transformer is introduced to our model as the translation block, which is an efficient and adaptive manner for retrieving the long-range interplay along the temporal domain. Importantly, the cyclic consistency constraint is presented to improve the translation performance. And due to the couple learning, CTFN is able to combine primal and dual process into a coupled structure, ensuring the robustness in respect to missing modalities. For the primal task, X a  RT a  d a is firstly delivered to a densely connected layer for receiving a linear transformation X a  RT a  L a , where L a is the output dimension of the linear layer. And the corresponding query matrix, key matrix and value matrix are denoted as Q a = X a WQ a  RT a  L a , K a = X a WK a  RT a  L a , V a = X a WV a  RT a  L a , where WQ a  RL a  L a , WK a  RL a  L a and WV a  RL a  L a are weight matrixes. The translation from modality A to V is performed as X v , = T ran A  V ( X a , X v )  RT a  L v , where X v , refers to the fake X v , and  L a is the scale coefficient. Note that the input X a is directly delivered to the translation process, while the input X v is used to analyze the difference between real data X v and fake output X v , . Subsequently, X v , is passed through the T ran V  A , leading to the reconstruct output X a , = T ran V  A ( X v , , X a ) , and the X a is only used to calculate the diversity between the real and reconstruct data. RT v  d v , X a , = T ran V  A ( X v , X a )  RT a  L a , and reconstructed representation X v , = T ran A  V ( X a , , X v )  RT v  L v . Essentially, T ran A  V and T ran V  A are implemented by several sequential encoder layers. During the translation period, we hypothesize that intermediate encoder layer contains the cross-modality fusion information and effectively balance the contribution of two modalities. Hence, the output of the middle encoder layer T ran A  V [ L/ 2] and T ran V  A [ L/ 2] stand for the multimodal fusion knowledge, where L refers to the number of layers, and when L is odd number, then L = L + 1 . As for the model reward, the primal process has an immediate reward r p = (cid:4) X a  T ran V  A ( X v , ) (cid:4) F , and the dual step related reward is r d = (cid:4) X v  T ran A  V ( X a , ) (cid:4) F , indicating the similarity between the real data and the reconstructed output of the translator. For simplicity, a linear transformation module is adopted to combine the primal and dual step reward into a total model reward, e.g., r all = r p + (1   ) r d , where  is employed to balance the contribution between dual and primal block. Additionally, the loss functions utilized in the coupled-translation multimodal fusion block are defined as follows: l A  V ( X a , X v ) = (cid:3) Tran A  V ( X a , X v )  X v (cid:3) F + (cid:3) Tran A  V ( X a , , X v )  X v (cid:3) F l V  A ( X v , X a ) = (cid:3) Tran V  A ( X v , X a )  X a (cid:3) F + (cid:3) Tran V  A ( X v , , X a )  X a (cid:3) F l A  V = l A  V ( X a , X v ) + (1   ) l V  A ( X v , X a ) , (2) where l A  V ( X a , X v ) and l V  A ( X v , X a ) refer to the training loss of the primal and dual translator respectively, and l A  V stands for the loss of bi-directional translator unit. Essentially, when the training process of all coupled-translation blocks 5305 V i d e o T e x t It is very interesting! are finished, our model only needs one input modality at predicting time, without the help of target modalities. Indeed, l A  V indicates the cycle-consistency constraint in our couple learning model. The cycle-consistency is well-known, which refers to combination of forward and backward cycle-consistency. However, our goal is to solve missing modality problem in multi-modal learning, which cannot be achieved by applying cycle-consistency straightforward. This is because that introducing this strict cycle-consistency to CTFN fail to effectively associate primal task with dual task of the couple learning model. To solve this problem, we relaxed constraint of original cycle-consistency by using a parameter   ' to balance the contribution of forward and backward cycle-consistency, leading to a much more flexible cycle-consistency. Thanks to the great flexibility of new proposed cycle-consistency, we could adaptively and adequately associate primal with dual task, resulting in much more balanced consistency among modalities. Based on CTFN, each modality is treated as the source moment for ( M  1) times, which means that each modality holds ( M  1) directional translations, { T ran modality source  modality m } Mm =1 , where M refers to the total number of modalities. For instance, given modality audio, we can retrieve the following two modality-guidance translations: [ T ran a  vL/ 2 , video , ] = T ran a  v ( audio, video ) [ T ran a  tL/ 2 , text , ] = T ran a  t ( audio, text ) . (3) Note that audio plays a key role in different cross-modality translations, and provides the strong guidance for capturing various cross-modality interplay. For blending the contribution of source modality (audio) effectively, a convolution fusion block is incorporated to explore explicit and local correlation among modality-guidance translations. Initially, the two cross-modality intermediate correlations T ran audio  vedioL/ 2 and T ran audio  textL/ 2 are concatenated along the temporal domain into a unit representation, where the size of time sequence is equal ( T a = T v = T t ), thus the concatenation is of size T a  ( L v + L t ) : Z concat = T ran a  vL/ 2  T ran a  tL/ 2 . (4) Subsequently, the temporal convolution is employed to further retrieve explicit interactions among cross-modality translations. Specifically, we adopt a 1D temporal convolutional layer to exploit the local patten in a light manner:  Z concat = Conv 1 D ( Z concat , K concat )  RT a  L d , (5) where K concat is the size of the convolutional kernel, and L d is the length of the cross-modality integration dimension. The temporal kernel is used to perform the convolutional operation along the feature dimension, allowing to further exploit local interplay among cross-modality translations. That is to say, the local interplay fully exploits the contribution from modality-guidance translations. On the basis of CTFN and convolutional multimodal fusion network, a hierarchical architecture was proposed for exploiting multiple bi-direction translations, leading to double multimodal fusing embeddings. For instance, given M modalities, our model could achieve double 5306 Audio Input Modality (cid:1839) (cid:2183)(cid:1372)(cid:2204) (cid:1839) (cid:2204)(cid:1372)(cid:2183) (cid:1839) (cid:2202)(cid:1372)(cid:2204) (cid:1839) (cid:2204)(cid:1372)(cid:2202) (cid:1839) (cid:2202)(cid:1372)(cid:2183) (cid:1839) (cid:2183)(cid:1372)(cid:2202) Multimodal Convolution Fusion (cid:2176)(cid:2200)(cid:2183)(cid:2196) (cid:2157)(cid:1372)(cid:2176) (cid:2176)(cid:2200)(cid:2183)(cid:2196) (cid:2157)(cid:1372)(cid:2178) (cid:2176)(cid:2200)(cid:2183)(cid:2196) (cid:2178)(cid:1372)(cid:2176) (cid:2176)(cid:2200)(cid:2183)(cid:2196) (cid:2176)(cid:1372)(cid:2157) (cid:2176)(cid:2200)(cid:2183)(cid:2196) (cid:2178)(cid:1372)(cid:2157) (cid:2176)(cid:2200)(cid:2183)(cid:2196) (cid:2176)(cid:1372)(cid:2178) (cid:2180) (cid:3028) (cid:2180) (cid:3028) (cid:2180) (cid:3049) (cid:1314) (cid:2180) (cid:3049) (cid:1314) (cid:2180) (cid:3047) (cid:1314) (cid:2180) (cid:3047) (cid:1314) Figure 5: We only employ a single input modality (audio) to do the multimodal fusion task during the predicting period. C 2 M embeddings. As illustrated in Figure 4, the proposed architecture consists of three CTFNs T ran A  V , T ran A  T and T ran V  T . Considering the contribution of the guidance (source) modality, the modality-guidance translations are denoted as T ran T  A  V = [ T ran L/ 2 A  V , T ran L/ 2 A  T ] , T ran T  V  A = [ T ran L/ 2 V  T , T ran L/ 2 V  A ] , and T ran A  T  V = [ T ran L/ 2 T  A , T ran L/ 2 T  V ] , respectively. Similarly, when taking the contribution of target modalities into account, corresponding modality-guidance translations are illustrated as T ran T  A  V = [ T ran L/ 2 V  A , T ran L/ 2 T  A ] , T ran T  V  A = [ T ran L/ 2 T  V , T ran L/ 2 A  V ] , and T ran A  T  V = [ T ran L/ 2 A  T , T ran L/ 2 V  T ] , respectively. Subsequently, the convolutional fusion layer is used to further exploit explicit local interplay among modality-guidance translations associated with the same source/target modality, which can fully leverage the contribution of source/target modality. Essentially, as demonstrated in Figure 4, our model has 12+1 loss constraints in total, which includes 3 CTFNs, each one has 4 training loss (primal & dual translator training loss), and 1 classification loss. However, we do not need to balance these targets together, which is achieved by our training strategy that 3 CTFNs are trained individually. For each CTFN, one hyper-parameter   ' is introduced to balance the loss of primal translator and dual translator, and this hyper-parameter is shared among 3 CTFNs. Hence, 3 CTFNs only need 1 hyper-parameter to balance the training loss, which is easy to be tuned. The classification loss is used for training the classifier on the 3 CTFNs's outputs. Datasets. CMU-MOSI consists of 2199 opinion video clips from online sharing websites (e.g., YouTube). Each utterance of the video clip is annotated with a specific sentimental label of positive or negative in the range scale of [  3 , +3] . The corresponding training, validation, and testing size refer to division set (1284 , 229 , 686) . Additionally, the same speaker will not appear in both training and testing sets, allowing to exploit speaker-independent joint representations. MELD dataset contains 13000 utterances from the famous TV-series F riends . Each utterance is annotated with emotion and sentiment labels, considering 7 classes of emotion tag (anger, disgust, fear, joy, neutral, sadness, and surprise) and 3 sentimental tendency levels (positive, neutral, and negative). Hence, the original dataset can be denoted as MELD (Senti-ment) and MELD (Emotion) with respect to the data annotation, we only verified our model on the MELD (Sentiment). Note that CMU-MOSI and MELD are the public and widely-used datasets which have been aligned and segmented already. Features. For CMU-MOSI dataset, we adopt the same preprocess manner mentioned in MFN (Zadeh et al., 2018) to extract the low-level representation of multimodal data, and synchronized at the utterance level that in consistent with text modality. For MELD benchmark, we follow 5307 Models CMU-MOSI MELD (Sentiment) Bi-modality Tri-modality Bi-modality (video, audio) (text, video) (text, audio) (text, audio, video) (text, audio) GME-LSTM (Chung et al., 2014) 52.90 74.30 73.50 76.50 66.46 bc-LSTM (Poria et al., 2017b) 56.52 78.59 78.86 79.26 66.09 MELD-based (Poria et al., 2019) 54.79 76.60 76.99 79.19 66.68 CHFusion (Majumder et al., 2018) 54.49 74.77 78.54 76.51 65.85 MMMU-BA (Ghosal et al., 2018) 57.45 80.85 79.92 81.25 65.56 SeqSeq2Sent (Pham et al., 2018) 58.00 67.00 66.00 70.00 63.84 MCTN (Pham et al., 2019) 53.10 76.80 76.40 79.30 66.27 TransModality (Wang et al., 2020) 59.97 80.58 81.25 82.71 67.04 CTFN (ours, L=1) 62.20 80.49 81.4 80.18 67.82 CTFN (ours, L=3) 63.11 81.55 82.16 82.77 67.78 CTFN (ours, L=6) 64.48 80.79 81.71 81.10 67.24 Table 1: Comparison of performance results for sentiment analysis on CMU-MOSI and MELD (Sentiment) benchmark using various SOTA models. the related work of MELD, in which the 300-dimensional GloVe (Pennington et al., 2014) text vectors are fed into a 1D-CNN (Chen et al., 2017) layer to extract textual representation, and audio-based descriptors are explored with the popular toolkit openSMILE (Eyben et al., 2010), while visual features were not taken into account for the sentiment analysis. Comparisons. We introduced the translation-based and non-translation based models to this work as the baselines. Translation-based: Multimodal Cyclic Translation Network (MCTN), Sequence to Sequence for Sentiment (Seq2Seq2Sent), Multimodal Sentiment Analysis with Transformer (TransModality). And non-translation based: bidirectional contextual LSTM (bc-LSTM), Gated Embedding LSTM (GME-LSTM), Multimodal EmotionLines Dataset baseline model (MELD-base), Hierarchical Fusion with Context Modeling (CHFusion), Multi-Modal Multi-Utterance -Bi-Modal Attention (MMMU-BA). Performance comparison with state-of-the-art models. Firstly, we analyzed the performance between state-of-the-art baselines and our proposed model. The bottom rows in Table 1 indicate the effectiveness and superiority of our model. Particularly, on CMU-MOSI dataset, CTFN exceeded the previous best TransModality on (video, audio) by a margin of 4.51. Additionally, on MELD (Senti-ment) dataset, the empirical improvement of CTFN was 0.78. It is interesting to note that the improvement of (video, audio) is more significant than (text, video) and (text, audio). This implies that coupled-translation structure is capable of decreasing the risk of interference between video and audio efficiently, and further leverage the explicit consistency between auxiliary features. As for (text, audio, video), CTFN exceeds the previous best TransModality with an improvement of 0.06, leading to a comparable performance. Indeed, for the same tri-modality fusion task, TransModality needs 4 encoders and 4 decoders, while CTFN only requires 6 encoders. It should be emphasized that the cyclic consistency mechanism could contribute to a much lighter model, as well as the more effective bi-directional translation. In addition, compared to the bi-modality setting, the tri-modality case achieved the improvement of 0.61, indicating the benefits brought by hierarchical architecture and convolution fusion. Effect of CTFN with missing modalities. Existing translation-based manners focus only on the join representation between modalities, and ignore the potential occurrence of the missing modalities. Therefore, we analyzed how does missing modality may affect the final performance of CTFN and the sequential translation-based model SeqSeq2Sent. Note that SeqSeq2Sent only employs LSTM to analyze uni-modality rather than the translation-based method. Specifically, we take the hierarchical architecture combined with three CTFNs as 5308 the testing model. From the Table 2, we observe that compared to the setting (text, audio, video), the text-based settings { (audio, video, text), (au-dio, video, text), (audio,video, text) } seem to reach the comparable result with only a relatively small performance drop. On the contrast, when text was missing, the model has a relatively large performance drop, which implies that language modality contains much more discriminative sentimental message than audio and video, leading to the significantly better performance. Essentially, the performance of (audio,video, text) demonstrates that hierarchical CTFN is able to maintain robustness and consistency when considering only a single input modality. In other words, the cyclic consistency mechanism allows CTFN to fully exploit the cross-modality interplay, thus hierarchical CTFN could transmit the single modality to various pre-trained CTFNs for retrieving multimodal fusion message. Effect of the translation direction. In this paper, we propose a coupled-translation block, which aims to embrace fusion messages from the bidirectional translation process. Hence, we are interested to investigate the impact of translation direction. Figure 6 depicts the performance of various translations, considering (audio, text), (au-dio, video), and (text, video) translation. For the (audio, text) instance, the translation text  audio achieves better performance than audio  text . Similarly, the translation text  video surpasses the result of video  text. However, the performance of audio  video and video  audio seems to be quite similar. The superiority of text  video and text  audio may demonstrate that text modality possesses much more sentimental information. Effect of the translator layer. As each translator is comprised of several sequential encoder layers. In this part, we assume that the output representation of a specific layer may affect the performance of the proposed model. For simplicity, we perform the related task on CMU-MOSI with the setting of (a, v, t), as well as the (t, a) on MELD (Sentiment). Initially, we retrieve the embedding from the specific layer, where the layer ranges from 1 to L (L is the total number of the layer). In Figure 7, it is interesting to note that the model reaches the peak value at layer 5 on CMU-MOSI, which means that the output of the fifth layer embraces the most discriminative fusion message. In comparison, on MELD (Sentiment), the model achieves the best performance at layer 1 , which may imply that the simple translator associated with only one layer is able to capture the joint representation for the simple case (text, audio). In conclusion, the lower encoder layer may involve low-level characteristics of interplay, while the higher encoder layer may embrace the explicit messages. Additionally, the output of the specific layer of the encoder lies on the corresponding task and dataset. We tried also (text, audio) on MOSI, and CTFN maximizes the performance at layer 3 . Compared to (text, audio, video), (text, audio) is the relatively simple case, thus the lower encoder layer may is sufficient to demonstrate the interaction between text and audio. Effect of concatenation strategy of translation. In our work, those translations associated MOSI dataset A cc u r ac y 0 22.5 45 67.5 90 Source-based Target-based [ A  T , A  V ] [ T  A , V  A ] [ V  A , V  T ] [ T  V , A  V ] [ T  A , T  V ] [ A  T , V  T ] MOSI dataset F 1 S c o r e 0 22.5 45 67.5 90 Source-based Target-based [ A  T , A  V ] [ T  A , V  A ] [ V  A , V  T ] [ T  V , A  V ] [ T  A , T  V ] [ A  T , V  T ] Figure 8: Effect of concatenation strategy via source/target modality on MOSI. with the same guidance (source) modality are concatenated along the feature domain. As each modality serves as the source and target modality in turn, we are interested to analyze the impact of the distinct concatenation strategies, e.g., concatenate the translations via the same source or target modality. As shown in Figure 8, it is obvious to find that audio-based target concatenation [(T  A)  (V  A)] performs significantly better than [(A  T)  (A  V)] with a large margin. Analogously, video-based target concatenation [(T  V)  (A  V)] works better than [(V  A)  (V  T)]. The above performance may indicate that joint presentation is able to achieve the significantly improved benefits with the help of guidance modality text. In conclusion, when text modality serves as the guidance modality, which may effectively leverage the contribution from audio and video, and further boost the task performance in a robust and consistent way. In this paper, we present a novel hierarchical multimodal fusion architecture using coupled-translation fusion network (CTFN). Initially, CTFN is utilized for exploiting bi-directional interplay via couple learning, ensuring the robustness in respect to missing modalities. Specifically, the cyclic mechanism directly discards the decoder and only embraces the encoder of Transformer, which could contribute to a much lighter model. Due to the couple learning, CTFN is able to conduct bi-direction cross-modality intercorrelation parallelly. Based on CTFN, a hierarchical architecture is further established to exploit multiple bi-direction translations, leading to double multimodal fusing embeddings compared with traditional translation methods. Additionally, a multimodal convolutional fusion block is employed to further explore the complementarity and consistency between cross-modality translations. Essentially, the parallel fusion strategy allows the model maintains robustness and flexibility when considering only one input modality. CTFN was verified on two public multimodal sentiment benchmarks, the experiments demonstrate the effectiveness and flexibility of CTFN, and CTFN achieves state-of-the-art or comparable performance on CMU-MOSI and MELD (Sentiment). For future work, we like to evaluate CTFN on more multimodal fusion tasks. The source code can be obtained from https://github.com/deepsuperviser/CTFN. We sincerely thank the anonymous reviewers for their insightful comments and valuable suggestions. This work was supported by National Key R&D Program of China for Intergovernmental International Science and Technology Innovation Cooperation Project (2017YFE0116800), National Natural Science Foundation of China (U20B2074, U1909202), Science and Technology Program of Zhejiang Province (2018C04012), Key Laboratory of Brain Machine Collaborative Intelligence of Zhejiang Province (2020E10010), JSPS KAKENHI (Grant No. 20H04249), and supported by the Ministry of Education and Science of the Russian Federation (Grant 14.756.31.0001).
abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain objective objective objective abstain abstain abstain abstain method abstain abstain objective other other other other other other other other other other other other other other other other other other abstain abstain abstain abstain abstain abstain abstain method abstain other method abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain other abstain other abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain other abstain method abstain abstain abstain other abstain abstain other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain method other other other