Information integration from different modalities is an active area of research. Human beings and, in general, biological neural systems are quite adept at using a multitude of signals from different sensory perceptive fields to interact with the environment and each other. Recent work in deep fusion models via neural networks has led to substantial improvements over unimodal approaches in areas like speech recognition, emotion recognition and analysis, captioning and image description. However, such research has mostly focused on architectural changes allowing for fusion of different modalities while keeping the model complexity manageable. Inspired by neuroscientific ideas about multisensory integration and processing, we investigate the effect of introducing neural dependencies in the loss functions. Experiments on multimodal sentiment analysis tasks with different models show that our approach provides a consistent performance boost. Human beings perceive the world as a unified whole, not in individual sensory modalities. While traditionally different sensory models have been studied in isolation, it has been well recognized that perception operates via integration of information from multiple sensory modalities. Research in multimodal fusion aims to achieve a similar goal in artificial models: extract and integrate all information from different input modalities. For example, if someone is sarcastic, the facial expression and voice intonation provide information not directly decipherable from the uttered words. If a model only looks at the text of the interaction, then it is unlikely to classify this interaction currently. Current research in deep multimodal fusion primarily deals with architectural improvements to create complex feature-rich, yet efficient representations (Zadeh et al., 2017; Liu et al., 2018; Hazarika et al., 2020). The hope is that more complex models will be able to integrate the complementary information from different unimodal representations into a unified common representation. Learning such unified representations, however, is a challenging task. Different modalities can present the same information in radically different ways with emphasis on different aspects of the content. These heterogeneities across different modalities mean that learning multimodal representations must deal with feature shifts, distributional effects, nuisance variation and a variety of related challenges (Baltruaitis et al., 2018). Inspiring from work in multisensory neural processing, we define a loss regularizer that we call synergy to train these models. Synergy has a specific meaning in information-theoretic literature (Cover, 1999). The synergy between random variables X and Y refers to the unique mutual information that X provides about Y . While our loss function is not the same as information theoretic synergy, the intuition behind our proposed loss is the same as actual synergy; to try to maximize dependencies between the representations. As our method uses neural networks or kernel-based methods to capture distributional divergences, we expect that this method will allow our model to capture complex dependencies which cannot be captured via techniques like subspace alignment. We test our proposed training loss on different multimodal fusion architectures including LFN(Zadeh et al., 2017), MFN (Zadeh et al., 2018a), MAGBERT(Rahman et al., 2020) and MIM (Han et al., 2021). Our experiments show that training with synergy maximization improves the result by a significant margin. In this section, we give an overview of the basic ideas relevant to this work; primarily mutual information, and existing work on deep multimodal fusion and neural synergy. The problem in the most abstract terms is a supervised learning problem. We are provided with a dataset of N observations D = ( x i , y i ) Ni =1 . All x i come from a space X and y i from Y . We are provided a loss function L : Y  Y  R which is the task loss. Our goal is to learn a model F  : X  Y such that the total loss L = (cid:80) i L ( F ( x i ) , y i ) is minimized. In multimodal fusion the space of inputs X naturally decomposes into K different modalities X = (cid:81) Kj =1 X j . We use X j to represent random variables which form the individual modality specific components of the input random variable X . A common way to learn such a multimodal function is to decompose it into two components: a) an embedding component E which fuses information into a high dimensional vector in R d and b) a predictive component P which maps vector from R d to Y . Furthermore since the different modalities are often no directly compatible with each other (for eg text and image), E itself is decomposed into a) modality specific readers F i X i  R d i which are specifically designed for each individual modality X i and b) a fusion component F : (cid:81) i R d i  R d which fuses information from eah individual modality embedding. F is provided with uni-modal representations of the inputs X i = ( X 1 , X 2 , . . . XK ) obtained through embedding networks f i . F has to retain both unimodal dependencies (i.e relations between features that span only one modality) and multi-modal dependency (i.e relationships between features across multiple modalities). This decomposition has two advantages a) the individual modality reader can be pre-trained on the task at hand or even from a larger dataset (for example BERT (Devlin et al., 2018) for language, Resnet (He et al., 2016) for images ) which allows us to leverage wider modality specific information and b) often but not always each individual modality is in principle enough to correctly predict the output 2.2 Distributional Divergences Divergence is a functional which characterizes the distance or "discrepancy" between two probability distributions on the same space. Divergence however is a different notion than distance because divergences are not necessarily symmetric. A common measure of discrepancy between two distributions is the Kullback-Liebler divergence (KL divergence) (Cover, 1999). This divergence is often also used implicitly for estimating dependence between two random variables. Mutual information (MI) is a measures of dependence between two random variable X and Y capable of incorporating multiple types of relationships between them. If we have variables X and Y , then the mutual information between them is given by I ( X ; Y ) = KL [ p XY ( x, y )  p X ( x ) p Y ( y )] where p XY is the joint probability density of the pair ( X, Y ) , and p X , p Y are the marginal probability densities of X, Y respectively. Estimation of Divergence Estimating entropic differences between two distributions purely from their samples is a difficult task (Kinney and Atwal, 2014). As such there have been multiple types of divergences proposed over the years (Gretton et al., 2005; Studen`y and Vejnarov, 1998). Moreover in recent years, several estimators have been proposed for entropic divergences based on variational methods (Belghazi et al., 2018; Hjelm et al., 2018; Amjad and Geiger, 2019). These estimators use flexible neural networks as a contrast function and optimize a variational bound. We describe two such methods which are used in our experiments  Neural Mutual Information (Belghazi et al., 2018) is a variational method to estimate the KL divergence between two distributions. It is estimated via gradient ascent on the Donsker-Varadhan bound (Donsker and Varadhan, 1985). The Donsker Varadhan bound shows that: KL ( P, Q )  sup g EX  P [ g ( X )]  EX  Q [exp g ( X ) ] The Young-Fenchel duality shows that the gap is zero; i.e. at the optima the right side of the above expression matches the KL divergence. Instead of a global maximization over all functions one can instead use a family of 1168 functions parameterized via neural networks. The bound obtained thus is necessarily lower than the actual KL, but now one can use gradient descent to optimize the network. Maximum Mean Discrepancy or MMD (Gretton et al., 2012) is a kernel based estimator of divergence between distributions. Mathematically the MMD between two distributions P and Q is given by the norm of the difference of the mean embeddings of P and Q in the RKHS space of the chosen kernel. Further extensions to MMD have been developed based on neural networks which provide non-universal but more powerful kernel based tests (Liu et al., 2020). The above formula can be estimated purely via samples by using the Kernel matrix K ( x i , x j ) =  ( x i ) T  ( x j ) where  represents the corresponding RKHS embedding function The final monte carlo estimator is given by: Kurtosis is a statistical measure which is used to categorize the behavior of the distribution tails. It is more sensitive to rare events and hence is used for distributions with "fatter tails". For univariate variables, kurtosis is the standardized fourth moment i.e E [( X   ) 4 ] ( E [( X   ) 2 ]) 2 It is often used to measure deviations from normality. random variables is also sometimes used as a measure of dependence between them. It is one of the metrics used by Rosas et al. (2019); Barrett and Seth (2011) to analyze neural complexity and brain functional connectivity. Earlier work on neural fusion models primarily relied on an early fusion of features. These approaches simply concatenated inputs of different modalities and used simple models to combine requisite information. Despite their simplicity, such models often perform well and are robust (Narayanan et al., 2019). More modern methods, however, deploy fancier methods to induce information aggregation. One set of models used gradient descent to try to force different feature networks to learn about each other and embed information jointly. This process can be enhanced by adding specific forms of regularization such as reconstruction loss (Mai et al., 2020), or auxiliary task loss (Chen et al., 2017; Yu et al., 2021). Another family of models uses linear algebra based methods to combine unimodal representations. Methods like those of Liu et al. (2018); Chen and Mitra (2018); Chachlakis et al. (2019) try to fuse information via tensor decomposition of high dimensional product tensors of individual unimodal representations. Other methods use subspace alignment (Lee et al., 2019; Yu et al., 2012) or correlation loss (Sun et al., 2020; Hazarika et al., 2020) to merge different representations. However, in some form or other, these models rely primarily on architectural changes. We, on the other hand, do not want to focus on such changes. Instead, our goal was to use insights from neuroscience to provide a methodology that can be deployed atop any standard multimodal fusion model. A common and vital feature of nervous systems is the integration of information arriving simultaneously from multiple sensory pathways. The underlying neural structures have been found to be related in both vertebrates and invertebrates. The classic understanding of this process is that different sensory modalities are processed individually and then combined in various multimodal convergence zones, including cortical and subcortical regions (Ghazanfar and Schroeder, 2006), as well as 1169 Figure 1: A general multimodal fusion Architecture. Studies in the superior colliculus (Meredith et al., 1987) showed that multiple sensory modalities are processed in this brain stem region, with some neurons being exclusively unimodal and others being multimodal. Hypotheses of encoding of multimodal information include changes in neuronal firing rates (Pennartz, 2009) or a combinatorial code in population of neurons (Osborne et al., 2008; Rohe and Noppeney, 2016). Evidence shows that while multimodal representations are distinct from unimodal ones, there is sufficient overlap between the set of neurons that process different sensory modalities. For example, Follmann et al. (2018) show that even in a simple crustacean organism, more than half the neurons in the commissural ganglion are multimodal. Moreover, they show that in 30% of these multimodal neurons, responses to one modality were predictive of responses to other modalities. Both these facts suggest that the neural representations across different modalities have high information about each other. Studies of multisensory collicular neurons suggest that their crossmodal receptive fields (RF) often overlap (Spence et al., 2004). This pattern is also found in multisensory neurons present in other brain regions. As such, a spatiotemporal hypothesis of multisensory integration has been suggested: superadditive multimodal processing is observed when information from different modalities comes from spatiotemporally overlapping receptive fields (Recanzone, 2003; Wallace et al., 2004; Stanford et al., 2005). Since multimodal cortical neurons are generally downstream of modality-specific regions, the information about RF overlap is present in their input unimodal neural representations. Moreover, the sensory-specific nuclei of the thalamus have been shown to feed multisensory information to primary sensory specific-cortices (Kayser et al., 2008). This suggests the existence of explicit feedback connection from the multimodal representations to unimodal representations . Cortical and subcortical networks often contain clusters of strongly connected neurons. Functionally the existence of such cliques imply highly integrated pyramidal cells that handle a disproportionately large amount of traffic (Harriger et al., 2012). In cortical circuits, around 20% of the neurons account for 80% of the information propagation (Nigam et al., 2016; Van Den Heuvel and Sporns, 2011). Timme et al. (2016); Faber et al. (2019) demonstrate that multimodal computation tends to concentrate in such local cortical clusters. They also found significantly lower kurtosis in such clusters and that dependence between oscillations was proportional to the amount of information flow. Sherrill et al. (2020) show that highly kurtotic neural activity positively related when multiple external stimuli are provided. Thus, kurtosis in neural firings is a representation of the dependence between inputs. This suggests that when input kurtosis is high there is more significant cognitive processing and information flow required to extract relevant 1170 information . For our purposes we will limit ourselves to talk about tasks similar to the MOSI dataset. In this setting the input has three modalities viz audio ( a ), visual ( v ), and textual language ( l ). The fusion problem involves learning a representation M f that combined the uni-modal representations of the inputs X a,v,l = ( X a , X v , X l ) . We modify the base neural architecture to incorporate the global structure explained in the last section. We propose a way to incorporate such changes without major architectural change into current baseline designs. The key component is the additional network (colored in red) in Figure 1 which we shall call as C-network. The C-network takes as input the individual unimodal representations and the fused representation and attempts to force a specific form of dependency as explained below. C-Network The purpose of the C-Network is to try to enforce on the model the three primary characteristics of real neural circuits explained in the earlier section. We list them here and describe how we attempt to incorporate those characteristics in a more standard model. Individual uni-modal representations should be predictive of other uni-modal representations. We try to achieve this by simply predicting on modality representation by the combination of others. Q i refers to a modality associated neural network which attempts to reconstruct the unimodal representation Z i from the other representations Z  i . The error between the two is penalized in the form of a reconstruction loss between modalities i.e. we add a penalty of the form: LL 2 = || Q i ( Z  i )  Z i || 2  Multimodal representation should be feedback into input neurons to align and capture information between them. Providing feedback during inference time from the multimodal representation would be ideal. However this would make the overall prediction recurrent, something fundamentally different from most current architectures. Moreover given current high dimensional encoders; doing such processing would be extremely resource intensive. As such we aim to achieve this feedback by treating the multimodal representation and unimodal representation spaces as different domains and adding a loss of the form: L d = d ( p ( g i ( Z i )) , p ( g i ( Z ))) The purpose of the aforementioned loss is to align the distributions of the features in the same embedding space of the mapping from the multimodal and unimodal domains. d represents a measure that captures the discrepancy between the distributions, g i refers to neural networks for projecting and aligning the combined representation Z with unimodal representations Z i , and p denotes the empiri-cal/sample distribution of the corresponding features. In our experiments, for d we use the MMD discrepancy (Gretton et al., 2012) and KL divergence as the metric; though other divergences can also be used. Note that this loss by itself can be minimized by forcing the g functions to ignore their inputs. We prevent this by first doing a random projection of the features 1 into a smaller dimensional vector space and then apply an invertible neural network. Such alignment losses have been used in works on domain adaptation (Motiian et al., 2017) under the name semantic loss or confusion loss. We refer the readers to Motiian et al. (2017); Li et al. (2019) for more details on semantic losses. Note that instead of aligning the features via some kind of embedding based distributional distance, one could try to maximize mutual information between the embeddings as well. We experiment with one such model in our experiment and as the results show, found it to be slightly worse than using MMD based alignment loss. Individual unimodal and multimodal representations should have low kurtosis. To ensure this condition we estimate the multivariate kurtosis by plugging in standard estimators for the mean and covariates. The final kurtosis estimator used is given by: 1 similar to Johnson Lindenstrauss projections (Landweber et al., 2016) 1171  = 1 n n (cid:88) i [(( z i   z ) TS  1 ( z i   z )) 2 ] where z i here are samples from the Z features in the model (where Z can be unimodal features like Z a or fused final feature Z ). z refers to the empirical mean feature  z = n (cid:80) i z i n and S is the empirical covariance matrix S = n (cid:80) i ( z i   z )( z i   z ) T n . An important thing to note here is that high dimensional kurtosis values can be highly sensitive to outliers. As such we regularize the estimate by doing three things: a) We cap the max norm of the difference vectors during estimation. b) We scale up the diagonal of the covariance matrix to reduce its condition number c) Finally the covariance matrix itself is computed via a decaying moving average over a window of multiple batches to produce smoother estimates before the inversion operation. During training we add the regularization penalties described earlier along with the usual maximum likelihood based objective. The different loss components are weighted with seperate hyper-parameters. Note that the C-Network is purely a training time addition, and is not invoked during inference. Hence the additional network invoke zero additional time during testing. An algorithmic description of the full method is presented in the Appendix D 5 Experiments 5.1 Datasets We empirically evaluate our methods on two commonly used datastes for multimodal training viz CMU-MOSI and CMU-MOSEI. CMU-MOSI (Wllmer et al., 2013) is sentiment prediction taks on a set of short youtube video clips. CMU-MOSEI (Zadeh et al., 2018b) is a similar dataset consisting of around 23k review videos taken from YouTube. The output in both cases is a sentiment score in [  3 , 3] . For each dataset, three modalities are available; audio, visual frames, and language. Preliminary features on each modality is obtained as follows:  Audio: Features are extracted from the sund recordings using the method of Degottex et al. (2014). Language: The video transcripts are converted to word embeddings using BERT (Devlin et al., 2018) or Glove (Pennington et al., 2014)  Visual: Visual features are extracted using FACET (iMotion) which provides facial action units vectors. We run our experiments with the following architectures:  FLSTM (Narayanan et al., 2019) is the baseline early fusion LSTM architecture used by Zadeh et al. (2017)  Tensor Fusion Network or TFN (Zadeh et al., 2017) combined information via pooling of a high dimensional tensor representation of multimodal features. More specifically it does a multimodal Hadamard product of the aggregated features with RNN based language features. Memory Fusion Network or MFN (Zadeh et al., 2018a) incorporate gated memory-units to store multiview representations. It then performs an attention augmented readout over the memory units to combine information into a single representation. MAGBERT (Rahman et al., 2020) is a transformer based architecture that uses the Wang gate (Wang et al., 2019). The multimodal information is send to the multimodal gate to compute modified embeddings which are passed to a BERT (Devlin et al., 2018) based model. This model achieves state-of the-art results on multimodal sentiment benchmark MOSI (Wllmer et al., 2013) and MOSEI (Zadeh et al., 2018c). MIM (Han et al., 2021) is a recent near SOTA architecture. It combined BERT based text embeddings with modality specific visual and acoustic LSTMs (Hazarika et al., 2020). Recently Colombo et al. (2021) conducted experiments introducing a information regularizer on existing architectures. method are a) our method focuses on synergy terms whereas their proposal is optimizing joint mutual information between different unimodal representations; and b) they experiment with variational measures of information. We replicate our experiments with their best performing model and present the results with the label I Was . We report both the Mean Absolute Error (MAE) and the correlation of model predictions with true labels. In the literature, the regression task is also turned into a binary classification task for polarity prediction. We follow Rahman et al. (2020) Accuracy Acc 7 denotes accuracy on 7 classes and Acc 2 the binary accuracy) of our best performing models. We also report the Mean Absolute Error (MAE) and the correlation of model intensity predictions with true values. We present and discuss here the results obtained in our experiments. Results on MOSI are presented in Table 2 while Table 3 present results for MOSEI dataset. We trained each of the models with the standard cross entropy loss (labeled as NLL); and with cross entropy loss regularized with the synergy penalty discussed earlier. On both datasets, regularization via synergy leads to performance improvement. For example, a MFN on CMU-MOSI trained with MMD based synergy (NLL+S MMD ) outperforms by more than 4 points on Acc 7 than standard likelihood training. On CMU-MOSEI too the gains are significant when trained with synergy regularization. In general training via MMD synergy tends to be better than via KL synergy. This might be the inherent behavior of the MMD dependency which is always well defined; or it might reflect the hardness of information estimation. For example it is well known that good bounds on standard mutual information are difficult to obtain (Kin-Acc 7 Acc 2 MAE CORR FLSTM NLL 31.2 75.9 1.01 0.64 NLL+S KL 31.6 76.3 1.01 0.66 NLL+S MMD 33.6 76.4 0.98 0.66 MFN NLL 31.3 76.6 1.01 0.62 NLL+S KL 32.5 76.6 0.94 0.65 NLL+S MMD 35.9 77.4 0.95 0.66 NLL+I Was 35.1 77.1 0.97 0.63 LFN NLL 31.9 76.9 1.01 0.64 NLL+S KL 32.6 77.6 0.97 0.64 NLL+S MMD 35.4 77.9 0.97 0.67 NLL+I Was 32.4 77.6 0.97 0.64 MAGBERT NLL 40.2 83.7 0.79 0.80 NLL+S KL 41.9 84.1 0.76 0.82 NLL+S MMD 41.9 85.6 0.76 0.82 NLL+I Was 41.8 84.2 0.76 0.82 MIM NLL 46.3 83.7 0.77 0.76 NLL+S KL 46.4 83.7 0.74 0.75 NLL+S MMD 46.7 84.2 0.72 0.79 NLL+I Was 46.6 84.2 0.75 0.79 Table 2: Results on sentiment analysis on CMU-MOSI. Acc 7 denotes accuracy on 7 classes and Acc 2 the binary accuracy. MAE denotes the Mean Absolute Error and Corr is the Pearson correlation ney and Atwal, 2014); while MMD estimator are asymptotically consistent (Gretton et al., 2012) 5.5 Modality Dropout Zadeh et al. (2018a); Rahman et al. (2020) have demonstrated that while multimodal fusion does improve performance, the primary modality continues to be textual data. Hence in this experiment, we want to assess the effect of corruptions of text modality in our model. Following Colombo et al. (2021) we experiment with dropping the text modality either by itself (T) or with one of the other modalities (T+V or T+A). The results are presented in Table 4 Since the C-Networks forces a reconstruction and distributional divergence loss between the unimodal and multimodal representations, one would expect that models trained using our approach would be more resistant to modality errors. This is borne out in the experiments, where we see that 1173 Acc 7 Acc 2 MAE CORR FLSTM NLL 44.1 75.1 0.72 0.52 NLL+S KL 44.4 75.6 0.70 0.52 NLL+S MMD 45.3 76.0 0.68 0.54 MFN NLL 44.3 74.7 0.72 0.52 NLL+S KL 44.3 74.8 0.72 0.56 NLL+S MMD 46.2 75.1 0.69 0.56 NLL+I Was 45.1 75.2 0.72 0.54 LFN NLL 45.2 74.3 0.70 0.54 NLL+S KL 46.1 75.3 0.69 0.56 NLL+S MMD 46.3 75.3 0.67 0.56 NLL+I Was 45.9 75.1 0.69 0.55 MAGBERT NLL 46.9 83.9 0.59 0.77 NLL+S KL 47.4 85.3 0.59 0.79 NLL+S MMD 47.9 85.4 0.59 0.79 NLL+I Was 47.2 85.0 0.59 0.78 MIM NLL 53.3 79.6 0.54 0.75 NLL+S KL 53.5 80.3 0.54 0.77 NLL+S MMD 54.3 82.4 0.52 0.77 NLL+I Was 53.5 82.1 0.53 0.77 Table 3: Results on sentiment analysis on CMU-MOSEI. Note that the C-network itself is not active at test time; instead this effect is due to the alignment forced by the network during training. An interesting future direction would be to explicitly use the C-network outputs to ameliorate modality corruption. Our overall proposal has multiple components viz a) the reconstruction loss (also called LL 2 loss); b) the distribution alignment loss (which we call L d Loss); and c) the kurtosis loss L  . As such we ran experiments to assess the importance of each component. Specifically we trained the model without each of the three loss components prescribed in our method, and assessed the test performance. The results are presented in Appendix A. First we note the performance improvement by incorporating kurtosis in the regularization which shows the efficacy of this term. Second one can also note that removing any individual component leads to reduction in performance, suggesting all components act together in a synergistic way to improve the results. In this paper, we used the idea of regularizing via a term which we label neural synergy maximization. This regularizer is inspired by neural cicruit design in the vertebral cortex. We experimented with different measures of synergy based on discrepancy measures such as KL and MMD. We also show that training with synergy can produce benefit on even SOTA architectures. Limitations The most prominent limitation of this approach, is that it is inherently limited by the architecture with which it is being used. While our additional loss did improve performance, one can observe that the final performance is dependent on the initial performance. For example, while we tested on four architectures, the final performance of each model was in the same range as the initial performance. An entirely different architecture can possibly improve over our results. On the other hand our approach is model agnostic and applicable on any model trained only via max-likelihood.
abstain abstain abstain abstain objective result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain objective method abstain result abstain abstain method abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain other abstain method abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain method abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain result abstain abstain abstain abstain abstain abstain abstain other abstain abstain result result abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain method result abstain result abstain result method