summaries: This paper proposes an online learning algorithm for supervised dimension reduction, called incremental sliced inverse regression (ISIR). The key idea is converting the SIR problem into PCA problem by using the inverse of covariance matrix. After the transformation, we can use incremental PCA to compute the top eigenvector and obtain the approximate solution of SIR in streaming way. The authors also extend ISIR to overlapping case.

summaries: Sliced Inverse Regression is a well-known technique for finding EDR space in supervised dimension reduction problems, under condition that input X is normally disributed. When the number of dimensions is large and an access to observations is online, finding eigenvalues of covariance matrices (online) could become computationally costly. So, the paper focuses the problem of updating few principal components of covariance matrices as new examples come.

abstractText: Online learning has attracted great attention due to the increasing demand for systems that have the ability of learning and evolving. When the data to be processed is also high dimensional and dimension reduction is necessary for visualization or prediction enhancement, online dimension reduction will play an essential role. The purpose of this paper is to propose new online learning approaches for supervised dimension reduction. Our first algorithm is motivated by adapting the sliced inverse regression (SIR), a pioneer and effective algorithm for supervised dimension reduction, and making it implementable in an incremental manner. The new algorithm, called incremental sliced inverse regression (ISIR), is able to update the subspace of significant factors with intrinsic lower dimensionality fast and efficiently when new observations come in. We also refine the algorithm by using an overlapping technique and develop an incremental overlapping sliced inverse regression (IOSIR) algorithm. We verify the effectiveness and efficiency of both algorithms by simulations and real data applications.

authors:
Anestis Antoniadis
Sophie Lambert-Lacroix
Frédérique Leblanc

title: Effective dimension reduction methods for tumor classification using gene expression

venue: data. Bioinformatics,

authors:
Zhidong Bai
Xuming He

title: A chi-square test for dimensionality with non-Gaussian data

venue: Journal of Multivariate Analysis,

authors:
M Pilar Barrios
Santiago Velilla

title: A bootstrap method for assessing the dimension of a general regression problem

venue: Statistics & Probability Letters,

authors:
Claudia Becker
Roland Fried

title: Sliced inverse regression for high-dimensional time series

venue: In Exploratory Data Analysis in Empirical Research,

authors:
Peter N. Belhumeur
João P Hespanha
David J. Kriegman

title: Eigenfaces vs. Fisherfaces: Recognition using class specific linear projection

venue: IEEE Transactions on Pattern Analysis and Machine Intelligence,

authors:
Efstathia Bura
R Dennis Cook

title: Extending sliced inverse regression: The weighted chi-squared test

venue: Journal of the American Statistical Association,

authors:
Shivkumar Chandrasekaran
Bangalore S Manjunath
Yuan-Fang Wang
Jay Winkeler
Henry Zhang

title: An eigenspace update algorithm for image analysis

venue: Graphical Models and Image Processing,

authors:
Delin Chu
Li-Zhi Liao
Michael Kwok-Po Ng
Xiaoyan Wang

title: Incremental linear discriminant analysis: a fast algorithm and comparisons

venue: IEEE Transactions on Neural Networks and Learning Systems,

authors:
R Dennis Cook
Sanford Weisberg

title: Sliced inverse regression for dimension reduction: Comment

authors:
R Dennis Cook
Xin Zhang

title: Fused estimators of the central subspace in sufficient dimension reduction

authors:
Jian J Dai
Linh Lieu
David Rocke

title: Dimension reduction for classification with gene expression microarray data

venue: Statistical Applications in Genetics and Molecular Biology,

authors:
Laura Elnitski
Ross C Hardison
Jia Li
Shan Yang
Diana Kolbe
Pallavi Eswara
Michael J O’Connor
Scott Schwartz
Webb Miller
Francesca Chiaromonte

title: Distinguishing regulatory DNA from neutral sites

venue: Genome Research,

title: Determining the dimension in sliced inverse regression and related methods

authors:
Kenji Fukumizu
Francis R Bach
Michael I Jordan

title: Kernel dimensionality reduction for supervised learning

venue: In Advances in Neural Information Processing Systems

authors:
Ali Gannoun
Stéphane Girard
Christiane Guinot
Jerôme Saracco

title: Sliced inverse regression in reference curves estimation

venue: Computational Statistics & Data Analysis,

authors:
Bin Gu
Victor S Sheng
Zhijie Wang
Derek Ho
Said Osman
Shuo Li

title: Incremental learning for ν-support vector regression

venue: Neural Networks,

authors:
Peter M Hall
David A Marshall
Ralph R Martin

title: Incremental eigenanalysis for classification

venue: In Proceedings of the British Machine Vision Conference,

title: Merging and splitting eigenspace models

authors:
Ping He
Kai-Tai Fang
Cheng-Jian Xu

title: The classification tree combined with sir and its applications to classification of mass spectra

venue: Journal of Data Science,

authors:
Tae-Kyun Kim
Shu-Fai Wong
Bjorn Stenger
Josef Kittler
Roberto Cipolla

title: Incremental linear discriminant analysis using sufficient spanning set approximations

venue: IEEE Conference on Computer Vision and Pattern Recognition,

title: Sliced inverse regression for dimension reduction

title: On principal hessian directions for data visualization and dimension reduction: Another application of Stein’s lemma

authors:
Lexin Li
Hongzhe Li

title: Dimension reduction methods for microarrays with application to censored survival

authors:
Lexin Li
Xiangrong Yin

title: Sliced inverse regression with regularizations

title: Consistent estimation of the dimensionality in sliced inverse regression

venue: Annals of the Institute of Statistical Mathematics,

authors:
Ahmet Öztaş
Murat Pala
Erdogan Özbay
Erdogan Kanca
Naci Caglar
M Asghar Bhatti

title: Predicting the compressive strength and slump of high strength concrete using neural network

venue: Construction and Building Materials,

authors:
Shaoning Pang
Seiichi Ozawa
Nikola Kasabov

title: Incremental linear discriminant analysis for classification of data streams

venue: IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics),

authors:
Chuan-Xian Ren
Dao-Qing Dai

title: Incremental learning of bidirectional principal components for face recognition

venue: Pattern Recognition,

authors:
Paul Rodriguez
Brendt Wohlberg

title: A matlab implementation of a fast incremental principal component pursuit algorithm for video background modeling

venue: IEEE International Conference on Image Processing (ICIP),

title: Determining the dimensionality in sliced inverse regression

authors:
Juyang Weng
Yilu Zhang
Wey-Shiuan Hwang

title: Candid covariance-free incremental principal component analysis

title: Kernel sliced inverse regression with applications to classification

venue: Journal of Computational and Graphical Statistics,

authors:
Qiang Wu
Sayan Mukherjee
Feng Liang

title: Localized sliced inverse regression

authors:
Yingcun Xia
Howell Tong
WK Li
Li-Xing Zhu

title: An adaptive estimation of dimension reduction space

venue: Journal of the Royal Statistical Society: Series B (Statistical Methodology),

authors:
Jieping Ye
Qi Li
Hui Xiong
Haesun Park
Ravi Janardan
Vipin Kumar

title: IDR/QR: An incremental dimension reduction algorithm via QR decomposition

venue: IEEE Transactions on Knowledge and Data Engineering,

title: Modeling of strength of high-performance concrete using artificial neural networks

venue: Cement and Concrete Research,

title: Design of high-performance concrete mixture using neural networks and nonlinear programming

venue: Journal of Computing in Civil Engineering,

title: Modeling slump flow of concrete using second-order regressions and artificial neural networks

venue: Cement and Concrete Composites,

authors:
Ning Zhang
Zhou Yu
Qiang Wu

title: Overlapping sliced inverse regression for dimension reduction

venue: arXiv preprint arXiv:1806.08911,

authors:
Ting Zhang
Wenhua Ye
Yicai Shan

title: Application of sliced inverse regression with fuzzy clustering for thermal error modeling of CNC machine tool

venue: The International Journal of Advanced Manufacturing Technology,

authors:
Haitao Zhao
Pong Chi Yuen

title: Incremental linear discriminant analysis for face recognition

authors:
Haitao Zhao
Pong Chi Yuen
James T Kwok

title: A novel incremental principal component analysis and its application for face recognition

heading: 1 INTRODUCTION

text:
Dimension reduction aims to explore low dimensional representation for high dimensional data. It helps to promote our understanding of the data structure through visualization and enhance the predictive performance of machine learning algorithms by preventing the “curse of dimensionality”. Therefore, as high dimensional data become ubiquitous in modern sciences, dimension reduction methods are playing more and more important roles in data analysis. Dimension reduction algorithms can be either unsupervised or supervised. Principle component analysis (PCA) might be the most popular unsupervised dimension reduction method. Other unsupervised dimension reduction methods include the kernel PCA, multidimensional scaling, and manifold learning based methods such as isometric mapping and local linear embedding. Unlike unsupervised dimension reduction, supervised dimension reduction involves a response variable. It finds the intrinsic lower-dimensional representations that are relevant to the prediction of the response values. Supervised dimension reduction methods can date back to the well known linear discriminant analysis (LDA) while its blossom occurred in the last twenty years. Many approaches have been proposed and successfully applied in various scientific domains; see Li (1991); Cook & Weisberg (1991); Li (1992); Xia et al. (2002); Setodji & Cook (2004); Fukumizu et al. (2004); Li & Yin (2008); Wu (2008); Wu et al. (2009); Cook & Zhang (2014) and the references therein.
We are in a big data era and facing the challenges of big data processing, thanks to the fast development of modern information technology. Among others, two primary challenges are the big volume and fast velocity of the data. When a data set is too big to be stored in a single machine or when the data arrives in real time and information update is needed frequently, analysis of the data in an online manner is necessary and efficient. If the data is simultaneously big and high dimensional, it becomes necessary to develop online learning approaches for dimension reduction. As PCA and LDA are the most wildly used dimension reduction techniques, a bunch of PCA-based and LDA-based online dimension reduction algorithms has been proposed. Incremental PCA have been described in Hall et al. (1998; 2000); Weng et al. (2003); Zhao et al. (2006); Rodriguez & Wohlberg (2014). Incremental LDA have been developed in Pang et al. (2005); Zhao & Yuen (2008); Kim et al. (2007); Chu et al. (2015). Other strategies like QR decomposition or SVD have also been used in Chandrasekaran et al. (1997); Ye et al. (2005); Ren & Dai (2010).
In this paper, our purpose is to propose a new online learning approach for supervised dimension reduction. Our motivation is to implement the sliced inverse regression (SIR) in an incremental manner. SIR was proposed in Li (1991) and has become one of the most efficient supervised dimension reduction method. SIR and its refined versions have been found successful in many scientific areas such as bioinformatics, hyperspectral image analysis, and physics; see Cook (1994); Belhumeur et al. (1997); Becker & Fried (2003); Elnitski et al. (2003); Gannoun et al. (2004); He et al. (2003); Antoniadis et al. (2003); Li & Li (2004); Dai et al. (2006); Zhang et al. (2016). SIR can be implemented by solving an generalized eigen-decomposition problem, Γβ = λΣβ, where Γ is a matrix depending on the response variable (whose definition is described in the next section) and Σ is the covariance matrix. To make it implementable in an online manner we rewrite it as standard eigendecomposition problem Σ− 1 2 ΓΣ− 1 2η = λη where η = Σ 1 2β and adopt the ideas from incremental PCA. We need to overcome two main challenges in this process. First, how do we transform the data so that they are appropriate for the transformed PCA problem? Note that simply normalizing the data does not work. Second, online update of Σ− 1 2 , if not impossible, seems very difficult. The first contribution of this paper is to overcome these difficulties and design a workable incremental SIR method. Our second contribution will be to refine the method by an overlapping technique and design an incremental overlapping SIR algorithm.
The rest of this paper is arranged as follows. We review SIR algorithm in Section 2 and the incremental PCA algorithm in Section 3. We propose the incremental SIR algorithm in Section 4 and refine it in Section 5. Simulations are done in Section 6. We close with discussions in Section 7.

heading: 2 SLICED INVERSE REGRESSION

text:
The goal of supervised dimension reduction is to find an intrinsic lower-dimensional subspace that contains all the information to predict the response variable. Assume a multivariate predictor x = (x1, x2, . . . , xp) > ∈ Rp and a scalar response y are linked by a semi-parametric regression model
y = f(β>1x,β > 2x, . . . ,β > Kx, ), (1)
where βk ∈ Rp is a p× 1 vector and is the error term independent of x. It implies
y |= x|B>x (2)
where |= denotes “statistical independence” and B = (β1,β2, . . . ,βK) is a p × K matrix. The column space of B is called the effective dimension reduction (EDR) space and each βi is an EDR direction. Note B>x contains all information for the prediction of y. The purpose of supervised dimension reduction is to learn the EDR directions from data.
Unlike the classical regression problem which regresses y against x, sliced inverse regression considers regressing x against y. With the semi-parametric model (1) and the assumption that x follows an elliptical contour distribution (e.g., normal distribution), it was proved in Li (1991) that the centered regression curve E[x|y] − E[x] falls into the K-dimensional subspace spanned by EDR directions Σβk, k = 1, . . . ,K. Consequently, all or part of the EDR directions can be recovered by solving a generalized eigenvalue decomposition problem:
Γβ = λΣβ (3)
where
Γ = E [( E[x|y]−E[x] )( E[x|y]−E[x] )>] is the covariance matrix of inverse regression curve E[x|y], Σ is the covariance matrix of x. Each eigenvector associated with a non-zero eigenvalue is an EDR direction.
Given i.i.d observations {(xi, yi)}ni=1, SIR algorithm can be implemented as follows:
1) Compute sample mean x̄ = 1n ∑n i=1 xi and the sample covariance matrix
Σ̂ = 1
n n∑ i=1 (xi − x̄)(xi − x̄)>.
2) Bin the observations into H slices according to y values. For each slice sh, h = 1, . . . ,H, compute the sample probability p̂h = nhn and the sample slice mean m̂h = 1 nh ∑ yi∈sh xi.
The matrix Γ is estimated by
Γ̂ = H∑ h=1 p̂h (m̂h − x̄) (m̂h − x̄)>.
3) Solve the generalized eigen-decomposition problem
Γ̂B̂ = Σ̂B̂Λ.
The EDR directions are estimated by the top K eigenvectors β̂k, k = 1, 2, . . . ,K.
This algorithm is not very sensitive to the choice of parameter H provided it is sufficiently larger than K while not greater than n2 . Root-n consistency is usually promised. It is suggested samples are evenly distributed into the H slices for the best performance.

heading: 3 INCREMENTAL PCA

text:
PCA looks for directions along which the data have the largest variances. It is implemented by solving an eigen-decomposition problem
Σ̂u = λu. (4)
The principal components are the eigenvectors corresponding to largest eigenvalues. Throughout this paper, we assume all eigenvalues are arranged in a descending order, i.e., λ1 ≥ λ2 ≥ . . . ≥ λp. Suppose that we need to retain the top K principal components. Denote UK = [u1, . . . ,uK ], ΛK = diag(λ1, . . . , λK). We have a reduced system Σ̂UK = UKΛK .
In incremental PCA, after receiving a new coming observation x0, we need to update the reduced eigen-system to a new one
Σ̂′U′K ≈ U′KΛ′K . (5) A “=” is generally impossible unless λK+1 = . . . = λp = 0 in (4).
The idea of updating the system in Hall et al. (1998) is as follows. Compute a residual vector
v = (x0 − x̄′)−UKU>K(x0 − x̄′)
where x̄′ is the mean of all observations (including x0). It defines the component of x0 that is perpendicular with the subspace defined by UK . If x0 lies exactly within the current eigenspace, then the residual vector is zero and there is no need to update the system. Otherwise, we normalize v to obtain v̄ = v‖v‖ .We may reasonably assume each column vector of U ′ K is a linear combination of column vectors of UK and v̄. (Note this is exactly true if λK+1 = . . . = λp = 0.) This allows us to write
[U′K ,u ′ K+1] = [UK , v̄]R.
where R is a (K + 1) × (K + 1) rotation matrix and u′K+1 is an approximation of the (K + 1)th eigenvector of Σ̂′. So we have
Σ̂′[UK , v̄]R = [UK , v̄]RΛ ′ K+1,
which is equivalent to
[UK , v̄] >Σ̂′[UK , v̂]R = RΛ ′ K+1.
This is an eigen-decomposition problem of dimensionality K + 1 p. It solves the rotation matrix R and allows us to update principal components to U′K , given by the first K column of [UK , v̄]R. If we need to increase the number of principal components, we can just update the system to K ′ = K+ 1 and U′K′ = [U ′ K ,u ′ K+1]. This incremental PCA algorithm was shown convergent to a stable solution when the sample size increases (Hall et al., 1998).

heading: 4 INCREMENTAL SIR

text:
Our idea to develop the incremental sliced inverse regression (ISIR) is motivated by reformulating SIR problem to a PCA problem. To this end, we define η = Σ 1 2β, called the standardized EDR direction, and rewrite the generalized eigen-decomposition problem (3) as an eigen-decomposition problem
Σ− 1 2 ΓΣ− 1 2η = λη. (6)
Note that Σ− 1 2 ΓΣ− 1 2 is the covariance matrix of Σ− 1 2 E[x|y]. So (6) can be regarded as a PCA problem with data collected for Σ− 1 2 E[x|y]. To apply the ideas from IPCA to this transformed PCA problem, however, is not as direct as it looks like. We face two main challenges. First, when a new observation (x0, y0) is received, we need to transform it to an observation for the standardized inverse regression curve. This is different from simply standardizing the data. Second, conceptually, we need to update Σ− 1 2 in an online manner in order to standardize the data. This does not seem feasible. In the following, we will describe in detail how we address these challenges and make the ISIR implementable.
Suppose we have n observations in hand with well defined sample slice probabilities p̂h and means (m̂h, ȳh) for h = 1, . . . ,H, and the eigenvectors B̂ = [β̂1, . . . , β̂K ] of the generalized eigendecomposition problem Γ̂ = λΣ̂β. Then with ΛK = diag(λ1, . . . , λK), we have
Γ̂B̂ = Σ̂B̂ΛK .
Denote Ξ = Σ̂ 1 2 B̂. We have
Σ̂ − 12 Γ̂Σ̂ − 12 Ξ = ΞΛK . (7)
When we have a new observation (x0, y0), we first locate which slice it belongs to according to the distances from y0 to sample slice mean values ȳh of the response variable. Let us suppose the distance from y0 to ȳk is the smallest. So we place the new observation into the slice k and update sample slice probabilities by p̂′h = np̂h n+1 for h 6= k and p ′ k = np′k+1 n+1 . Let nk = npk be the number of observations in slice k before receiving the new observation. For slice mean values we update
m̂′k = nk
nk + 1 m̂k +
1
nk + 1 x0
for slice k only. We can regard z0 = Σ̂ − 12 m′k as a new observation for the standardized inverse regression curve Σ− 1 2 E[x|y]. Following the idea of IPCA, we define a residual vector
v = ( z0 − Σ̂ − 12 x̄′ ) −ΞΞ> ( z0 − Σ̂ − 12 x̄′ )
and normalize it to v̄ = v‖v‖ when v is not zero. To update the eigen-decomposition system to
Σ̂′ − 12 Γ̂′ Σ̂′ − 12
Ξ′ = Ξ′Λ′K , (8)
we assume [Ξ′,η′K+1] = [Ξ, v̄]R with R being a (K+ 1)× (K+ 1) rotation matrix and η′K+1 the (K + 1)th eigenvector of Σ̂′ − 12 Γ̂′ Σ̂′ − 12 . So we have
Σ̂′ − 12 Γ̂′ Σ̂′ − 12
[Ξ, v̄]R = [Ξ, v̄]RΛ′K+1
where Λ′K+1 = diag(Λ ′ K , λ ′ K+1) and λ ′ K+1 is the (K + 1)th eigenvalue. Multiplying both sides by [Ξ, v̄]>, we obtain ( Σ̂′ − 12 [Ξ, v̄] )> Γ̂′ ( Σ̂′ − 12 [Ξ, v̄] ) R = RΛ′K+1. (9)
Note, however, since Σ̂′ − 12
cannot be easily updated, we have to avoid using it. To overcome this challenge, we notice that
Σ̂′ = n
n+ 1 Σ̂′ +
n
(n+ 1)2 (x0 − x̄)(x0 − x̄)> (10)
and the well known Sherman-Morisson formula allows us to update the inverse covariance matrix
Σ̂′ −1 = n+ 1 n Σ̂ −1 −
1 nΣ̂ −1 (x0 − x̄)(x0 − x̄)>Σ̂ −1
1 + 1n+1 (x0 − x̄)>Σ̂ −1 (x0 − x̄) . (11)
If we store Σ̂ −1 and update it incrementally, we can approximate the quantities in (9) as follows:
Σ̂′ − 12 Ξ ≈ Σ̂ − 12 Ξ = B̂,
Σ̂′ − 12 v ≈ Σ̂ −1
(m̂′k − x̄′)− B̂B̂> (m̂′k − x̄′) ,
‖v‖2 = (m̂′k − x̄′) > ( Σ̂ −1 − B̂B̂> ) (m̂′k − x̄′) ,
ṽ = Σ̂′ − 12 v̄ = Σ̂′ − 12 v
‖v‖
≈
( Σ̂ −1 − B̂B̂> ) (m̂′k − x̄′)√
(m̂′k − x̄′) > ( Σ̂ −1 − B̂B̂> ) (m̂′k − x̄′) .
So the problem (9) is approximated by[ B̂, ṽ ]> Γ̂′ [ B̂, ṽ ] R = RΛ′K+1. (12)
Finally notice that the new EDR space B̂′ = Σ̂′ − 12 Ξ′ is the first K columns of Σ̂′ − 12
[Ξ′,η′K+1] =
Σ̂′ − 12 [Ξ, v̄]R and can be approximated by the first K columns of [B̂, ṽ]R.
Note that we avoided updating the inverse square root of the covariance matrix by using the approx-
imation Σ̂ − 12 ≈ Σ̂′ − 12 . This approximation can be very accurate when n is large enough because both converge to Σ− 1 2 . Therefore, we may expect the convergence of ISIR as a corollary of the convergence of IPCA. However, when n is small, the approximation may be less accurate and result in larger difference between EDR spaces estimated by ISIR and SIR. So we recommend that ISIR be used with a warm start, that is, using SIR first on a small amount of data before using ISIR.
In terms of memory, the primary requirement is the storage of Σ̂ −1
, the slice mean matrix M̂ = [m̂1, . . . , m̂H ], and the EDR space B̂. So the memory requirement is O(p2 + pH + pK). As for the computational complexity, notice that the update of Σ̂′ −1
in (11) requires the calculation of Σ̂ −1
(x0 − x̄) and matrix addition and thus has a complexity of O(p2). Since we need to store M̂ and update it sequentially, it is not efficient to store and update Γ̂′ for either memory or computation consideration. Instead, we use the fact Γ̂′ = M̂′P̂′M̂′> where P̂′ = diag(p̂′1, . . . , p̂ ′ H) and write
[B̂, ṽ]>Γ̂′ [B̂, ṽ] = [ B̂>M̂′, ṽ>M̂′ ] P̂′ [ B̂>M̂′, ṽ>M̂′ ]> .
Notice that
ṽ>M̂′ =
( Σ̂ −1 (m̂′k − x̄′) )> M̂′ − ( B̂>(m̂′k − x̄′) )>( B̂>M̂′ ) √
(m̂′k − x̄′)> ( Σ̂ −1 (m̂′k − x̄′) ) − ( B̂>(m̂′k − x̄′) )>( B̂>(m̂′k − x̄′) ) and B̂>(m̂′k− x̄′) is just the kth column of B̂>M̂′. The most time consuming computations for the matrix [B̂, ṽ]>Γ̂′ [B̂, ṽ] are B̂>M̂′ and Σ̂ −1 (m̂′k− x̄′) which have a complexity of O(pKH+ p2).
The complexity of the eigen-decomposition in (12) is O((K+ 1)3) and it requires O(p(K+ 1)2) to update B̂ to B̂′. So the computational complexity for each ISIR update isO(p2+pKH+pK2+K3). For a high dimensional problem, this is much smaller than the complexity O(p3 + p2n) of SIR.

heading: 5 REFINEMENT BY OVERLAPPING

text:
In Zhang et al. (2018) an overlapping technique was introduced to SIR algorithm and shown effectively improving the accuracy of EDR space estimation. It is motivated by placing each observation in two or more adjacent slices to reduce the deviations of the sample slice means m̂h from the EDR subspace. This is equivalent to using each observation two or more times. In this section, we adopt the overlapping technique to ISIR algorithm above to develop an incremental overlapping sliced inverse regression (IOSIR) algorithm and wish it refines ISIR.
To apply the overlapping idea, we use each observation twice. So when we have n observations, we duplicate them and assume we have N = 2n observations. When a new observation (x0, y0) is received, we duplicate it and assume we receive two identical observations. Based on the y0 value we place the first copy into the slice sk if ȳk is closest to y0 and run ISIR update as described in Section 4. Note that if ȳ1 < y0 < ȳH , then y0 must fall into the interval [ȳk′ , ȳk] with k′ = k− 1 or it falls into [ȳk, ȳk′ ] with k′ = k + 1. So we place the second copy of the new observation into the slice sk′ , an adjacent slice to sk, and run ISIR algorithm again. If y0 ≤ ȳ1 or y0 > ȳH , the second copy will be still placed into sk to guarantee all observations are weighted equally.
As OSIR has superior performance over SIR, we expect IOSIR will perform better than ISIR by a price of double calculation time.
We remark that SIR and ISIR can be used for both regression problems and classification problems. But since the concept of “adjacent slice” cannot be defined for categorical values (as is the case in classification problems), IOSIR can only be used for regression problems where the response variable is numeric.

heading: 6 SIMULATIONS

text: In this section, we will verify the effectiveness of ISIR and IOSIR with simulations on artificial and real-world data. Comparisons will be made between them and SIR.

heading: 6.1 ARTIFICIAL DATA

text:
In the simulations with artificial data, since we know the true model, we measure the performance by the accuracy of the estimated EDR space. We adopt the trace correlation r(K) = trace(PBPB̂)/K used in Ferré (1998) as the criterion, where PB and PB̂ are the projection operators onto the true EDR space B and the estimated EDR space B̂, respectively.
We consider the following model from Li (1991)
y = x1(x1 + x2 + 1) + , (13)
where x = [x1, x2, . . . , xp]> follows a multivariate normal distribution, follows standard normal distribution and is independent of x. It has K = 2 effective dimensions with β1 = (1, 0, 0, . . . , 0) > and β2 = (0, 1, 0 . . . , 0) >. We conduct the simulation in p = 10 dimensional space and select the number of slices as H = 10. We give the algorithm a warm start with the initial guess of the EDR space obtained by applying SIR algorithm to a small data set of 40 observations. Then a total of 400 new observations will be fed to update the EDR space one by one. SIR, ISIR, and IOSIR are applied when each observation was fed in and we calculate their trace correlation and cumulative computation time. We repeat this process 100 times. The mean trace correlations of all three methods are reported in Figure 1(a) and the mean cumulative time is in Figure 1(b). We see that ISIR performs quite similar to SIR. IOSIR slightly outperforms ISIR and SIR. ISIR is much faster than SIR. IOSIR gains higher accuracy by sacrificing on computation time. This verifies the convergence and efficiency of ISIR and IOSIR.

heading: 6.2 REAL DATA APPLICATION

text:
We validate the reliability of ISIR on two real data sets: the Concrete Compressive Strength and Cpusmall (available on https://www.csie.ntu.edu.tw/˜cjlin/libsvmtools/ datasets/regression.html). There have been many proposed algorithms to increase the prediction accuracy on these data sets (Yeh, 1998; 1999; 2007; Öztaş et al., 2006; Gu et al., 2015). We do not intend to outperform those methods. Our goal is to compare the performance of supervised dimension reduction algorithms and verify the effectiveness of our incremental methods.
The Concrete Compressive Strength data has p = 8 predictors and 1030 samples. We use H = 10 and K = 3 to run SIR, ISIR, and IOSIR. We select 50 observations to warm start ISIR and IOSIR algorithms, then 700 observations are fed sequentially. The left 280 observations are used as test data. After each new observation is received we estimate the EDR space, project the available training set to the estimated EDR space, build a regression model using the k-nearest neighbor method, and compute the MSE on the test data set. This process is repeated 100 times and the average MSE was reported in Figure 2(a). For the Cpusmall data, which has p = 12 predictors and 8192 samples, we do the experiment with H = 10, K = 3, 50 observations to warm start ISIR and IOSIR, 2000 observations for sequential training, and 6142 observations for testing. The average MSE was plotted in Figure 2(b). The results indicate both ISIR and IOSIR are as effective as SIR.

heading: 7 CONCLUSIONS AND DISCUSSIONS

text:
We proposed two online learning approaches for supervised dimension reduction, namely, ISIR and IOSIR. They are motivated by standardizing the data and reformulate the SIR algorithm to a PCA problem. However, data standardization is only used to motivate the algorithm while not explicitly calculated in the algorithms. We proposed to use Sherman Morrison formula to online update Σ−1 and some approximated calculations to circumvent explicit data standardization. This novel idea played a key role in our algorithm design. Both algorithms are shown effective and efficient. While IOSIR does not apply to classification problems, it is usually superior over ISIR in regression problems.
We remark that the purpose of ISIR and IOSIR is to keep the dimension reduction accuracy in the situation that a batch learning is not suitable. This is especially the case for streaming data where information update and system involving is necessary whenever new data becomes available. When the whole data set is given and one only needs the EDR space from batch learning, ISIR or IOSIR is not necessarily more efficient than SIR because their complexity to run over the whole sample path is O(p2N), comparable to the complexity O(p3 + p2N) of SIR.
There are two open problems worth further investigation. First, the need to store and use Σ−1 during the updating process is the main bottleneck for ISIR and IOSIR when the dimensionality of
the data is ultrahigh. Second, for SIR and other batch dimension reduction methods, many methods have been proposed to determine the intrinsic dimension K; see e.g. Li (1991); Schott (1994); Bura & Cook (2001); Bai & He (2004); Barrios & Velilla (2007); Nkiet (2008). They depend on all p eigenvalues of the generalized eigen-decomposition problem and are impractical for incremental learning. We do not have obvious solutions to these problems at this moment and would like to leave them for future research.


summaries: This paper presents 'Multimodal Factorization model' that factorizes representations into shared multimodal discriminative factors and modality specific generative factors. This work applies 'Wassertein Auto-Encoders' by Tolstikhin et al (with proofs that this setup works in the multimodal case) for handling factorized joint distributions over the multimodal space. Can this method be considered as a generalization of the wasserstein autoencoder based method with a broader application? - the authors should discuss this more broadly in the paper.

summaries: Multimodality learning is an important topic in multimedia and human computer interaction.  How to efficiently leverage the additional information cross multimodality is the key to the task. Authors proposed the Bayesian latent variable model to factorize the multimodality representation into multimodal discriminative factors and modality-specific generating factors, which is interesting. Approximate inference is also proposed to learn this model via a generalised mean-field assumption.

abstractText: Learning multimodal representations is a fundamentally complex research problem due to the presence of multiple heterogeneous sources of information. Although the presence of multiple modalities provides additional valuable information, there are two key challenges to address when learning from multimodal data: 1) models must learn the complex intra-modal and cross-modal interactions for prediction and 2) models must be robust to unexpected missing or noisy modalities during testing. In this paper, we propose to optimize for a joint generative-discriminative objective across multimodal data and labels. We introduce a model that factorizes representations into two sets of independent factors: multimodal discriminative and modality-specific generative factors. Multimodal discriminative factors are shared across all modalities and contain joint multimodal features required for discriminative tasks such as sentiment prediction. Modality-specific generative factors are unique for each modality and contain the information required for generating data. Experimental results show that our model is able to learn meaningful multimodal representations that achieve state-of-the-art or competitive performance on six multimodal datasets. Our model demonstrates flexible generative capabilities by conditioning on independent factors and can reconstruct missing modalities without significantly impacting performance. Lastly, we interpret our factorized representations to understand the interactions that influence multimodal learning.

authors:
Tadas Baltrušaitis
Chaitanya Ahuja
Louis-Philippe Morency

title: Multimodal machine learning: A survey and taxonomy

venue: arXiv preprint arXiv:1705.09406,

authors:
Yoshua Bengio
Aaron Courville
Pascal Vincent

title: Representation learning: A review and new perspectives

venue: IEEE Trans. Pattern Anal. Mach. Intell.,

authors:
Carlos Busso
Murtaza Bulut
Chi-Chun Lee
Abe Kazemzadeh
Emily Mower
Samuel Kim
Jeannette Chang
Sungbok Lee
Shrikanth S. Narayanan

title: Iemocap: Interactive emotional dyadic motion capture database

venue: Journal of Language Resources and Evaluation,

authors:
Devendra Singh Chaplot
Kanthashree Mysore Sathyendra
Rama Kumar Pasumarthi
Dheeraj Rajagopal
Ruslan Salakhutdinov

title: Gated-attention architectures for task-oriented language grounding

venue: arXiv preprint arXiv:1706.07230,

authors:
Minghai Chen
Sen Wang
Paul Pu Liang
Tadas Baltrušaitis
Amir Zadeh
Louis-Philippe Morency

title: Multimodal sentiment analysis with word-level fusion and reinforcement learning

venue: In Proceedings of the 19th ACM International Conference on Multimodal Interaction,

authors:
Xi Chen
Yan Duan
Rein Houthooft
John Schulman
Ilya Sutskever
Pieter Abbeel

title: Infogan: Interpretable representation learning by information maximizing generative adversarial nets

venue: In Advances in Neural Information Processing Systems,

authors:
Brian Cheung
Jesse A Livezey
Arjun K Bansal
Bruno A Olshausen

title: Discovering hidden factors of variation in deep networks

venue: arXiv preprint arXiv:1412.6583,

authors:
Kyunghyun Cho
Bart van Merriënboer
Çağlar Gülçehre
Dzmitry Bahdanau
Fethi Bougares
Holger Schwenk
Yoshua Bengio

title: Learning phrase representations using rnn encoder–decoder for statistical machine translation

venue: In EMNLP. ACL,

authors:
KyungHyun Cho
Aaron C. Courville
Yoshua Bengio

title: Describing multimedia content using attention-based encoder-decoder networks

authors:
Marco Cuturi
Jean-Philippe Vert
Oystein Birkenes
Tomoko Matsui

title: A kernel for time series based on global alignments

venue: In Acoustics, Speech and Signal Processing,

authors:
Gilles Degottex
John Kane
Thomas Drugman
Tuomo Raitio
Stefan Scherer

title: Covarepa collaborative voice analysis repository for speech technologies

venue: In ICASSP

title: An argument for basic emotions

venue: Cognition & emotion,

authors:
Paul Ekman
Wallace V Freisen
Sonia Ancoli

title: Facial signs of emotional experience

venue: Journal of personality and social psychology,

authors:
C.A. Frantzidis
C. Bratsas
M.A. Klados
E. Konstantinidis
C.D. Lithari
A.B. Vivas
C.L. Papadelis
E. Kaldoudi
C. Pappas
P.D. Bamidis

title: On the classification of emotional biosignals evoked while viewing affective pictures: An integrated data-mining-based approach for healthcare applications

venue: IEEE Transactions on Information Technology in Biomedicine,

authors:
Andrea Frome
Greg S Corrado
Jon Shlens
Samy Bengio
Jeff Dean
Tomas Mikolov

title: Devise: A deep visual-semantic embedding model

venue: In NIPS,

authors:
A. Graves
A. r. Mohamed
G. Hinton

title: Speech recognition with deep recurrent neural networks

venue: IEEE International Conference on Acoustics, Speech and Signal Processing,

authors:
Arthur Gretton
Olivier Bousquet
Alex Smola
Bernhard Schölkopf

title: Measuring statistical dependence with hilbert-schmidt norms

venue: In International conference on algorithmic learning theory,

authors:
Arthur Gretton
Karsten M Borgwardt
Malte J Rasch
Bernhard Schölkopf
Alexander Smola

title: A kernel two-sample test

venue: Journal of Machine Learning Research,

authors:
Sylvain Guimond
Wael Massrieh

title: Intricate correlation between body posture, personality trait and incidence of body pain: A cross-referential study report

venue: PLOS ONE,

authors:
Irina Higgins
Loic Matthey
Arka Pal
Christopher Burgess
Xavier Glorot
Matthew Botvinick
Shakir Mohamed
Alexander Lerchner

title: beta-vae: Learning basic visual concepts with a constrained variational framework

venue: In ICLR,

authors:
Sepp Hochreiter
Jürgen Schmidhuber

title: Long short-term memory

venue: Neural computation,

authors:
Wei-Ning Hsu
James Glass

title: Disentangling by partitioning: A representation learning framework for multimodal sensory data

venue: arXiv preprint arXiv:1805.11264,

authors:
Theofanis Karaletsos
Serge Belongie
Gunnar Rätsch

title: Bayesian representation learning with oracle constraints

venue: arXiv preprint arXiv:1506.05011,

authors:
Diederik P Kingma
Max Welling

title: Auto-encoding variational bayes

venue: arXiv preprint arXiv:1312.6114,

title: A new view of language acquisition

venue: Proceedings of the National Academy of Sciences,

authors:
Tejas D Kulkarni
William F Whitney
Pushmeet Kohli
Josh Tenenbaum

title: Deep convolutional inverse graphics network

authors:
Brenden M Lake
Tomer D Ullman
Joshua B Tenenbaum
Samuel J Gershman

title: Building machines that learn and think like people

venue: Behavioral and Brain Sciences,

authors:
Yann Lecun
Léon Bottou
Yoshua Bengio
Patrick Haffner

title: Gradient-based learning applied to document recognition

venue: In Proceedings of the IEEE,

authors:
Sergey Levine
Christian Theobalt
Vladlen Koltun

title: Real-time prosody-driven synthesis of body language

venue: ACM Trans. Graph.,

authors:
Paul Pu Liang
Amir Zadeh
Louis-Philippe Morency

title: Multimodal local-global ranking fusion for emotion recognition

venue: In Proceedings of the 20th ACM International Conference on Multimodal Interaction,

authors:
Z. Liu
M. Wu
W. Cao
L. Chen
J. Xu
R. Zhang
M. Zhou
J. Mao

title: A facial expression emotion recognition based human-robot interaction system

venue: IEEE/CAA Journal of Automatica Sinica,

authors:
Zhun Liu
Ying Shen
Paul Pu Liang
AmirAli Bagher Zadeh
Louis-Philippe Morency

title: Efficient low-rank multimodal fusion with modality-specific factors

authors:
Gelareh Mohammadi
Alessandro Vinciarelli
Marcello Mortillaro

title: The voice of personality: Mapping nonverbal vocal behavior into trait attributions

venue: In Proceedings of ACM Multimedia Workshop on Social Signal Processing,

authors:
Louis-Philippe Morency
Ariadna Quattoni
Trevor Darrell

title: Latent-dynamic discriminative models for continuous gesture recognition

venue: In CVPR. IEEE,

authors:
Louis-Philippe Morency
Rada Mihalcea
Payal Doshi

title: Towards multimodal sentiment analysis: Harvesting opinions from the web

venue: In ICMI,

authors:
Yuval Netzer
Tao Wang
Adam Coates
Alessandro Bissacco
Bo Wu
Andrew Y. Ng

title: Reading digits in natural images with unsupervised feature learning

authors:
Jiquan Ngiam
Aditya Khosla
Mingyu Kim
Juhan Nam
Honglak Lee
Andrew Y Ng

title: Multimodal deep learning

authors:
Behnaz Nojavanasghari
Deepak Gopinath
Jayanth Koushik
Tadas Baltrušaitis
Louis-Philippe Morency

title: Deep multimodal fusion for persuasiveness prediction

venue: In Proceedings of the 18th ACM International Conference on Multimodal Interaction,

authors:
Sunghyun Park
Han Suk Shim
Moitreya Chatterjee
Kenji Sagae
Louis-Philippe Morency

title: Computational analysis of persuasiveness in social multimedia: A novel dataset and multimodal prediction approach

venue: ICMI ’14,

authors:
Jeffrey Pennington
Richard Socher
Christopher D Manning

title: Glove: Global vectors for word representation

venue: In EMNLP,

authors:
Veronica Perez-Rosas
Rada Mihalcea
Louis-Philippe Morency

title: Utterance-Level Multimodal Sentiment Analysis

venue: In Association for Computational Linguistics (ACL),

authors:
Sintija Petrovica
Alla Anohina-Naumeca
HazÄ±m Kemal Ekenel

title: Emotion recognition in affective tutoring systems: Collection of ground-truth data

venue: Procedia Computer Science,

authors:
Hai Pham
Thomas Manzini
Paul Pu Liang
Barnabas Poczos

title: Seq2seq2sentiment: Multimodal sequence to sequence models for sentiment analysis

venue: In Proceedings of Grand Challenge and Workshop on Human Multimodal Language (Challenge-HML)

authors:
Johannes Pittermann
Angela Pittermann
Wolfgang Minker

title: Emotion recognition and adaptation in spoken dialogue systems

venue: International Journal of Speech Technology,

authors:
Soujanya Poria
Erik Cambria
Devamanyu Hazarika
Navonil Majumder
Amir Zadeh
LouisPhilippe Morency

title: Context-dependent sentiment analysis in user-generated videos

authors:
Ariadna Quattoni
Sybor Wang
Louis-Philippe Morency
Michael Collins
Trevor Darrell

title: Hidden conditional random fields

authors:
Shyam Sundar Rajagopalan
Louis-Philippe Morency
Tadas Baltrušaitis
Goecke Roland

title: Extending long short-term memory for multi-view structured learning

venue: In ECCV,

authors:
Scott Reed
Kihyuk Sohn
Yuting Zhang
Honglak Lee

title: Learning to disentangle factors of variation with manifold interaction

venue: In International Conference on Machine Learning,

authors:
Paul K Rubenstein
Bernhard Schoelkopf
Ilya Tolstikhin

title: On the latent space of wasserstein auto-encoders

venue: arXiv preprint arXiv:1802.03761,

authors:
M. Schuster
K.K. Paliwal

title: Bidirectional recurrent neural networks

venue: Trans. Sig. Proc.,

authors:
Richard Socher
Milind Ganjoo
Christopher D Manning
Andrew Ng

title: Zero-shot learning through cross-modal transfer

venue: In Advances in neural information processing systems,

authors:
Kihyuk Sohn
Wenling Shang
Honglak Lee

title: Improved multimodal deep learning with variation of information

authors:
Yale Song
Louis-Philippe Morency
Randall Davis

title: Multi-view latent variable discriminative models for action recognition

venue: In CVPR,

title: Action recognition by hierarchical sequence summarization

authors:
Nitish Srivastava
Ruslan R Salakhutdinov

title: Multimodal learning with deep boltzmann machines

authors:
Masashi Sugiyama
Makoto Yamada

title: On kernel parameter selection in hilbert-schmidt independence criterion

venue: IEICE TRANSACTIONS on Information and Systems,

authors:
Masahiro Suzuki
Kotaro Nakayama
Yutaka Matsuo

title: Joint multimodal learning with deep generative models

venue: arXiv preprint arXiv:1611.01891,

authors:
Yao-Hung Hubert Tsai
Ruslan Salakhutdinov

title: Improving one-shot learning through fusing side information

venue: arXiv preprint arXiv:1710.08347,

authors:
Yao-Hung Hubert Tsai
Liang-Kang Huang
Ruslan Salakhutdinov

title: Learning robust visualsemantic embeddings

venue: arXiv preprint arXiv:1703.05908,

authors:
Martin Wöllmer
Felix Weninger
Tobias Knaup
Björn Schuller
Congkai Sun
Kenji Sagae
Louis-Philippe Morency

title: Youtube movie reviews: Sentiment analysis in an audio-visual context

venue: IEEE Intelligent Systems,

authors:
Denny Wu
Yixiu Zhao
Yao-Hung Hubert Tsai
Makoto Yamada
Ruslan Salakhutdinov

title: dependency bottleneck" in auto-encoding architectures: an empirical study

venue: arXiv preprint arXiv:1802.05408,

authors:
Jimei Yang
Scott E Reed
Ming-Hsuan Yang
Honglak Lee

title: Weakly-supervised disentangling with recurrent transformations for 3d view synthesis

authors:
Jiahong Yuan
Mark Liberman

title: Speaker identification on the scotus corpus

venue: Journal of the Acoustical Society of America,

authors:
Amir Zadeh
Rowan Zellers
Eli Pincus
Louis-Philippe Morency

title: Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages

authors:
Amir Zadeh
Minghai Chen
Soujanya Poria
Erik Cambria
Louis-Philippe Morency

title: Tensor fusion network for multimodal sentiment analysis

authors:
Amir Zadeh
Paul Pu Liang
Navonil Mazumder
Soujanya Poria
Erik Cambria
Louis-Philippe Morency

title: Memory fusion network for multi-view sequential learning

authors:
Amir Zadeh
Paul Pu Liang
Soujanya Poria
Prateek Vij
Erik Cambria
Louis-Philippe Morency

title: Multi-attention recurrent network for human communication comprehension

authors:
Shengjia Zhao
Jiaming Song
Stefano Ermon

title: Infovae: Information maximizing variational autoencoders

venue: arXiv preprint arXiv:1706.02262,

title: 2018) use an additional hinge loss to separate the latent factors

heading: 1 INTRODUCTION

text:
Multimodal machine learning involves learning from data across multiple modalities (Baltrušaitis et al., 2017). It is a challenging yet crucial research area with real-world applications in robotics (Liu et al., 2017), dialogue systems (Pittermann et al., 2010), intelligent tutoring systems (Petrovica et al., 2017), and healthcare diagnosis (Frantzidis et al., 2010). At the heart of many multimodal modeling tasks lies the challenge of learning rich representations from multiple modalities. For example, analyzing multimedia content requires learning multimodal representations across the language, visual, and acoustic modalities (Cho et al., 2015). Although the presence of multiple modalities provides additional valuable information, there are two key challenges to address when learning from multimodal data: 1) models must learn the complex intra-modal and cross-modal interactions for prediction (Zadeh et al., 2017), and 2) trained models must be robust to unexpected missing or noisy modalities during testing (Ngiam et al., 2011).
In this paper, we propose to optimize for a joint generative-discriminative objective across multimodal data and labels. The discriminative objective ensures that the representations learned are rich in intra-modal and cross-modal features useful towards predicting the label, while the generative objective allows the model to infer missing modalities at test time and deal with the presence of noisy modalities. To this end, we introduce the Multimodal Factorization Model (MFM in Figure 1) that factorizes multimodal representations into multimodal discriminative factors and modality-specific generative factors. Multimodal discriminative factors are shared across all modalities and contain joint multimodal features required for discriminative tasks. Modality-specific generative factors are unique for each modality and contain the information required for generating each modality. We believe that factorizing multimodal representations into different explanatory factors can help each factor focus on learning from a subset of the joint information across multimodal data and labels. This method is in contrast to jointly learning a single factor that summarizes all generative and
∗equal contributions
discriminative information (Srivastava & Salakhutdinov, 2012). To sum up, MFM defines a joint distribution over multimodal data, and by the conditional independence assumptions in the assumed graphical model, both generative and discriminative aspects are taken into account. Our model design further provides interpretability of the factorized representations.
Through an extensive set of experiments, we show that MFM learns improved multimodal representations with these characteristics: 1) The multimodal discriminative factors achieve state-of-the-art or competitive performance on six multimodal time series datasets. We also demonstrate that MFM can generalize by integrating it with other existing multimodal discriminative models. 2) MFM allows flexible generation concerning multimodal discriminative factors (labels) and modality-specific generative factors (styles). We further show that we can perform reconstruction of missing modalities from observed modalities without significantly impacting discriminative performance. Finally, we interpret our learned representations using information-based and gradient-based methods, allowing us to understand the contributions of individual factors towards multimodal prediction and generation.

heading: 2 MULTIMODAL FACTORIZATION MODEL

text:
Multimodal Factorization Model (MFM) is a latent variable model (Figure 1(a)) with conditional independence assumptions over multimodal discriminative factors and modality-specific generative factors. According to these assumptions, we propose a factorization over the joint distribution of multimodal data (Section 2.1). Since exact posterior inference on this factorized distribution can be intractable, we propose an approximate inference algorithm based on minimizing a joint-distribution Wasserstein distance over multimodal data (Section 2.2). Finally, we derive the MFM objective by approximating the joint-distribution Wasserstein distance via a generalized mean-field assumption.
Notation: We define X1∶M as the multimodal data from M modalities and Y as the labels, with joint distribution PX1∶M ,Y = P (X1∶M ,Y). Let X̂1∶M denote the generated multimodal data and Ŷ denote the generated labels, with joint distribution PX̂1∶M ,Ŷ = P (X̂1∶M , Ŷ).

heading: 2.1 FACTORIZED MULTIMODAL REPRESENTATIONS

text:
To factorize multimodal representations into multimodal discriminative factors and modality-specific generative factors, MFM assumes a Bayesian network structure as shown in Figure 1(a). In this graphical model, factors Fy and Fa{1∶M} are generated from mutually independent latent variables Z = [Zy,Za{1∶M}] with prior PZ. In particular, Zy generates the multimodal discriminative factor Fy and Za{1∶M} generate modality-specific generative factors Fa{1∶M}. By construction, Fy contributes to the generation of Ŷ while {Fy,Fai} both contribute to the generation of X̂i. As a result, the joint distribution P (X̂1∶M , Ŷ) can be factorized as follows:
P (X̂1∶M , Ŷ) = ∫ F,Z P (X̂1∶M , Ŷ∣F)P (F∣Z)P (Z)dFdZ
= ∫Fy,Fa{1∶M} Zy,Za{1∶M}
(P (Ŷ∣Fy) M
∏ i=1
P (X̂i∣Fai,Fy))(P (Fy ∣Zy) M
∏ i=1
P (Fai∣Zai))(P (Zy) M
∏ i=1
P (Zai))dFdZ, (1)
with dF = dFy∏Mi=1 dFai and dZ = dZy∏ M i=1 dZai.
Exact posterior inference in Equation 1 may be analytically intractable due to the integration over Z. We therefore resort to using an approximate inference distribution Q(Z∣X1∶M ,Y) as detailed in the
following subsection. As a result, MFM can be viewed as an autoencoding structure that consists of encoder (inference) and decoder (generative) modules (Figure 1(c)). The encoder module for Q(⋅∣⋅) allows us to easily sample Z from an approximate posterior. The decoder modules are parametrized according to the factorization of P (X̂1∶M , Ŷ∣Z) as given by Equation 1 and Figure 1(a).

heading: 2.2 MINIMIZING JOINT-DISTRIBUTION WASSERSTEIN DISTANCE OVER MULTIMODAL DATA

text:
Two common choices for approximate inference in autoencoding structures are Variational Autoencoders (VAEs) (Kingma & Welling, 2013) and Wasserstein Autoencoders (WAEs) (Zhao et al., 2017; Tolstikhin et al., 2017). The former optimizes the evidence lower bound objective (ELBO), and the latter derives an approximation for the primal form of the Wasserstein distance. We consider the latter since it simultaneously results in better latent factor disentanglement (Zhao et al., 2017; Rubenstein et al., 2018) and better sample generation quality than its counterparts (Chen et al., 2016; Higgins et al., 2016; Kingma & Welling, 2013). However, WAEs are designed for unimodal data and do not consider factorized distributions over latent variables that generate multimodal data. Therefore, we propose a variant for handling factorized joint distributions over multimodal data.
As suggested by Kingma & Welling (2013), we adopt the design of nonlinear mappings (i.e. neural network architectures) in the encoder and decoder (Figure 1 (c)). For the encoder Q(Z∣X1∶M ,Y), we learn a deterministic mapping Qenc ∶X1∶M ,Y → Z (Rubenstein et al., 2018; Tolstikhin et al., 2017). For the decoder, we define the generation process from latent variables as Gy ∶ Zy → Fy, Ga{1∶M} ∶ Za{1∶M} → Fa{1∶M}, D ∶ Fy → Ŷ, and F1∶M ∶ Fy,Fa{1∶M} → X̂1∶M , where Gy,Ga{1∶M},D and F1∶M are deterministic functions parametrized by neural networks.
Let Wc(PX1∶M ,Y, PX̂1∶M ,Ŷ) denote the joint-distribution Wasserstein distance over multimodal data under cost function cXi and cY . We choose the squared cost c(a, b) = ∥a − b∥22, allowing us to minimize the 2-Wasserstein distance. The cost function can be defined not only on static data but also on time series data such as text, audio and videos. For example, given time series data X = [X1,X2,⋯,XT ] and X̂ = [X̂1, X̂2,⋯, X̂T ], we define c(X, X̂) = ∑Tt=1 ∥Xt − X̂t∥22. With conditional independence assumptions in Equation 1, we express Wc(PX1∶M ,Y, PX̂1∶M ,Ŷ) as:
Proposition 1. For any functions Gy ∶ Zy → Fy, Ga{1∶M} ∶ Za{1∶M} → Fa{1∶M}, D ∶ Fy → Ŷ, and F1∶M ∶ Fa{1∶M},Fy → X̂1∶M , we have Wc(PX1∶M ,Y, PX̂1∶M ,Ŷ) =
inf QZ=PZ EPX1∶M,YEQ(Z∣X1∶M ,Y) ⎡⎢⎢⎢⎢⎣ M ∑ i=1 cXi(Xi, Fi(Gai(Zai),Gy(Zy))) + cY (Y,D(Gy(Zy))) ⎤⎥⎥⎥⎥⎦ , (2) where PZ is the prior over Z = [Zy,Za{1,M}] and QZ is the aggregated posterior of the proposed approximate inference distribution Q(Z∣X1∶M ,Y). Proof: The proof is adapted from Tolstikhin et al. (Tolstikhin et al., 2017). The two differences are: (1) we show that P (X̂1∶M , Ŷ∣Z = z) are Dirac for all z ∈ Z , and (2) we use the fact that c((X1∶M ,Y), (X̂1∶M , Ŷ)) = ∑Mi=1 cXi(Xi, X̂i) + cY (Y, Ŷ). Please refer to the supplementary material for proof details. ∎ The constraint on QZ = PZ in Proposition 1 is hard to satisfy. To obtain a numerical solution, we first relax the constraint by performing a generalized mean field assumption on Q according to the conditional independence as shown in the inference network of Figure 1 (b):
Q(Z∣X1∶M ,Y) ∶= Q(Z∣X1∶M) ∶= Q(Zy∣X1∶M) M
∏ i=1 Q(Zai∣Xi). (3)
The intuition here is based on our design that Zy generates the multimodal discriminative factor Fy and Za{1∶M} generate modality-specific generative factors Fa{1∶M}. Therefore, the inference for Zy should depend on all modalities X1∶M and the inference for Zai should depend only on the specific modality Xi. Following this assumption, we define Q as a nonparametric set of all encoders that fulfill the factorization in Equation 3. A penalty term is added into our objective to find the Q(Z∣⋅) ∈ Q that is the closest to prior PZ, thereby approximately enforcing the constraint QZ = PZ:
min F,Ga{1∶M},Gy,D inf Q(Z∣⋅)∈Q EPX1∶M,YEQ(Za1 ∣X1)⋯EQ(ZaM ∣XM )EQ(Zy ∣X1∶M ) ⎡⎢⎢⎢⎢⎣ M ∑ i=1 cXi(Xi, F (Gai(Zai),Gy(Zy))) + cY (Y,D(Gy(Zy))) ⎤⎥⎥⎥⎥⎦ + λMMD(QZ, PZ),
(4)
where λ is a hyper-parameter andMMD is the Maximum Mean Discrepancy (Gretton et al., 2012) as a divergence measure between QZ and PZ. The prior PZ is chosen as a centered isotropic Gaussian N (0, I), so that it implicitly enforces independence between the latent variables Z = [Zy,Za{1,M}] (Higgins et al., 2016; Kingma & Welling, 2013; Rubenstein et al., 2018). Equation 4 represents our hybrid generative-discriminative optimization objective over multimodal data: the first loss term ∑Mi=1 cXi(Xi, F (Gai(Zai),Gy(Zy))) is the generative objective based on reconstruction of multimodal data and the second term cY (Y,D(Gy(Zy))) is the discriminative objective. In practice we compute the expectations in Equation 4 using empirical estimates over the training data. The neural architecture of MFM is illustrated in Figure 1(c).

heading: 2.3 SURROGATE INFERENCE FOR MISSING MODALITIES

text:
A key challenge in multimodal learning involves dealing with missing modalities. A good multimodal model should be able to infer the missing modality conditioned on the observed modalities and perform predictions based only on the observed modalities. To achieve this objective, the inference process of MFM can be easily adapted using a surrogate inference network to reconstruct the missing modality given the observed modalities. Formally, let Φ denote the surrogate inference network. The generation of missing modality X̂1 given the observed modalities X2∶M can be formulated as
Φ∗ = argmin Φ EPX2∶M,X̂1 ( − logPΦ(X̂1∣X2∶M))
with PΦ(X̂1∣X2∶M) ∶= ∫ P (X̂1∣Za1,Zy)QΦ(Za1∣X2∶M)QΦ(Zy∣X2∶M)dZa1dZy. (5)
Similar to Section 2.2, we use deterministic mappings in QΦ(⋅∣⋅) and QΦ(Zy∣⋅) is also used for prediction PΦ(Ŷ∣X2∶M) ∶= ∫ P (Ŷ∣Zy)QΦ(Zy∣X2∶M)dZy. Equation 5 suggests that in the presence of missing modalities, we only need to infer the latent codes rather than the entire modality.

heading: 2.4 ENCODER AND DECODER DESIGN

text: We now discuss the implementation choices for the MFM neural architecture in Figure 1(c). The encoder Q(Zy∣X1∶M) can be parametrized by any model that performs multimodal fusion (Morency et al., 2011; Zadeh et al., 2017). For multimodal image datasets, we adopt Convolutional Neural Networks (CNNs) and Fully-Connected Neural Networks (FCNNs) with late fusion (Nojavanasghari et al., 2016) as our encoder Q(Zy∣X1∶M). The remaining functions in MFM are also parametrized by CNNs and FCNNs. For multimodal time series datasets, we choose the Memory Fusion Network (MFN) (Zadeh et al., 2018a) as our multimodal encoder Q(Zy∣X1∶M). We use Long Short-term Memory (LSTM) networks (Hochreiter & Schmidhuber, 1997) for functions Q(Za{1∶M}∣X1∶M), decoder LSTM networks (Cho et al., 2014) for functions F1∶M , and FCNNs for functions Gy, Ga{1∶M} and D. Details are provided in the appendix and the code is available at <anonymous>.

heading: 3 EXPERIMENTS

text:
In order to show that MFM learns multimodal representations that are discriminative, generative and interpretable, we design the following experiments. We begin with a multimodal synthetic image dataset that allows us to examine whether MFM displays discriminative and generative capabilities from factorized latent variables. Utilizing image datasets allows us to clearly visualize the generative capabilities of MFM. We then transition to six more challenging real-world multimodal video datasets
to 1) rigorously evaluate the discriminative capabilities of MFM in comparison with existing baselines, 2) analyze the importance of each design component through ablation studies, 3) assess the robustness of MFM’s modality reconstruction and prediction capabilities to missing modalities, and 4) interpret the learned representations using information-based and gradient-based methods to understand the contributions of individual factors towards multimodal prediction and generation.

heading: 3.1 MULTIMODAL SYNTHETIC IMAGE DATASET

text:
In this section, we study MFM on a synthetic image dataset that considers SVHN (Netzer et al., 2011) and MNIST (Lecun et al., 1998) as the two modalities. SVHN and MNIST are images with different styles but the same labels (digits 0 ∼ 9). We randomly pair 100,000 SVHN and MNIST images that have the same label, creating a multimodal dataset which we call SVHN+MNIST. 80,000 pairs are used for training and the rest for testing. To justify that MFM is able to learn improved multimodal representations, we show both classification and generation results on SVHN+MNIST in Figure 2.
Prediction: We perform experiments on both unimodal and multimodal classification tasks. UM denotes a unimodal baseline that performs prediction given only one modality as input and MM denotes a multimodal discriminative baseline that performs prediction given both images (Nojavanasghari et al., 2016). We compare the results for UM(SVHN), UM(MNIST), MM and MFM on SVHN+MNIST in Figure 2(b). We achieve better classification performance from unimodal to multimodal which is not surprising since more information is given. More importantly, MFM outperforms MM, which suggests that MFM learns improved factorized representations for discriminative tasks.
Generation: We generate images using the MFM generative network (Figure 2(a)). We fix one variable out of Z = [Za1, Za2, and Zy] and randomly sample the other two variables from prior PZ. From Figure 2(c), we observe that MFM shows flexible generation of SVHN and MNIST images based on labels and styles. This suggests that MFM is able to factorize multimodal representations into multimodal discriminative factors (labels) and modality-specific generative factors (styles).

heading: 3.2 MULTIMODAL TIME SERIES DATASETS

text:
In this section, we transition to more challenging multimodal time series datasets. All the datasets consist of monologue videos. Features are extracted from the language (GloVe word embeddings (Pen-
nington et al., 2014)), visual (Facet (iMotions, 2017)), and acoustic (COVAREP (Degottex et al., 2014)) modalities. For a detailed description of feature extraction, please refer to the appendix.
We consider the following six datasets across three domains: 1) Multimodal Personality Trait Recognition: POM (Park et al., 2014) contains 903 movie review videos annotated for the following personality traits: confident (con), passionate (pas), voice pleasant (voi), dominant (dom), credible (cre), vivid (viv), expertise (exp), entertaining (ent), reserved (res), trusting (tru), relaxed (rel), outgoing (out), thorough (tho), nervous (ner), persuasive (per) and humorous (hum). The short form is indicated in parenthesis. 2) Multimodal Sentiment Analysis: CMU-MOSI (Zadeh et al., 2016) is a collection of 2199 monologue opinion video clips annotated with sentiment. ICT-MMMO (Wöllmer et al., 2013) consists of 340 online social review videos annotated for sentiment. YouTube (Morency et al., 2011) contains 269 product review and opinion video segments from YouTube each annotated for sentiment. MOUD (Perez-Rosas et al., 2013) consists of 79 product review videos in Spanish. Each video consists of multiple segments labeled as either positive, negative or neutral sentiment. 3) Multimodal Emotion Recognition: IEMOCAP (Busso et al., 2008) consists of 302 videos of recorded dyadic dialogues. The videos are divided into multiple segments each annotated for the presence of 6 discrete emotions (happy, sad, angry, frustrated, excited and neutral), resulting in a total of 7318 segments in the dataset. We report results using the following metrics: Acc_C = multiclass accuracy across C classes, F1 = F1 score, MAE = Mean Absolute Error, r = Pearson’s correlation.
Prediction: We first compare the performance of MFM with existing multimodal prediction methods. For a detailed description of the baselines, please refer to the appendix. From Table 1, we first observe that the best performing baseline results are achieved by different models across different datasets (most notably MFN, MARN, and TFN). On the other hand, MFM consistently achieves state-of-the-art or competitive results for all six multimodal datasets. We believe that the multimodal discriminative factor Fy in MFM has successfully learned more meaningful representations by distilling discriminative features. This highlights the benefit of learning factorized multimodal representations towards discriminative tasks. Furthermore, MFM is model-agnostic and can be applied to other multimodal encoders Q(Zy∣X1∶M). We perform experiments to show consistent improvements in discriminative performance for several choices of the encoder: EF-LSTM (Morency et al., 2011) and TFN (Zadeh et al., 2017). For Acc_2 on CMU-MOSI, our factorization framework improves the performance of EF-LSTM from 74.3 to 75.2 and TFN from 74.6 to 75.5.
Ablation Study: In Figure 3, we present the models M{A,B,C,D,E} used for ablation studies. These models are designed to analyze the effects of using a multimodal discriminative factor, a hybrid generative-discriminative objective, factorized generative-discriminative factors and modality-specific generative factors towards both modality reconstruction and label prediction. The simplest variant is MA which represents a purely discriminative model without a joint multimodal discriminative factor (i.e. early fusion (Morency et al., 2011)). MB models a joint multimodal discriminative factor which incorporates more general multimodal fusion encoders (Zadeh et al., 2018a). MC extends
MA by optimizing a hybrid generative-discriminative objective over modality-specific factors. MD extends MB by optimizing a hybrid generative-discriminative objective over a joint multimodal factor (resembling Srivastava & Salakhutdinov (2012)). ME factorizes the representation into separate generative and discriminative factors. Finally, MFM is obtained from ME by using modality-specific generative factors instead of a joint multimodal generative factor.
From the table in Figure 3, we observe the following general trends. For sentiment prediction, using 1) a multimodal discriminative factor outperforms modality-specific discriminative factors (MD >MC, MB >MA), and 2) adding generative capabilities to the model improves performance (MC >MA, ME >MB). For both sentiment prediction and modality reconstruction, 3) factorizing into separate generative and discriminative factors improves performance (ME >MD), and 4) using modality-specific generative factors outperforms multimodal generative factors (MFM >ME). These observations support our design decisions of factorizing multimodal representations into multimodal discriminative factors and modality-specific generative factors.
Missing Modalities: We now evaluate the performance of MFM in the presence of missing modalities using the surrogate inference model as described in Subsection 2.3. We compare with two baselines: 1) a purely generative Seq2Seq model (Cho et al., 2014) ΦG from observed modalities to missing modalities by optimizing EPX1∶M (−logPΦD(X1∣X2∶M)), and 2) a purely discriminative model ΦD from observed modalities to the label by optimizing EPX2∶M,Y (−logPΦG(Y∣X2∶M)). Both models are modified from MFM by using only the two observed modalities as input and not explicitly accounting for missing modalities. We compare the reconstruction error of each modality (language, visual and acoustic) as well as the performance on sentiment prediction.
Table 2 shows that MFM with missing modalities outperforms the generative (ΦG) or discriminative baselines (ΦD) in terms of modality reconstruction and sentiment prediction. Additionally, MFM with missing modalities performs close to MFM with all modalities observed. This fact indicates that MFM can learn representations that are relatively robust to missing modalities. In addition, discriminative performance is most affected when the language modality is missing, which is consistent with prior work which indicates that language is most informative in human multimodal language (Zadeh et al., 2017). On the other hand, sentiment prediction is more robust to missing acoustic and visual features. Finally, we observe that reconstructing the low-level acoustic and visual features is easier as compared to the high-dimensional language features that contain high-level semantic meaning.
Interpretation of Multimodal Representations: We devise two methods to study how individual factors in MFM influence the dynamics of multimodal prediction and generation. These interpretation methods represent both overall trends and fine-grained analysis that could be useful towards deeper understandings of multimodal representation learning. For more details, please refer to the appendix.
Firstly, an information-based interpretation method is chosen to summarize the contribution of each modality towards the multimodal representations. Since Fy is a common cause of X̂1∶M , we can compare MI(Fy, X̂1), ⋯,MI(Fy, X̂M), where MI(⋅, ⋅) denotes the mutual information measure between Fy and generated modality X̂i. Higher MI(Fy, X̂i) indicates greater contribution from Fy to X̂i. Figure 4 reports the ratios ri = MI(Fy, X̂i)/MI(Fai, X̂i) which measure a normalized version of the mutual information between Fai and X̂i. We observe that on CMU-MOSI, the language modality is most informative towards sentiment prediction, followed by the acoustic modality. We
Umm, in a way, a lot of the themes in “never let me go”, which were very profound and deep.
(hesitancy) (emphasis) (neutral)
language
visual
acoustic
𝑡 = 1 𝑡 = 20
(u ni nf or m at iv e)
(u ni nf or m at iv e)
(s lig
ht sm
ile )
believe that this result represents a prior over the expression of sentiment in human multimodal language and is closely related to the connections between language and speech (Kuhl, 2000).
Secondly, a gradient-based interpretation method to used analyze the contribution of each modality for every time step in multimodal time series data. We measure the gradient of the generated modality with respect to the target factors (e.g., Fy). Let {x1, x2,⋯, xM} denote multimodal time series data where xi represents modality i, and x̂i = [x̂1i ,⋯, x̂ti,⋯, x̂Ti ] denote generated modality i across time steps t ∈ [1, T ]. The gradient ∇fy(x̂i) measures the extent to which changes in factor fy ∼ P (Fy∣X1∶M = x1∶M) influences the generation of sequence x̂i. Figure 4 plots ∇fy(x̂i) for a video in CMU-MOSI. We observe that multimodal communicative behaviors that are indicative of speaker sentiment such as positive words (e.g. “very profound and deep”) and informative acoustic features (e.g. hesitant and emphasized tone of voice) indeed correspond to increases in ∇fy(x̂i).

heading: 4 RELATED WORK

text:
The two main pillars of research in multimodal representation learning have considered the discriminative and generative objectives individually. Discriminative representation learning (Liang et al., 2018; Chen et al., 2017; Chaplot et al., 2017; Frome et al., 2013; Socher et al., 2013; Tsai et al., 2017) models the conditional distribution P (Y∣X1∶M). Since these approaches are not concerned with modeling P (X1∶M) explicitly, they use parameters more efficiently to model P (Y∣X1∶M). For instance, recent works learn visual representations that are maximally dependent with linguistic attributes for improving one-shot image recognition (Tsai & Salakhutdinov, 2017) or introduce tensor product mechanisms to model interactions between the language, visual and acoustic modalities (Liu et al., 2018; Zadeh et al., 2017). On the other hand, generative representation learning captures the interactions between modalities by modeling the joint distribution P (X1,⋯,XM) using either undirected graphical models (Srivastava & Salakhutdinov, 2012), directed graphical models (Suzuki et al., 2016b), or neural networks (Sohn et al., 2014). Some generative approaches compress multimodal data into lower-dimensional feature vectors which can be used for discriminative tasks (Pham et al., 2018; Ngiam et al., 2011). To unify the advantages of both approaches, MFM factorizes multimodal representations into generative and discriminative components and optimizes for a joint objective.
Factorized representation learning resembles learning disentangled data representations which have been shown to improve the performance on many tasks (Kulkarni et al., 2015; Lake et al., 2017; Higgins et al., 2016; Bengio et al., 2013). Several methods involve specifying a fixed set of latent attributes that individually control particular variations of data and performing supervised training (Cheung et al., 2014; Karaletsos et al., 2015; Yang et al., 2015; Reed et al., 2014; Zhu et al., 2014), assuming an isotropic Gaussian prior over latent variables to learn disentangled generative representations (Kingma & Welling, 2013; Rubenstein et al., 2018) and learning latent variables in charge of specific variations in the data by maximizing the mutual information between a subset of latent variables and the data (Chen et al., 2016). However, these methods study factorization of a single modality. MFM factorizes multimodal representations and demonstrates the importance of modality-specific and multimodal factors towards generation and prediction. A concurrent and parallel work that factorizes latent factors in multimodal data was proposed by Hsu & Glass (2018). They differ from us in the graphical model design, discriminative objective, prior matching criterion, and scale of experiments. We provide a detailed comparison with their model in the appendix.

heading: 5 CONCLUSION

text:
In this paper, we proposed the Multimodal Factorization Model (MFM) for multimodal representation learning. MFM factorizes the multimodal representations into two sets of independent factors: multimodal discriminative factors and modality-specific generative factors. The multimodal discriminative
factor achieves state-of-the-art or competitive results on six multimodal datasets. The modalityspecific generative factors allow us to generate data based on factorized variables, account for missing modalities, and have a deeper understanding of the interactions involved in multimodal learning. Our future work will explore extensions of MFM for video generation, semi-supervised learning, and unsupervised learning. We believe that MFM sheds light on the advantages of learning factorizing multimodal representations and potentially opens up new horizons for multimodal machine learning.

heading: ACKNOWLEDGEMENTS

text: This work was supported in part by the DARPA grants D17AP00001 and FA875018C0150, Office of Naval Research, Apple, and Google focused award. We would also like to acknowledge NVIDIA’s GPU support. This material is also based upon work partially supported by the National Science Foundation (Award 1750439). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of National Science Foundation, and no official endorsement should be inferred.

heading: A PROOF OF PROPOSITION 1

text:
To simplify the proof, we first prove it for the unimodal case by considering the Wasserstein distance between PX,Y and PX̂,Ŷ.
A.1 UNIMODAL JOINT-DISTRIBUTION WASSERSTEIN DISTANCE
Proposition 2. For any functionsGy ∶ Zy → Fy, Ga ∶ Za → Fa, D ∶ Fy → Ŷ, and F ∶ Fa,Fy → X̂, we have
Wc(PX,Y, PX̂,Ŷ) = infQZ=PZ EPX,YEQ(Z∣X) ⎡⎢⎢⎢⎢⎣ cX(X, F (Ga(Za),Gy(Zy))) + cY (Y,D(Gy(Zy))) ⎤⎥⎥⎥⎥⎦ ,
(6)
where Wc is the Wasserstein distance under cost function cX and cY , PZ is the prior over Z = [Za,Zy] and QZ is the aggregated posterior of the proposed inference distribution Q(Z∣X).
Proof: See the following.
To begin the proof, we abuse some notations as follows.
By definition, the Wasserstein distance under cost function c between PX,Y and PX̂,Ŷ is
Wc(PX,Y, PX̂,Ŷ) ∶= inf Γ∈P((X,Y)∼PX,Y,(X̂,Ŷ)∼PX̂,Ŷ) E((X,Y),(X̂,Ŷ))∼Γ[c((X,Y), (X̂, Ŷ))], (7)
where c((X,Y), (X̂, Ŷ)) ∶ (X ,Y) × (X ,Y) →R+ is any measurable cost function. P((X,Y) ∼
PX,Y, (X̂, Ŷ) ∼ PX̂,Ŷ) is the set of all joint distributions of ((X,Y), (X̂, Ŷ)) with marginals
PX,Y and PX̂,Ŷ, respectively. Note that c((X,Y), (X̂, Ŷ)) = cX(X, X̂) + cY (Y, Ŷ).
Next, we denote the set of all joint distributions of (X,Y, X̂, Ŷ,Z) such that (X,Y) ∼ PX,Y, (X̂, Ŷ,Z) ∼ PX̂,Ŷ,Z, and ((X,Y) ⊥ (X̂, Ŷ)∣Z) as PX,Y,X̂,Ŷ,Z. PX,Y,X̂,Ŷ and PX,Y,Z are the sets of the marginals (X,Y, X̂, Ŷ) and (X,Y,Z) induced by PX,Y,X̂,Ŷ,Z.
We now introduce two Lemmas to help the proof.
Lemma 1. P (X̂, Ŷ∣Z = z) are Dirac for all z ∈ Z .
Proof: First, we have X̂ = F (Ga(Za),Gy(Zy)) and Ŷ = D(Gy(Zy)) with Z = {Za,Zy}. Since the functions F,Ga,Gy,D are all deterministic, then P (X̂, Ŷ∣Z) are Dirac measures. ◻
Lemma 2. P(PX,Y, PX̂,Ŷ) = PX,Y,X̂,Ŷ when P (X̂, Ŷ∣Z = z) are Dirac for all z ∈ Z .
Proof: When X̂, Ŷ are deterministic functions of Z, for any A in the sigma-algebra induced by X̂, Ŷ, we have
E[I[X̂,Ŷ∈A]∣X,Y,Z] = E[I[X̂,Ŷ∈A]∣Z].
Therefore, this implies that (X,Y) ⊥ (X̂, Ŷ)∣Z which concludes the proof. A similar argument is made in Lemma 1 of (Tolstikhin et al., 2017).
◻
Now, we use the fact that P(PX,Y, PX̂,Ŷ) = PX,Y,X̂,Ŷ (Lemma 1 + Lemma 2),
c((X,Y), (X̂, Ŷ)) = cX(X, X̂) + cY (Y, Ŷ), X̂ = F (Ga(Za),Gy(Zy)), and Ŷ = D(Gy(Zy)), Eq. equation 7 becomes
inf P ∈PX,Y,X̂,Ŷ EX,Y,X̂,Ŷ∼P [cX(X, X̂) + cY (Y, Ŷ)]
= inf P ∈PX,Y,X̂,Ŷ,Z EX,Y,X̂,Ŷ,Z∼P [cX(X, X̂) + cY (Y, Ŷ)]
= inf P ∈PX,Y,X̂,Ŷ,Z EPZEP (X,Y∣Z)EP (X̂,Ŷ∣Z)[cX(X, X̂) + cY (Y, Ŷ)]
= inf P ∈PX,Y,X̂,Ŷ,Z EPZEP (X,Y∣Z)[cX(X, F (Ga(Za),Gy(Zy))) + cY (Y,D(Gy(Zy)))]
= inf P ∈PX,Y,Z EPZEP (X,Y∣Z)[cX(X, F (Ga(Za),Gy(Zy))) + cY (Y,D(Gy(Zy)))]
= inf P ∈PX,Y,Z EX,Y,Z∼P [cX(X, F (Ga(Za),Gy(Zy))) + cY (Y,D(Gy(Zy)))].
(8)
Note that in Eq. equation 8, PX,Y,Z = P((X,Y) ∼ PX,Y,Z ∼ PZ) and with a proposed Q(Z∣X), we can rewrite Eq. equation 8 as
inf P ∈PX,Y,Z
EPX,YEPZ[cX(X, F (Ga(Za),Gy(Zy))) + cY (Y,D(Gy(Zy)))]
= inf QZ=PZ EPX,YEQ(Z∣X) ⎡⎢⎢⎢⎢⎣ cX(X, F (Ga(Za),Gy(Zy))) + cY (Y,D(Gy(Zy))) ⎤⎥⎥⎥⎥⎦ .
(9)
∎
A.2 FROM UNIMODAL TO MULTIMODAL
The proof is similar to Proposition 2, and we present a sketch to it. We can first show P (X̂1∶M , Ŷ∣Z = z) are Dirac for all z ∈ Z . Then we use the fact that c((X1∶M ,Y), (X̂1∶M , Ŷ)) =
∑Mi=1 cXi(Xi, X̂i) + cY (Y, Ŷ). Finally, we follow the tower rule of expectation and the conditional independence property similar to the proof in Proposition 2 and this concludes the proof.
∎

heading: B FULL BASELINE MODELS & RESULTS

text:
For a detailed description of the baselines, we point the reader to MFN (Zadeh et al., 2018a), MARN (Zadeh et al., 2018b), TFN (Zadeh et al., 2017), BC-LSTM (Poria et al., 2017), MVLSTM (Rajagopalan et al., 2016), EF-LSTM (Hochreiter & Schmidhuber, 1997; Graves et al., 2013; Schuster & Paliwal, 1997), DF (Nojavanasghari et al., 2016), MV-HCRF (Song et al., 2012; 2013), EF-HCRF (Quattoni et al., 2007; Morency et al., 2007), THMM (Morency et al., 2011), SVM-MD (Zadeh et al., 2016) and RF (Breiman, 2001).
We use the following extra notations for full descriptions of the baseline models described in Section 3.2, paragraph 3:
Variants of EF-LSTM: EF-LSTM (Early Fusion LSTM) uses a single LSTM (Hochreiter & Schmidhuber, 1997) on concatenated multimodal inputs. We also implement the EF-SLSTM (stacked) (Graves et al., 2013), EF-BLSTM (bidirectional) (Schuster & Paliwal, 1997) and EFSBLSTM (stacked bidirectional) versions.
Variants of EF-HCRF: EF-HCRF: (Hidden Conditional Random Field) (Quattoni et al., 2007) uses a HCRF to learn a set of latent variables conditioned on the concatenated input at each time step.
EF-LDHCRF (Latent Discriminative HCRFs) (Morency et al., 2007) are a class of models that learn hidden states in a CRF using a latent code between observed concatenated input and hidden output. EF-HSSHCRF: (Hierarchical Sequence Summarization HCRF) (Song et al., 2013) is a layered model that uses HCRFs with latent variables to learn hidden spatio-temporal dynamics.
Variants of MV-HCRF: MV-HCRF: Multi-view HCRF (Song et al., 2012) is an extension of the HCRF for Multi-view data, explicitly capturing view-shared and view specific sub-structures. MVLDHCRF: (Morency et al., 2007) is a variation of the MV-HCRF model that uses LDHCRF instead of HCRF. MV-HSSHCRF: (Song et al., 2013) further extends EF-HSSHCRF by performing Multiview hierarchical sequence summary representation.
In the following, we provide the full results for all baselines models described in Section 3.2, paragraph 3. Table 3 contains results for multimodal speaker traits recognition on the POM dataset. Table 4 contains results for the multimodal sentiment analysis on the CMU-MOSI, ICT-MMMO, YouTube, and MOUD datasets. Table 5 contains results for multimodal emotion recognition on the IEMOCAP dataset. MFM consistently achieves state-of-the-art or competitive results for all six multimodal datasets. We believe that by our MFM design, the multimodal discriminative factor Fy has successfully learned more meaningful representations by distilling discriminative features. This highlights the benefit of learning factorized multimodal representations towards discriminative tasks.

heading: C MULTIMODAL FEATURES

text:
For each of the multimodal time series datasets as mentioned in Section 3.2, paragraph 3, we extracted the following multimodal features: Language: We use pre-trained word embeddings (glove.840B.300d) (Pennington et al., 2014) to convert the video transcripts into a sequence of 300 dimensional word vectors. Visual: We use Facet (iMotions, 2017) to extract a set of features including per-frame basic and advanced emotions and facial action units as indicators of facial muscle movement (Ekman et al., 1980; Ekman, 1992). Acoustic: We use COVAREP (Degottex et al., 2014) to extract low level acoustic features including 12 Mel-frequency cepstral coefficients (MFCCs), pitch tracking and voiced/unvoiced segmenting features, glottal source parameters, peak slope parameters and maxima dispersion quotients. To reach the same time alignment between different modalities we choose the granularity of the input to be at the level of words. The words are aligned with audio using P2FA (Yuan & Liberman, 2008) to get their exact utterance times. We use expected feature values across the entire word for visual and acoustic features since they are extracted at a higher frequencies.
We make a note that the features for some of these datasets are constantly being updated. The authors of Zadeh et al. (2018a) notified us of a discrepancy in the sampling rate for acoustic feature extraction in the ICT-MMMO, YouTube and MOUD datasets which led to inaccurate word-level alignment between the three modalities. They publicly released the updated multimodal features. We performed all experiments on the latest versions of these datasets which can be accessed from https: //github.com/A2Zadeh/CMU-MultimodalSDK. All baseline models were retrained with extensive hyperparameter search for fair comparison.
D INFORMATION AND GRADIENT-BASED INTERPRETATION
Information-Based Interpretation: We choose the normalized Hilbert-Schmidt Independence Criterion (Gretton et al., 2005; Wu et al., 2018) as the approximation (see Sugiyama & Yamada (2012); Wu et al. (2018)) of our MI measure:
MI(F⋅, X̂i) = HSICnorm(F⋅, X̂i) = tr(KF⋅HKX̂iH)
∥HKF⋅H∥F ∥HKX̂iH∥F , (10)
where ⋅ represents y or ai, n is the number of {F⋅, X̂i} pairs, H = I − 1n11 ⊺, KF⋅ ∈ Rn×n is the Gram matrix of F⋅ with KF⋅ ij = k1(F⋅i,F⋅j), KX̂i ∈ R n×n is the Gram matrix of X̂i with KX̂ijk = k2(X̂ij , X̂ik). k1(⋅, ⋅) and k2(⋅, ⋅) are predefined kernel functions.
The most common choice for the kernel is the RBF kernel. However, if we consider time series data with various time steps, we need to either perform data augmentation or choose another kernel choice. For example, we can adopt the Global Alignment Kernel (Cuturi et al., 2007) which considers the

heading: DF 0.240 0.273 0.017 0.139 0.112 0.173 0.118 0.217 0.148 0.143 0.019 0.093 0.041 0.136 0.168 0.259

text:
EF-LSTM 0.200 0.302 0.031 0.079 0.170 0.244 0.265 0.240 0.142 0.062 0.083 0.152 0.260 0.105 0.217 0.227 EF-SLSTM 0.221 0.327 0.042 0.151 0.177 0.239 0.268 0.248 0.204 0.069 0.092 0.215 0.252 0.159 0.218 0.196 EF-BLSTM 0.162 0.289 -0.034 0.135 0.191 0.279 0.274 0.231 0.184 0.154 0.093 0.147 0.245 0.166 0.243 0.272 EF-SBLSTM 0.174 0.310 0.021 0.088 0.170 0.224 0.261 0.241 0.155 0.163 0.097 0.120 0.215 0.121 0.216 0.171 MV-LSTM 0.358 0.416 0.131 0.146 0.280 0.347 0.323 0.326 0.295 0.237 0.119 0.238 0.284 0.258 0.239 0.317 BC-LSTM 0.359 0.425 0.081 0.234 0.358 0.417 0.450 0.361 0.293 0.109 0.075 0.078 0.363 0.184 0.344 0.319 TFN 0.089 0.201 0.030 0.020 0.124 0.204 0.171 0.223 -0.051 -0.064 0.114 0.060 0.048 -0.002 0.106 0.213 MARN 0.340 0.410 0.166 0.235 0.340 0.374 0.406 0.378 0.282 0.147 0.215 0.204 0.348 0.235 0.303 0.287 MFN 0.395 0.428 0.193 0.313 0.367 0.431 0.452 0.395 0.333 0.296 0.255 0.259 0.381 0.318 0.377 0.386 MFM 0.431 0.450 0.197 0.411 0.380 0.448 0.467 0.452 0.368 0.212 0.309 0.333 0.404 0.333 0.334 0.408 ∆SOTA ↑ 0.036 ↑ 0.022 ↑ 0.004 ↑ 0.097 ↑ 0.013 ↑ 0.017 ↑ 0.015 ↑ 0.057 ↑ 0.035 – ↑ 0.054 ↑ 0.074 ↑ 0.023 ↑ 0.015 – ↑ 0.022
alignment between two varying-length time series when computing the kernel score between them. To simplify our analysis, we choose to augment data before we calculate the kernel score with the RBF kernel. More specifically, we perform averaging over time series data:
Xaug = 1
n
T
∑ t=1 Xt with X = [X1,X2,⋯,XT ]. (11)
The bandwidth of the RBF kernel is set as 1.0 throughout the experiments.
Here, we provide an additional interpretation result for the POM dataset in Table 6. We observe that the language modality is also the most informative while the visual and acoustic modalities are almost equally informative. This result is in agreement with behavioral studies which have observed that non-verbal behaviors are particularly informative of personality traits (Guimond & Massrieh, 2012; Levine et al., 2009; Mohammadi et al., 2010). For example, the same sentence “this movie was great” can convey significantly different messages on speaker confidence depending on whether it was said in a loud and exciting voice, with eye contact, or powerful gesticulation.
Gradient-Based Interpretation: MFM reconstructs xi as follows: x̂i = Fi(fai, fy), fai = Gai(zai), fy = Gy(zy), zai ∼ Q(Zai∣Xi = xi), zy ∼ Q(Zy∣X1∶M = x1∶M). (12)
Equation equation 12 also explains how we obtain fy ∼ P (Fy∣X1∶M = x1∶M). The gradient flow through time is defined as:
∇fy(x̂i) ∶=[∥∇fy x̂1i ∥2F , ∥∇fy x̂2i ∥2F ,⋯, ∥∇fy x̂Ti ∥2F ]. (13)

heading: E ENCODER AND DECODER DESIGN FOR MULTIMODAL SYNTHETIC IMAGE DATASET

text: For experiments on the multimodal synthetic image dataset, we use convolutional+fully-connected layers for the encoder and deconvolutional+fully-connected layers for the decoder (Zeiler et al., 2010). Different convolutional layers are each applied on the input SVHN and MNIST images to learn modality-specific generative factors. Next, we concatenate the features from two more convolutional layers on SVHN and MNIST to learn the multimodal-discriminative factor. The multimodal discriminative factor is passed through fully-connected layers to predict the label. For generation, we concatenate the multimodal discriminative factors and the modality-specific generative factor together and use a deconvolutional layer to generate digits.

heading: F ENCODER AND DECODER DESIGN FOR MULTIMODAL TIME SERIES DATASETS

text: Figure 5 illustrates how MFM operates on multimodal time series data. The encoder Q(Zy∣X1∶M) can be parametrized by any model that performs multimodal fusion (Nojavanasghari et al., 2016; Zadeh et al., 2018a). We choose the Memory Fusion Network (MFN) (Zadeh et al., 2018a) as our encoder Q(Zy∣X1∶M). We use encoder LSTM networks and decoder LSTM networks (Cho et al., 2014) to parametrize functions Q(Za1∶M ∣X1∶M) and F1∶M respectively, and FCNNs to parametrize functions Gy , Ga{1∶M} and D.

heading: G SURROGATE INFERENCE GRAPHICAL MODEL

text: We illustrate the surrogate inference for addressing the missing modalities issue in Figure 6. The surrogate inference model infers the latent codes given the present modalities. These inferred latent codes can then be used for reconstructing the missing modalities or label prediction in the presence of missing modalities.

heading: H COMPARISON WITH HSU & GLASS (2018)

text:
A similar approach for factorizing the latent factors was recently proposed by Hsu & Glass (2018) in work that was performed independently and in parallel. In comparison with MFM, there are several major differences that can be categorized into the (1) prior matching discrepancy, (2) inference network, (3) discriminative objective, (4) multimodal fusion, (5) scale of experiments.
1. MFM usesMMD(QZ, PZ) (see Equation 4) as the prior matching discrepancy while Hsu & Glass (2018) use KL(QZ∣X, PZ)) (see Section 2.2.1 in Hsu & Glass (2018)).
2. MFM considers multimodal and unimodal inference in a single network (see Figure 1(b)), while Hsu & Glass (2018) considers separate networks (see Figure 1 in Hsu & Glass (2018)). They further propose to match the coherence between these two networks using an additional loss term (see Equation 7 in Hsu & Glass (2018)).
3. MFM learns to predict the labels using a generative framework (see Figure 1(a)), while Hsu & Glass (2018) use an additional hinge loss to separate the latent factors from different labels (see Equation 9 in Hsu & Glass (2018)).
4. MFM is a flexible framework that can be combined with any multimodal fusion encoder (see Section 2.4), while Hsu & Glass (2018) considers a fixed multimodal encoder (similar to early fusion) (see Section 4.1 in Hsu & Glass (2018)).
5. We evaluate the performance of MFM over a much larger scale of datasets. We perform experiments on six multimodal time-series datasets that take on the form of videos with the language, visual, and acoustic modalities. These datasets span three core research areas of multimodal personality traits recognition, multimodal sentiment analysis, and multimodal emotion recognition. On the other hand, Hsu & Glass (2018) evaluates their model on a spoken digit dataset which randomly combines a digit image with a spoken digit (see Section 4 in Hsu & Glass (2018)). MFM further considers experiments to evaluate reconstruction and prediction in the presence of missing modalities (see Section 3.2) which Hsu & Glass (2018) do not. Lastly, we compares to over 20 baseline models in our experiments (see Section 3.2) and explore the choice of various multimodal encoders in MFM. Hsu & Glass (2018) only compares to the JMVAE baseline model (Suzuki et al., 2016a) which resembles the MD model in our ablation study (see Section 4.4 in Hsu & Glass (2018)).
In Table 7, we provide a comparison on the CMU-MOSI, ICT-MMMO, YouTube and MOUD datasets to test the disentanglement and prediction performance for the model described in Hsu & Glass (2018). These experimental results show that across these datasets and metrics, MFM performs better than the model proposed in Hsu & Glass (2018). We would like to highlight that at the time of submission, the code for (Hsu & Glass, 2018) had not been made public and we reimplemented their model to experiment on our datasets.
I COMPARISON WITH β-VAE
Although β-VAE (Higgins et al., 2017) was designed to handle unimodal data, we provide an extension to multimodal data. To achieve this, we set the choice of prior matching discrepancy as the KL-divergence KL(QZ∣X, PZ) and set β large (i.e. β ∈ {10,50,100,200}) to encourage disentanglement of latent variables. We train a β-VAE to model multimodal data using the same
factorization as proposed in our model (i.e. modality-specific generative factors Za{1∶M} and a multimodal discriminative factor Zy). To provide a fair comparison to our discriminative model, we fine tune by training a classifier on top of the multimodal discriminative factor Zy to the label Y. We provide experimental results in Table 8 on the CMU-MOSI, ICT-MMMO, YouTube and MOUD datasets. MFM outperforms β-VAE across these datasets and metrics.

title: LEARNING FACTORIZED MULTIMODAL REPRESENTATIONS