DEEP GENERALIZED CANONICAL CORRELATION ANALYSIS
We present Deep Generalized Canonical Correlation Analysis (DGCCA) – a method for learning nonlinear transformations of arbitrarily many views of data, such that the resulting transformations are maximally informative of each other. While methods for nonlinear two-view representation learning (Deep CCA, (Andrew et al., 2013)) and linear many-view representation learning (Generalized CCA (Horst, 1961)) exist, DGCCA is the first CCA-style multiview representation learning technique that combines the flexibility of nonlinear (deep) representation learning with the statistical power of incorporating information from many independent sources, or views. We present the DGCCA formulation as well as an efficient stochastic optimization algorithm for solving it. We learn DGCCA representations on two distinct datasets for three downstream tasks: phonetic transcription from acoustic and articulatory measurements, and recommending hashtags and friends on a dataset of Twitter users. We find that DGCCA representations soundly beat existing methods at phonetic transcription and hashtag recommendation, and in general perform no worse than standard linear many-view techniques.
We present Deep Generalized Canonical Correlation Analysis (DGCCA) – a method for learning nonlinear transformations of arbitrarily many views of data, such that the resulting transformations are maximally informative of each other. While methods for nonlinear two-view representation learning (Deep CCA, (Andrew et al., 2013)) and linear many-view representation learning (Generalized CCA (Horst, 1961)) exist, DGCCA is the first CCA-style multiview representation learning technique that combines the flexibility of nonlinear (deep) representation learning with the statistical power of incorporating information from many independent sources, or views. We present the DGCCA formulation as well as an efficient stochastic optimization algorithm for solving it. We learn DGCCA representations on two distinct datasets for three downstream tasks: phonetic transcription from acoustic and articulatory measurements, and recommending hashtags and friends on a dataset of Twitter users. We find that DGCCA representations soundly beat existing methods at phonetic transcription and hashtag recommendation, and in general perform no worse than standard linear many-view techniques.
1 INTRODUCTION
Multiview representation learning refers to settings where one has access to many “views” of data, at train time. Views often correspond to different modalities or independent information about examples: a scene represented as a series of audio and image frames, a social media user characterized by the messages they post and who they friend, or a speech utterance and the configuration of the speaker’s tongue. Multiview techniques learn a representation of data that captures the sources of variation common to all views.
Multiview representation techniques are attractive for intuitive reasons. A representation that is able to explain many views of the data is more likely to capture meaningful variation than a representation that is a good fit for only one of the views. They are also attractive for the theoretical reasons. For example, Anandkumar et al. (2014) show that certain classes of latent variable models, such as Hidden Markov Models, Gaussian Mixture Models, and Latent Dirichlet Allocation models, can be optimally learned with multiview spectral techniques. Representations learned from many views will generalize better than one, since the learned representations are forced to accurately capture variation in all views at the same time (Sridharan & Kakade, 2008) – each view acts as a regularizer, constraining the possible representations that can be learned. These methods are often based on canonical correlation analysis (CCA), a classical statisical technique proposed by Hotelling (1936).
In spite of encouraging theoretical guarantees, multiview learning techniques cannot freely model nonlinear relationships between arbitrarily many views. Either they are able to model variation across many views, but can only learn linear mappings to the shared space (Horst, 1961), or they simply cannot be applied to data with more than two views using existing techniques based on kernel CCA (Hardoon et al., 2004) and deep CCA (Andrew et al., 2013).
Here we present Deep Generalized Canonical Correlation Analysis (DGCCA). Unlike previous correlation-based multiview techniques, DGCCA learns a shared representation from data with arbitrarily many views and simultaneously learns nonlinear mappings from each view to this shared space. The only (mild) constraint is that these nonlinear mappings from views to shared space must be differentiable. Our main methodological contribution is the derivation of the gradient update for the Generalized Canonical Correlation Analysis (GCCA) objective (Horst, 1961). As a practical contribution, we have also released an implementation of DGCCA1.
We also evaluate DGCCA-learned representations on two distinct datasets and three downstream tasks: phonetic transcription from aligned speech and articulatory data, and Twitter hashtag and friend recommendation from six text and network feature views. We find that downstream performance of DGCCA representations is ultimately task-dependent. However, we find clear gains in performance from DGCCA for tasks previously shown to benefit from representation learning on more than two views, with up to 4% improvement in heldout accuracy for phonetic transcription.
The paper is organized as follows. We review prior work in Section 2. In Section 3 we describe DGCCA. Empirical results on a synthetic dataset, and three downstream tasks are presented in Section 4. In Section 5, we describe the differences between DGCCA and other non-CCA-based multiview learning work and conclude with future directions in Section 6.
2 PRIOR WORK
Some of most successful techniques for multiview representation learning are based on canonical correlation analysis (Wang et al., 2015a;b) and its extension to the nonlinear and many view settings, which we describe in this section. For other related multiview learning techniques, see Section 5.
2.1 CANONICAL CORRELATION ANALYSIS (CCA)
Canonical correlation analysis (CCA) (Hotelling, 1936) is a statistical method that finds maximally correlated linear projections of two random vectors and is a fundamental multiview learning technique. Given two input views, X1 ∈ Rd1 and X2 ∈ Rd2 , with covariance matrices, Σ11 and Σ22, respectively, and cross-covariance matrix, Σ12, CCA finds directions that maximize the correlation between them:
(u∗1, u ∗ 2) = argmax u1∈Rd1 ,u2∈Rd2 corr(u>1 X1, u > 2 X2) = argmax u1∈Rd1 ,u2∈Rd2 u>1 Σ12u2√ u>1 Σ11u1u > 2 Σ22u2
Since this formulation is invariant to affine transformations of u1 and u2, we can write it as the following constrained optimization formulation:
(u∗1, u ∗ 2) = argmax
u>1 Σ11u1=u > 2 Σ22u2=1
u>1 Σ12u2 (1)
This technique has two limitations that have led to significant extensions: First, it is limited to learning representations that are linear transformations of the data in each view, and second, it can only leverage two input views.
2.2 DEEP CANONICAL CORRELATION ANALYSIS (DCCA)
Deep CCA (DCCA) (Andrew et al., 2013) is an extension of CCA that addresses the first limitation by finding maximally linearly correlated non-linear transformations of two vectors. It does this by passing each of the input views through stacked non-linear representations and performing CCA on the outputs.
Let us use f1(X1) and f2(X2) to represent the network outputs. The weights, W1 and W2, of these networks are trained through standard backpropagation to maximize the CCA objective:
(u∗1, u ∗ 2,W ∗ 1 ,W ∗ 2 ) = argmax
u1,u2
corr(u>1 f1(X1), u > 2 f2(X2))
DCCA is still limited to only 2 input views. 1See https://bitbucket.org/adrianbenton/dgcca-py3 for implementation of DGCCA along with data from the synthetic experiments.
2.3 GENERALIZED CANONICAL CORRELATION ANALYSIS (GCCA)
Another extension of CCA, which addresses the limitation on the number of views, is Generalized CCA (GCCA) (Horst, 1961). It corresponds to solving the optimization problem in Equation (2), of finding a shared representation G of J different views, where N is the number of data points, dj is the dimensionality of the jth view, r is the dimensionality of the learned representation, and Xj ∈ Rdj×N is the data matrix for the jth view.2
minimize Uj∈Rdj×r,G∈Rr×N J∑ j=1 ‖G− U>j Xj‖2F
subject to GG> = Ir
(2)
Solving GCCA requires finding an eigendecomposition of an N ×N matrix, which scales quadratically with sample size and leads to memory constraints. Unlike CCA and DCCA, which only learn projections or transformations on each of the views, GCCA also learns a view-independent representation G that best reconstructs all of the view-specific representations simultaneously. The key limitation of GCCA is that it can only learn linear transformations of each view.
3 DEEP GENERALIZED CANONICAL CORRELATION ANALYSIS (DGCCA)
In this section, we present deep GCCA (DGCCA): a multiview representation learning technique that benefits from the expressive power of deep neural networks and can also leverage statistical strength from more than two views in data, unlike Deep CCA which is limited to only two views. More fundamentally, deep CCA and deep GCCA have very different objectives and optimization problems, and it is not immediately clear how to extend deep CCA to more than two views.
DGCCA learns a nonlinear map for each view in order to maximize the correlation between the learnt representations across views. In training, DGCCA passes the input vectors in each view through multiple layers of nonlinear transformations and backpropagates the gradient of the GCCA objective with respect to network parameters to tune each view’s network, as illustrated in Figure 1. The objective is to train networks that reduce the GCCA reconstruction error among their outputs. At test time, new data can be projected by feeding them through the learned network for each view.
We now formally define the DGCCA problem. We consider J views in our data, and let Xj ∈ Rdj×N denote the jth input matrix.3 The network for the jth view consists of Kj layers. Assume, for simplicity, that each layer in the jth view network has cj units with a final (output) layer of size oj . The output of the kth layer for the jth view is h j k = s(W j kh j k−1), where s : R → R is a nonlinear activation function and W jk ∈ Rck×ck−1 is the weight matrix for the kth layer of the jth view network. We denote the output of the final layer as fj(Xj).
2Horst’s original presentation of GCCA is in a very different form from the one used here but has been shown to be equivalent (Kettenring, 1971).
3Our notation for this section closely follows that of Andrew et al. (2013)
DGCCA can be expressed as the following optimization problem: find weight matrices W j = {W j1 , . . . ,W j Kj } defining the functions fj , and linear transformations Uj (of the output of the jth network), for j = 1, . . . , J , that minimize
Uj∈Roj×r,G∈Rr×N J∑ j=1 ‖G− U>j fj(Xj)‖2F
subject to GG> = Ir
, (3)
where G ∈ Rr×N is the shared representation we are interested in learning.
Optimization: We solve the DGCCA optimization problem using stochastic gradient descent (SGD) with mini-batches. In particular, we estimate the gradient of the DGCCA objective in Problem 3 on a mini-batch of samples that is mapped through the network and use back-propagation to update the weight matrices, W j’s. However, note that the DGCCA optimization problem is a constrained optimization problem. It is not immediately clear how to perform projected gradient descent with back-propagation. Instead, we characterize the objective function of the GCCA problem at an optimum, and compute its gradient with respect to the inputs to GCCA, i.e. with respect to the network outputs. These gradients are then back-propagated through the network to update W j’s.
Although the relationship between DGCCA and GCCA is analogous to the relationship between DCCA and CCA, derivation of the GCCA objective gradient with respect to the network output layers is non-trivial. The main difficulty stems from the fact that there is no natural extension of the correlation objective to more than two random variables. Instead, we consider correlations between every pair of views, stack them in a J × J matrix and maximize a certain matrix norm for that matrix. For GCCA, this suggests an optimization problem that maximizes the sum of correlations between a shared representation and each view. Since the objective as well as the constraints of the generalized CCA problem are very different from that of the CCA problem, it is not immediately obvious how to extend Deep CCA to Deep GCCA.
Next, we show a sketch of the gradient derivation, the full derivation is given in appendix A. It is straightforward to show that the solution to the GCCA problem is given by solving an eigenvalue problem. In particular, defineCjj = f(Xj)f(Xj)> ∈ Roj×oj , to be the scaled empirical covariance matrix of the jth network output, and Pj = f(Xj)>C−1jj f(Xj) ∈ RN×N be the corresponding projection matrix that whitens the data; note that Pj is symmetric and idempotent. We define M =∑J
j=1 Pj . Since each Pj is positive semi-definite, so is M . Then, it is easy to check that the rows of G are the top r (orthonormal) eigenvectors of M , and Uj = C−1jj f(Xj)G
>. Thus, at the minimum of the objective, we can rewrite the reconstruction error as follows:
J∑ j=1 ‖G− U>j fj(Xj)‖2F = J∑ j=1 ‖G−Gfj(Xj)>C−1jj fj(Xj)‖ 2 F = rJ − Tr(GMG>)
Minimizing the GCCA objective (w.r.t. the weights of the neural networks) means maximizing Tr(GMG>), which is the sum of eigenvalues L = ∑r i=1 λi(M). Taking the derivative of L with respect to each output layer fj(Xj) we have: ∂L
∂fj(Xj) = 2UjG− 2UjU>j fj(Xj)
Thus, the gradient is the difference between the r-dimensional auxiliary representation G embedded into the subspace spanned by the columns of Uj (the first term) and the projection of the actual data in fj(Xj) onto the said subspace (the second term). Intuitively, if the auxiliary representation G is far away from the view-specific representation U>j fj(Xj), then the network weights should receive a large update. Computing the gradient descent update has time complexity O(JNrd), where d = max(d1, d2, . . . , dJ) is the largest dimensionality of the input views.
4 EXPERIMENTS
4.1 SYNTHETIC MULTIVIEW MIXTURE MODEL
In this section, we apply DGCCA to a small synthetic data set to show how it preserves the generative structure of data sampled from a multiview mixture model. The data we use for this experiment are
plotted in Figure 2. Points that share the same color across different views are sampled from the same mixture component.
Importantly, in each view, there is no linear transformation of the data that separates the two mixture components, in the sense that the generative structure of the data could not be exploited by a linear model. This point is reinforced by Figure 3(a), which shows the two-dimensional representation G learned by applying (linear) GCCA to the data in Figure 2. The learned representation completely loses the structure of the data.
We can contrast the failure of GCCA to preserve structure with the result of applying DGCCA; in this case, the input neural networks had three hidden layers with ten units each with weights randomly initialized. We plot the representation G learned by DGCCA in Figure 3 (b). In this representation, the mixture components are easily separated by a linear classifier; in fact, the structure is largely preserved even after projection onto the first coordinate of G.
It is also illustrative to consider the view-specific representations learned by DGCCA, that is, to consider the outputs of the neural networks that were trained to maximize the GCCA objective. We plot the representations in Figure 4. For each view, we have learned a nonlinear mapping that does remarkably well at making the mixture components linearly separable. Recall that absolutely no direct supervision was given about which mixture component each point was generated from. The only training signals available to the networks were the reconstruction errors between the network outputs and the learned representation G.
4.2 PHONEME CLASSIFICATION
In this section, we discuss experiments on the University of Wisconsin X-ray Microbeam Database (XRMB) (Westbury, 1994). XRMB contains acoustic and articulatory recordings as well as phonemic labels. We present phoneme classification results on the acoustic vectors projected using DCCA,
GCCA, and DGCCA. We set acoustic and articulatory data as the two views and phoneme labels as the third view for GCCA and DGCCA. For classification, we run K-nearest neighbor classification (Cover & Hart, 1967) on the projected result.
4.2.1 DATA
We use the same train/tune/test split of the data as Arora & Livescu (2014). To limit experiment runtime, we use a subset of speakers for our experiments. We run a set of cross-speaker experiments using the male speaker JW11 for training and two splits of JW24 for tuning and testing. We also perform parameter tuning for the third view with 5-fold cross validation using a single speaker, JW11. For both experiments, we use acoustic and articulatory measurements as the two views in DCCA. Following the pre-processing in Andrew et al. (2013), we get 273 and 112 dimensional feature vectors for the first and second view respectively. Each speaker has ∼50,000 frames. For the third view in GCCA and DGCCA, we use 39-dimensional one-hot vectors corresponding to the labels for each frame, following Arora & Livescu (2014).
4.2.2 PARAMETERS
We use a fixed network size and regularization for the first two views, each containing three hidden layers with sigmoid activation functions. Hidden layers for the acoustic view were all width 1024, and layers in the articulatory view all had width 512 units. L2 penalty constants of 0.0001 and 0.01 were used to train the acoustic and articulatory view networks, respectively. The output layer dimension of each network is set to 30 for DCCA and DGCCA. For the 5-fold speaker-dependent experiments, we performed a grid search for the network sizes in {128, 256, 512, 1024} and covariance matrix regularization in {10−2, 10−4, 10−6, 10−8} for the third view in each fold. We fix the hyperparameters for these experiments optimizing the networks with minibatch stochastic gradient descent with a step size of 0.005, batch size of 2000, and no learning decay or momentum. The third view neural network had an L2 penalty of 0.0005.
4.2.3 RESULTS
As we show in Table 1, DGCCA improves upon both the linear multiview GCCA and the non-linear 2-view DCCA for both the cross-speaker and speaker-dependent cross-validated tasks.
In addition to accuracy, we examine the reconstruction error, i.e. the objective in Equation 3, obtained from the objective in GCCA and DGCCA.4 This sharp improvement in reconstruction error shows that a non-linear algorithm can better model the data.
In this experimental setup, DCCA under-performs the baseline of simply running KNN on the original acoustic view. Prior work considered the output of DCCA stacked on to the central frame of the original acoustic view (39 dimensions). This poor performance, in the absence of original features, indicates that it was not able to find a more informative projection than original acoustic features based on correlation with the articulatory view within the first 30 dimensions.
To highlight the improvements of DGCCA over GCCA, Figure 5 presents a subset of the the confusion matrices on speaker-dependent test data. In particular, we observe large improvements in the classification of D, F , K, SH , V and Y . GCCA outperforms DGCCA for UH and DH . These matrices also highlight the common misclassifications that DGCCA improves upon. For instance,
4For 2-view experiments, correlation is a common metric to compare performance. Since that metric is unavailable in a multiview setting, reconstruction error is the analogue.
DGCCA rectifies the frequent misclassification of V as P , R and B by GCCA. In addition, commonly incorrect classification of phonemes such as S and T is corrected by DGCCA, which enables better performance on other voiceless consonants such as like F , K and SH . Vowels are classified with almost equal accuracy by both the methods.
4.3 TWITTER USER HASHTAG & FRIEND RECOMMENDATION
Linear multiview techniques are effective at recommending hashtag and friends for Twitter users (Benton et al., 2016). In this experiment, six views of a Twitter user were constructed by applying principal component analysis (PCA) to the bag-of-words representations of (1) tweets posted by the ego user, (2) other mentioned users, (3) their friends, and (4) their followers, as well as one-hot encodings of the local (5) friend and (6) follower networks. We learn and evaluate DGCCA models on identical training, development, and test sets as Benton et al. (2016), and evaluate the DGCCA representations on macro precision at 1000 (P@1000) and recall at 1000 (R@1000) for the hashtag and friend recommendation tasks described there.
We trained 40 different DGCCA model architectures, each with identical architectures across views, where the width of the hidden and output layers, c1 and c2, for each view are drawn uniformly from [10, 1000], and the auxiliary representation width r is drawn uniformly from [10, c2]5. All networks used ReLUs as activation functions, and were optimized with Adam (Kingma & Ba, 2014) for 200 epochs6. Networks were trained on 90% of 102,328 Twitter users, with 10% of users used as a tuning set to estimate heldout reconstruction error for model selection. We report development and test results for the best performing model on the downstream task development set. Learning rate was set to 10−4 with an L1 and L2 regularization constants of 0.01 and 0.001 for all weights 7.
Table 2 displays the performance of DGCCA compared to PCA[text+net] (PCA applied to concatenation of view feature vectors), linear GCCA applied to the four text views, [text], and all
5We chose to restrict ourselves to a single hidden layer with non-linear activation and identical architectures for each view, so as to avoid a fishing expedition. If DGCCA is appropriate for learning Twitter user representations, then a good architecture should require little exploration.
6From preliminary experiments, we found that Adam pushed down reconstruction error more quickly than SGD with momentum, and that ReLUs were easier to optimize than sigmoid activations.
7This setting of regularization constants led to low reconstruction error in preliminary experiments.
views, [text+net], along with a weighted GCCA variant (WGCCA). We learned PCA, GCCA, and WGCCA representations of width r ∈ {10, 20, 50, 100, 200, 300, 400, 500, 750, 1000}, and report the best performing representations on the development set.
There are several points to note: First is that DGCCA outperforms linear methods at hashtag recommendation by a wide margin in terms of recall. This is exciting because this task was shown to benefit from incorporating more than just two views from Twitter users. These results suggest that a nonlinear transformation of the input views can yield additional gains in performance. In addition, WGCCA models sweep over every possible weighting of views with weights in {0, 0.25, 1.0}. WGCCA has a distinct advantage in that the model is allowed to discriminatively weight views to maximize downstream performance. The fact that DGCCA is able to outperform WGCCA at hashtag recommendation is encouraging, since WGCCA has much more freedom to discard uninformative views, whereas the DGCCA objective forces networks to minimize reconstruction error equally across all views. As noted in Benton et al. (2016), only the friend network view was useful for learning representations for friend recommendation (corroborated by performance of PCA applied to friend network view), so it is unsurprising that DGCCA when applied to all views cannot compete with WGCCA representations learned on the single useful friend network view8.
5 OTHER MULTIVIEW LEARNING WORK
There has been strong work outside of CCA-related methods to combine nonlinear representation and learning from multiple views. Kumar et al. (2011) elegantly outlines two main approaches these methods take to learn a joint representation from many views: either by 1) explicitly maximizing pairwise similarity/correlation between views or by 2) alternately optimizing a shared, “consensus” representation and view-specific transformations to maximize similarity. Models such as the siamese network proposed by Masci et al. (2014), fall in the former camp, minimizing the squared error between embeddings learned from each view, leading to a quadratic increase in the terms of the loss function size as the number of views increase. Rajendran et al. (2015) extend Correlational Neural Networks (Chandar et al., 2015) to many views and avoid this quadratic explosion in the loss function by only computing correlation between each view embedding and the embedding of a “pivot” view. Although this model may be appropriate for tasks such as multilingual image captioning, there are many datasets where there is no clear method of choosing a pivot view. The DGCCA objective does not suffer from this quadratic increase w.r.t. the number of views, nor does it require a privileged pivot view, since the shared representation is learned from the per-view representations.
Approaches that estimate a “consensus” representation, such as the multiview spectral clustering approach in Kumar et al. (2011), typically do so by an alternating optimization scheme which depends on a strong initialization to avoid bad local optima. The GCCA objective our work builds on is particularly attractive, since it admits a globally optimal solution for both the view-specific projections U1 . . . UJ , and the shared representation G by singular value decomposition of a single matrix: a sum of the per-view projection matrices. Local optima arise in the DGCCA objective only because we are also learning nonlinear transformations of the input views. Nonlinear multiview methods often avoid learning these nonlinear transformations by assuming that a kernel or graph Laplacian (e.g. in multiview clustering) is given (Kumar et al., 2011; Xiaowen, 2014; Sharma et al., 2012).
6 CONCLUSION
We present DGCCA, a method for non-linear multiview representation learning from an arbitrary number of views. We show that DGCCA clearly outperforms prior work when using labels as a third view (Andrew et al., 2013; Arora & Livescu, 2014; Wang et al., 2015c), and can successfully exploit multiple views to learn user representations useful for downstream tasks such as hashtag recommendation for Twitter users. To date, CCA-style multiview learning techniques were either restricted to learning representations from no more than two views, or strictly linear transformations of the input views. This work overcomes these limitations.
8The performance of WGCCA suffers compared to PCA because whitening the friend network data ignores the fact that the spectrum of the decays quickly with a long tail – the first few principal components made up a large portion of the variance in the data, but it was also important to compare users based on other components.
APPENDIX A DERIVING THE GCCA OBJECTIVE GRADIENT
In order to train the neural networks in DGCCA, we need to compute the gradient of the GCCA objective with respect to any one of its input views. This gradient can then be backpropagated through the input networks to derive updates for the network weights.
Let N be the number of data points and J the number of views. Let Yj ∈ Rc j K×N be the data matrix representing the output of the jth neural network, i.e. Yj = fj(Xj), where c j K is the number of neurons in the output layer of the jth network. Then, GCCA can be written as the following optimization problem, where r is the dimensionality of the learned auxiliary representation:
minimize Uj∈Rc j K ×r,G∈Rr×N J∑ j=1 ‖G− U>j Yj‖2F
subject to GG> = Ir It can be shown that the solution is found by solving a certain eigenvalue problem. In particular, define Cjj = YjY >j ∈ Rc j K×c j K , Pj = Y >j C −1 jj Yj (note that Pj is symmetric and idempotent), and
M = ∑J
j=1 Pj (since each Pj is psd, so is M ). Then the rows of G are the top r (orthonormal) eigenvectors of M , and Uj = C−1jj YjG
>. Thus, at the minima of the objective, we can rewrite the reconstruction error as follows:
J∑ j=1 ‖G− U>j Yj‖2F = J∑ j=1 ‖G−GY >j C−1jj Yj‖ 2 F
= J∑ j=1 ‖G(IN − Pj)‖2F
= J∑ j=1 Tr[G(IN − Pj)G>]
= J∑ j=1 Tr(Ir)− Tr(GMG>)
= Jr − Tr(GMG>) Note that we can write the rank-1 decomposition of M as ∑N
k=1 λkgkg > k . Furthermore, since the
kth row of G is gk, and since the matrix product Ggk = êk,
GMG> = N∑ k=1 λkGgk(Ggk) > = r∑ k=1 λkêkê > k
But this is just an N × N diagonal matrix containing the top r eigenvalues of M , so we can write the GCCA objective as
Jr − r∑
i=1
λi(M)
Thus, minimizing the GCCA objective (w.r.t. the weights of the neural nets) means maximizing the sum of eigenvalues ∑r i=1 λi(M), which we will henceforth denote by L.
Now, we will derive an expression for ∂L∂Yj for any view Yj . First, by the chain rule, and using the fact that ∂L∂M = G >G (Petersen & Pedersen, 2012),
∂L
∂(Yj)ab = N∑ c,d=1 ∂L ∂Mcd ∂Mcd ∂(Yj)ab
= N∑ c,d=1 (G>G)cd ∂Mcd ∂(Yj)ab
Since M = ∑J
j′=1 Pj′ , and since the only projection matrix that depends on Yj is Pj , ∂M ∂Yj = ∂Pj ∂Yj .
Since Pj = Y >j C −1 jj Yj ,
(Pj)cd = cjK∑ k,`=1 (Yj)kc(C −1 jj )k`(Yj)`d
Thus, by the product rule,
∂(Pj)cd ∂(Yj)ab = δcb cjK∑ `=1 (Yj)`d(C −1 jj )a` +
δdb cjK∑ k=1 (Yj)kc(C −1 jj )ka +
cjK∑ k,`=1 (Yj)kc(Yj)`d ∂(C−1jj )k` ∂(Yj)ab
= δcb(C −1 jj Yj)ad + δdb(C −1 jj Yj)ac
+ cjK∑ k,`=1 (Yj)kc(Yj)`d ∂(C−1jj )k` ∂(Yj)ab
The derivative in the last term can also be computed using the chain rule:
∂(C−1jj )k`
∂(Yj)ab = N∑ m,n=1 ∂(C−1jj )k` ∂(Cjj)mn ∂(Cjj)mn ∂(Yj)ab
= − N∑
m,n=1
{ (C−1jj )km(C −1 jj )n`
[δam(Yj)nb + δan(Yj)mb] }
= − N∑
n=1
(C−1jj )ka(C −1 jj )n`(Yj)nb
− N∑
m=1
(C−1jj )km(C −1 jj )a`(Yj)mb
= −(C−1jj )ka(C −1 jj Yj)`b
− (C−1jj )a`(C −1 jj Yj)kb
Substituting this into the expression for ∂(Pj)cd∂(Yj)ab and simplifying matrix products, we find that
∂(Pj)cd ∂(Yj)ab = δcb(C −1 jj Yj)ad + δdb(C −1 jj Yj)ac
− (C−1jj Yj)ac(Y > j C −1 jj Yj)bd − (C−1jj Yj)ad(Y > j C −1 jj Yj)bc
= (IN − Pj)cb(C−1jj Yj)ad + (IN − Pj)db(C−1jj Yj)ac
Finally, substituting this into our expression for ∂L∂(Yj)ab , we find that
∂L
∂(Yj)ab = N∑ c,d=1 (G>G)cd(IN − Pj)cb(C−1jj Yj)ad
+ N∑ c,d=1 (G>G)cd(IN − Pj)db(C−1jj Yj)ac
= 2[C−1jj YjG >G(IN − Pj)]ab
Therefore, ∂L
∂Yj = 2C−1jj YjG >G(IN − Pj)
But recall that Uj = C−1jj YjG >. Using this, the gradient simplifies as follows:
∂L ∂Yj = 2UjG− 2UjU>j Yj
Thus, the gradient is the difference between the r-dimensional auxiliary representation G embedded into the subspace spanned by the columns of Uj (the first term) and the projection of the network outputs in Yj = fj(Xj) onto said subspace (the second term). Intuitively, if the auxiliary representation G is far away from the view-specific representation U>j fj(Xj), then the network weights should receive a large update.
APPENDIX B DGCCA OPTIMIZATION PSEUDOCODE
Algorithm 1 contains the pseudocode for the DGCCA optimization algorithm. In practice we use stocastic optimization with minibatches, following Wang et al. (2015c).
Algorithm 1 Deep Generalized CCA Input: multiview data: X1, X2, . . . , XJ ,
number of iterations T , learning rate η Output: O1, O2, . . . , OJ Initialize weights W1,W2, . . . ,WJ for iteration t = 1, 2, . . . , T do
for each view j = 1, 2, . . . , J do Oj ← forward pass of Xj with weights Wj mean-center Oj end for U1, . . . , UJ , G← gcca(O1, . . . , OJ ) for each view j = 1, 2, . . . , J do ∂F/∂Oj ← UjU>j Oj − UjG ∇Wj ← backprop(∂F/∂Oj ,Wj) Wj ←Wj − η∇Wj
end for end for for each view j = 1, 2, . . . , J do Oj ← forward pass of Xj with weights Wj mean-center Oj end for U1, . . . , UJ , G← gcca(O1, . . . , OJ ) for each view j = 1...J do Oj ← U>j Oj end for
APPENDIX C RECONSTRUCTION ERROR AND DOWNSTREAM
PERFORMANCE
CCA methods are typically evaluated intrinsically by the amount of correlation captured, or reconstruction error. These measures are dependent on the width of the shared embeddings and viewspecific output layers, and do not necessarily predict downstream performance. Although reconstruction error cannot solely be relied on for model selection for a downstream task, we found that it was a useful as a signal to weed out very poor models. Figure 6 shows the reconstruction error against hashtag prediction Recall at 1000 for an initial grid search of DGCCA hyperparameters. Models with tuning reconstruction error greater than 103 can safely be ignored, while there is some variability in the performance of models with achieving lower error.
Since a DGCCA model with high reconstruction error suggests that the views do not agree with each other at all, it makes sense that the shared embedding will likely be noisy, whereas a relatively lowly reconstruction error suggests that the transformed views have converged to a stable solution.

The proposed method is simple and elegant; it builds upon the huge success of gradient based optimization for deep non-linear function approximators and combines it with established (linear) many-view CCA methods. A major contribution of this paper is the derivation of the gradients with respect to the non-linear encoding networks which project the different views into a common space. The derivation seems correct. In general this approach seems very interesting and I could imagine that it might be applicable to many other similarly structured problems.
The paper is well written; but it could be enhanced with an explicit description of the complete algorithm which also highlights how the joint embeddings G and U are updated.

I don’t have prior experience with CCA-style many-view techniques and it is therefore hard for me to judge the practical/empirical progress presented here. But the experiments seem reasonable convincing; although generally only performed on small and medium sized datasets.

Detailed comments:
The colours or the sign of the x-axis in figure 3b seem to be flipped compared to figure 4.

It would be nice to additionally see a continuous (rainbow-coloured) version for Figures 2, 3 and 4 to better identify neighbouring datapoints; but more importantly: I’d like to see how the average reconstruction error between the individual network outputs and the learned representation develop during training.  Is the mismatch between different views on a validation/test-set a useful metric for cross validation? In general, it seems the method is sensitive to regularization and hyperparameter selection  (because it has many more parameters compared to GCCA and different regularization parameters have been chosen for different views) and I wonder if there is a clear metric to optimize these.


GENERATIVE MULTI-ADVERSARIAL NETWORKS
Generative adversarial networks (GANs) are a framework for producing a generative model by way of a two-player minimax game. In this paper, we propose the Generative Multi-Adversarial Network (GMAN), a framework that extends GANs to multiple discriminators. In previous work, the successful training of GANs requires modifying the minimax objective to accelerate training early on. In contrast, GMAN can be reliably trained with the original, untampered objective. We explore a number of design perspectives with the discriminator role ranging from formidable adversary to forgiving teacher. Image generation tasks comparing the proposed framework to standard GANs demonstrate GMAN produces higher quality samples in a fraction of the iterations when measured by a pairwise GAM-type metric.
1 INTRODUCTION
Generative adversarial networks (Goodfellow et al. (2014)) (GANs) are a framework for producing a generative model by way of a two-player minimax game. One player, the generator, attempts to generate realistic data samples by transforming noisy samples, z, drawn from a simple distribution (e.g., z ∼ N (0, 1)) using a transformation function Gθ(z) with learned weights, θ. The generator receives feedback as to how realistic its synthetic sample is from another player, the discriminator, which attempts to discern between synthetic data samples produced by the generator and samples drawn from an actual dataset using a function Dω(x) with learned weights, ω.
The GAN framework is one of the more recent successes in a line of research on adversarial training in machine learning (Schmidhuber (1992); Bagnell (2005); Ajakan et al. (2014)) where games between learners are carefully crafted so that Nash equilibria coincide with some set of desired optimality criteria. Preliminary work on GANs focused on generating images (e.g., MNIST (LeCun et al. (1998)), CIFAR (Krizhevsky (2009))), however, GANs have proven useful in a variety of application domains including learning censored representations (Edwards & Storkey (2015)), imitating expert policies (Ho & Ermon (2016)), and domain transfer (Yoo et al. (2016)). Work extending GANs to semi-supervised learning (Chen et al. (2016); Mirza & Osindero (2014); Gauthier (2014); Springenberg (2015)), inference (Makhzani et al. (2015); Dumoulin et al. (2016)), feature learning (Donahue et al. (2016)), and improved image generation (Im et al. (2016); Denton et al. (2015); Radford et al. (2015)) have shown promise as well.
Despite these successes, GANs are reputably difficult to train. While research is still underway to improve training techniques and heuristics (Salimans et al. (2016)), most approaches have focused on understanding and generalizing GANs theoretically with the aim of exploring more tractable formulations (Zhao et al. (2016); Li et al. (2015); Uehara et al. (2016); Nowozin et al. (2016)).
In this paper, we theoretically and empirically justify generalizing the GAN framework to multiple discriminators. We review GANs and summarize our extension in Section 2. In Sections 3 and 4, we present our N -discriminator extension to the GAN framework (Generative Multi-Adversarial Networks) with several variants which range the role of the discriminator from formidable adversary to forgiving teacher. Section 4.2 explains how this extension makes training with the untampered minimax objective tractable. In Section 5, we define an intuitive metric (GMAM) to quantify GMAN ∗Equal contribution
performance and evaluate our framework on a variety of image generation tasks. Section 6 concludes with a summary of our contributions and directions for future research.
Contributions—To summarize, our main contributions are: i) a multi-discriminator GAN framework, GMAN, that allows training with the original, untampered minimax objective; ii) a generative multi-adversarial metric (GMAM) to perform pairwise evaluation of separately trained frameworks; iii) a particular instance of GMAN, GMAN∗, that allows the generator to automatically regulate training and reach higher performance (as measured by GMAM) in a fraction of the training time required for the standard GAN model.
2 GENERATIVE ADVERSARIAL NETWORKS TO GMAN
The original formulation of a GAN is a minimax game between a generator, Gθ(z) : z → x, and a discriminator, Dω(x) : x→ [0, 1],
min G max D∈D
V (D,G) = Ex∼pdata(x) [ log(D(x)) ] + Ez∼pz(z) [ log(1−D(G(z))) ] , (1)
where pdata(x) is the true data distribution and pz(z) is a simple (usually fixed) distribution that is easy to draw samples from (e.g., N (0, 1)). We differentiate between the function space of discriminators, D, and elements of this space, D. Let pG(x) be the distribution induced by the generator, Gθ(z). We assume D,G to be deep neural networks as is typically the case.
In their original work, Goodfellow et al. (2014) proved that given sufficient network capacities and an oracle providing the optimal discriminator, D∗ = argmaxD V (D,G), gradient descent on pG(x) will recover the desired globally optimal solution, pG(x) = pdata(x), so that the generator distribution exactly matches the data distribution. In practice, they replaced the second term, log(1− D(G(z))), with − log(D(G(z))) to enhance gradient signals at the start of the game; note this is no longer a zero-sum game. Part of their convergence and optimality proof involves using the oracle, D∗, to reduce the minimax game to a minimization over G only:
min G V (D∗, G) = min G
{ C(G) = − log(4) + 2 · JSD(pdata||pG) } (2)
where JSD denotes Jensen-Shannon divergence. Minimizing C(G) necessarily minimizes JSD, however, we rarely know D∗ and so we instead minimize V (D,G), which is only a lower bound.
This perspective of minimizing the distance between the distributions, pdata and pG, motivated Li et al. (2015) to develop a generative model that matches all moments of pG(x) with pdata(x) (at optimality) by minimizing maximum mean discrepancy (MMD). Another approach, EBGAN, (Zhao et al. (2016)) explores a larger class of games (non-zero-sum games) which generalize the generator and discriminator objectives to take real-valued “energies” as input instead of probabilities. Nowozin et al. (2016) and then Uehara et al. (2016) extended the JSD perspective on GANs to more general divergences, specifically f -divergences and then Bregman-divergences respectively.
In general, these approaches focus on exploring fundamental reformulations of V (D,G). Similarly, our work focuses on a fundamental reformulation, however, our aim is to provide a framework that accelerates training of the generator to a more robust state irrespective of the choice of V .
2.1 GMAN: A MULTI-ADVERSARIAL EXTENSION
We propose introducing multiple discriminators, which brings with it a number of design possibilities. We explore approaches ranging between two extremes: 1) a more discriminating D (better approximating maxD V (D,G)) and 2) a D better matched to the generator’s capabilities. Mathematically, we reformulate G’s objective as minG maxF (V (D1, G), . . . , V (DN , G)) for different choices of F (see Figure 1). Each Di is still expected to independently maximize its own V (Di, G) (i.e. no cooperation). We sometimes abbreviate V (Di, G) with Vi and F (V1, . . . , VN ) with FG(Vi).
3 A FORMIDABLE ADVERSARY
Here, we consider multi-discriminator variants that attempt to better approximate maxD V (D,G), providing a harsher critic to the generator.
3.1 MAXIMIZING V(D,G)
For a fixed G, maximizing FG(Vi) with F := max and N randomly instantiated copies of our discriminator is functionally equivalent to optimizing V (e.g., stochastic gradient ascent) with random restarts in parallel and then presenting maxi∈{1,...,N} V (Di, G) as the loss to the generator —a very pragmatic approach to the difficulties presented by the non-convexity of V caused by the deep net. Requiring the generator to minimize the max forces G to generate high fidelity samples that must hold up under the scrutiny of all N discriminators, each potentially representing a distinct max.
In practice, maxDi∈D V (Di, G) is not performed to convergence (or global optimality), so the above problem is oversimplified. Furthermore, introducing N discriminators affects the dynamics of the game which affects the trajectories of the discriminators. This prevents us from claiming max{V1(t), . . . , VN (t)} > max{V ′1(t)} ∀t even if we initalize D1(0) = D′1(0) as it is unlikely that D1(t) = D ′ 1(t) at some time t after the start of the game.
3.2 BOOSTING
We can also consider taking the max over N discriminators as a form of boosting for the discriminator’s online classification problem (online because G can produce an infinite data stream). The boosted discriminator is given a sample xt and must predict whether it came from the generator or the dataset. The booster then makes its prediction using the predictions of the N weaker Di.
There are a few differences between taking the max (case 1) and online boosting (case 2). In case 1, our booster is limited to selecting a single weak discriminator (i.e. a pure strategy), while in case 2, many boosting algorithms more generally use linear combinations of the discriminators. Moreover, in case 2, a booster must make a prediction before receiving a loss function. In case 1, we assume access to the loss function at prediction time, which allows us to compute the max.
It is possible to train the weak discriminators using boosting and then ignore the booster’s prediction by instead presenting max{Vi}. We explore both variants in our experiments, using the adaptive algorithm proposed in Beygelzimer et al. (2015). Unfortunately, boosting failed to produce promising results on the image generation tasks. It is possible that boosting produces too strong an adversary for learning which motivates the next section. Boosting results appear in Appendix A.7.
4 A FORGIVING TEACHER
The previous perspectives focus on improving the discriminator with the goal of presenting a better approximation of maxD V (D,G) to the generator. Our next perspective asks the question, “Is maxD V (D,G) too harsh a critic?”
4.1 Soft-DISCRIMINATOR
In practice, training against a far superior discriminator can impede the generator’s learning. This is because the generator is unlikely to generate any samples considered “realistic” by the discriminator’s standards, and so the generator will receive uniformly negative feedback. This is problem-
atic because the information contained in the gradient derived from negative feedback only dictates where to drive down pG(x), not specifically where to increase pG(x). Furthermore, driving down pG(x) necessarily increases pG(x) in other regions of X (to maintain ∫ X pG(x) = 1) which may or may not contain samples from the true dataset (whack-a-mole dilemma). In contrast, a generator is more likely to see positive feedback against a more lenient discriminator, which may better guide a generator towards amassing pG(x) in approximately correct regions of X . For this reason, we explore a variety of functions that allow us to soften the max operator. We choose to focus on soft versions of the three classical Pythagorean means parameterized by λ where λ = 0 corresponds to the mean and the max is recovered as λ→∞:
AMsoft(V, λ) = N∑ i wiVi (3)
GMsoft(V, λ) = − exp ( N∑
i
wi log(−Vi) )
(4)
HMsoft(V, λ) = ( N∑
i
wiV −1 i
)−1 (5)
wherewi = eλVi/ΣjeλVj with λ ≥ 0, Vi < 0. Using a softmax also has the well known advantage of being differentiable (as opposed to subdifferentiable for max). Note that we only require continuity to guarantee that computing the softmax is actually equivalent to computing V (D̃,G) where D̃ is some convex combination of Di (see Appendix A.5).
4.2 USING THE ORIGINAL MINIMAX OBJECTIVE
To illustrate the effect the softmax has on training, observe that the component of AMsoft(V, 0) relevant to generator training can be rewritten as
1
N N∑ i Ex∼pG(x) [ log(1−Di(x)) ] = 1 N Ex∼pG(x) [ log(z) ] . (6)
where z = ∏N i (1−Di(x)). Note that the generator gradient, | ∂ log(z) ∂z |, is minimized at z = 1 over z ∈ (0, 1]1. From this form, it is clear that z = 1 if and only if Di = 0∀i, so G only receives a vanishing gradient if all Di agree that the sample is fake; this is especially unlikely for large N . In other words,G only needs to fool a singleDi to receive constructive feedback. This result allows the generator to successfully minimize the original generator objective, log(1−D). This is in contrast to the more popular − log(D) introduced to artificially enhance gradients at the start of training. At the beginning of training, when maxDi V (Di, G) is likely too harsh a critic for the generator, we can set λ closer to zero to use the mean, increasing the odds of providing constructive feedback to the generator. In addition, the discriminators have the added benefit of functioning as an ensemble, reducing the variance of the feedback presented to the generator, which is especially important when the discriminators are far from optimal and are still learning a reasonable decision boundary. As training progresses and the discriminators improve, we can increase λ to become more critical of the generator for more refined training.
4.3 MAINTAINING MULTIPLE HYPOTHESES
We argue for this ensemble approach on a more fundamental level as well. Here, we draw on the density ratio estimation perspective of GANs (Uehara et al. (2016)). The original GAN proof assumes we have access to pdata(x), if only implicitly. In most cases of interest, the discriminator only has access to a finite dataset sampled from pdata(x); therefore, when computing expectations of V (D,G), we only draw samples from our finite dataset. This is equivalent to training a GAN with pdata(x) = p̃data which is a distribution consisting of point masses on all the data points in the dataset. For the sake of argument, let’s assume we are training a discriminator and generator, each
1∇GV = − ∑
i Di z ∂Di ∂G ∏ j 6=i(1−Dj) = − 1 z ∂Dk ∂G for Dk = 1, D 6=k = 0. Our argument ignores ∂Dk∂G .
with infinite capacity. In this case, the global optimum (pG(x) = p̃data(x)) fails to capture any of the interesting structure from pdata(x), the true distribution we are trying to learn. Therefore, it is actually critical that we avoid this global optimum.
In practice, this degenerate result is avoided by employing learners with limited capacity and corrupting data samples with noise (i.e., dropout), but we might better accomplish this by simultaneously training a variety of limited capacity discriminators. With this approach, we might obtain a diverse set of seemingly tenable hypotheses for the true pdata(x). Averaging over these multiple locally optimal discriminators increases the entropy of p̃data(x) by diffusing the probability mass over the data space (see Figure 2 for an example).
4.4 AUTOMATING REGULATION
The problem of keeping the discriminator and generator in balance has been widely recognized in previous work with GANs. Issues with unstable dynamics, oscillatory behavior, and generator collapse are not uncommon. In addition, the discriminator is often times able to achieve a high degree of classification accuracy (producing a single scalar) before the generator has made sufficient progress on the arguably more difficult generative task (producing a high dimensional sample). Salimans et al. (2016) suggested label smoothing to reduce the vulnerability of the generator to a relatively superior discriminator. Here, we explore an approach that enables the generator to automatically temper the performance of the discriminator when necessary, but still encourages the generator to challenge itself against more accurate adversaries. Specifically, we augment the generator objective:
min G,λ>0
FG(Vi)− f(λ) (7)
where f(λ) is monotonically increasing in λ which appears in the softmax equations, (3)—(5). In experiments, we simply set f(λ) = cλ with c a constant (e.g., 0.001). The generator is incentivized to increase λ to reduce its objective at the expense of competing against the best available adversary D∗ (see Appendix A.6).
5 EVALUATION
Evaluating GANs is still an open problem. In their original work, Goodfellow et al. (2014) report log likelihood estimates from Gaussian Parzen windows, which they admit, has high variance and is known not to perform well in high dimensions. Theis et al. (2016) recommend avoiding Parzen windows and argue that generative models should be evaluated with respect to their intended application. Salimans et al. (2016) suggest an Inception score, however, it assumes labels exist for the dataset. Recently, Im et al. (2016) introduced the Generative Adversarial Metric (GAM) for making pairwise comparisons between independently trained GAN models. The core idea behind their approach is given two generator, discriminator pairs (G1, D1) and (G2, D2), we should be able to learn their relative performance by judging each generator under the opponent’s discriminator.
5.1 METRIC
In GMAN, the opponent may have multiple discriminators, which makes it unclear how to perform the swaps needed for GAM. We introduce a variant of GAM, the generative multi-adversarial metric (GMAM), that is amenable to training with multiple discriminators,
GMAM = log (F aGb(V ai ) F aGa(V a i ) /F bGa(V bi ) F bGb(V b i ) ) . (8)
where a and b refer to the two GMAN variants (see Section 3 for notation FG(Vi)). The idea here is similar. If G2 performs better than G1 with respect to both D1 and D2, then GMAM>0 (remember V≤0 always). IfG1 performs better in both cases, GMAM<0, otherwise, the result is indeterminate.
5.2 EXPERIMENTS
We evaluate the aforementioned variations of GMAN on a variety of image generation tasks: MNIST (LeCun et al. (1998)), CIFAR-10 (Krizhevsky (2009)) and CelebA (Liu et al. (2015)). We focus on rates of convergence to steady state along with quality of the steady state generator according to the GMAM metric. To summarize, loosely in order of increasing discriminator leniency, we compare
• F-boost: A single AdaBoost.OL-boosted discriminator (see Appendix A.7). • P-boost: Di is trained according to AdaBoost.OL. A max over the weak learner losses is
presented to the generator instead of the boosted prediction (see Appendix A.7).
• GMAN-max: max{Vi} is presented to the generator. • GAN: Standard GAN with a single discriminator (see Appendix A.2). • mod-GAN: GAN with modified objective (generator minimizes − log(D(G(z))). • GMAN-λ: GMAN with F :=arithmetic softmax with parameter λ. • GMAN∗: The arithmetic softmax is controlled by the generator through λ.
All generator and discriminator models are deep (de)convolutional networks (Radford et al. (2015)), and aside from the boosted variants, all are trained with Adam (Kingma & Ba (2014)) and batch normalization (Ioffe & Szegedy (2015)). Discriminators convert the real-valued outputs of their networks to probabilities with squashed-sigmoids to prevent saturating logarithms in the minimax objective ( + 1−2 1+e−z ). See Appendix A.8 for further details. We test GMAN systems with N = {2, 5} discriminators. We maintain discriminator diversity by varying dropout and network depth.
5.2.1 MNIST
Figure 3 reveals that increasing the number of discriminators reduces the number of iterations to steady-state by 2x on MNIST; increasing N (the size of the discriminator ensemble) also has the added benefit of reducing the variance the minimax objective over runs. Figure 4 displays the variance of the same objective over a sliding time window, reaffirming GMAN’s acceleration to steadystate. Figure 5 corroborates this conclusion with recognizable digits appearing approximately an epoch before the single discriminator run; digits at steady-state appear slightly sharper as well.
Our GMAM metric (see Table 1) agrees with the relative quality of images in Figure 5 with GMAN∗ achieving the best overall performance. Figure 6 reveals GMAN∗’s attempt to regulate the difficulty
of the game to accelerate learning. Figure 7 displays the GMAM scores comparing fixed λ’s to the variable λ controlled by GMAN∗.
5.2.2 CELEBA & CIFAR-10
We see similar accelerated convergence behavior for the CelebA dataset in Figure 8.
Figure 9 displays images generated by GMAN-0 on CIFAR-10. See Appendix A.3 for more results.
We also found that GMAN is robust to mode collapse. We believe this is because the generator must appease a diverse set of discriminators in each minibatch. Emitting a single sample will score well for one discriminator at the expense of the rest of the discriminators. Current solutions (e.g., minibatch discrimination) are quadratic in batch size. GMAN, however, is linear in batch size.
6 CONCLUSION
We introduced multiple discriminators into the GAN framework and explored discriminator roles ranging from a formidable adversary to a forgiving teacher. Allowing the generator to automatically tune its learning schedule (GMAN∗) outperformed GANs with a single discriminator on MNIST. In general, GMAN variants achieved faster convergence to a higher quality steady state on a variety of tasks as measured by a GAM-type metric (GMAM). In addition, GMAN makes using the original GAN objective possible by increasing the odds of the generator receiving constructive feedback.
In future work, we will look at more sophisticated mechanisms for letting the generator control the game as well as other ways to ensure diversity among the discriminators. Introducing multiple generators is conceptually an obvious next step, however, we expect difficulties to arise from more complex game dynamics. For this reason, game theory and game design will likely be important.
ACKNOWLEDGMENTS
We acknowledge helpful conversations with Stefan Dernbach, Archan Ray, Luke Vilnis, Ben Turtel, Stephen Giguere, Rajarshi Das, and Subhransu Maji. We also thank NVIDIA for donating a K40 GPU. This material is based upon work supported by the National Science Foundation under Grant Nos. IIS-1564032. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF.
A APPENDIX
A.1 ACCELERATED CONVERGENCE & REDUCED VARIANCE
See Figures 10, 11, 12, and 13.
A.2 ADDITIONAL GMAM TABLES
See Tables 2, 3, 4, 5, 6. Increasing the number of discriminators from 2 to 5 on CIFAR-10 significantly improves scores over the standard GAN both in terms of the GMAM metric and Inception scores.
A.3 GENERATED IMAGES
See Figures 14 and 15.
A.4 SOMEWHAT RELATED WORK
A GAN framework with two discriminators appeared in Yoo et al. (2016), however, it is applicable only in a semi-supervised case where a label can be assigned to subsets of the dataset (e.g., X = {X1 = Domain 1,X2 = Domain 2, . . .}). In contrast, our framework applies to an unsupervised scenario where an obvious partition of the dataset is unknown. Furthermore, extending GMAN to the semi-supervised domain-adaptation scenario would suggest multiple discriminators per domain, therefore our line of research is strictly orthogonal to that of their multi-domain discriminator approach. Also, note that assigning a discriminator to each domain is akin to prescribing a new discriminator to each value of a conditional variable in conditional GANs (Mirza & Osindero (2014)). In this case, we interpret GMAN as introducing multiple conditional discriminators and not a discriminator for each of the possibly exponentially many conditional labels.
In Section 4.4, we describe an approach to customize adversarial training to better suit the development of the generator. An approach with similar conceptual underpinnings was described in Ravanbakhsh et al. (2016), however, similar to the above, it is only admissible in a semi-supervised scenario whereas our applies to the unsupervised case.
A.5 Softmax REPRESENTABILITY
Let softmax(Vi) = V̂ ∈ [minVi ,maxVi ]. Also let a = argmini Vi, b = argmaxi Vi, and V(t) = V ((1 − t)Da + tDb) so that V(0) = Va and V(1) = Vb. The softmax and minimax objective V (Di, G) are both continuous in their inputs, so by the intermediate value theorem, we have that ∃ t̂ ∈ [0, 1] s.t. V(t̂) = V̂ , which implies ∃ D̂ ∈ D s.t. V (D̂,G) = V̂ . This result implies that the softmax (and any other continuous substitute) can be interpreted as returning V (D̂,G) for some D̂ selected by computing an another, unknown function over the space of the discriminators. This result holds even if D̂ is not representable by the architecture chosen for D’s neural network.
A.6 UNCONSTRAINED OPTIMIZATION
To convert GMAN∗ minimax formulation to an unconstrained minimax formulation, we introduce an auxiliary variable, Λ, define λ(Λ) = log(1 + eΛ), and let the generator minimize over Λ ∈ R.
A.7 BOOSTING WITH AdaBoost.OL
AdaBoost.OL (Beygelzimer et al. (2015)) does not require knowledge of the weak learner’s slight edge over random guessing (P (correct label) = 0.5 + γ ∈ (0, 0.5]), and in fact, allows γ < 0. This is crucial because our weak learners are deep nets with unknown, possibly negative, γ’s.
A.8 EXPERIMENTAL SETUP
All experiments were conducted using an architecture similar to DCGAN (Radford et al. (2015)). We use convolutional transpose layers (Zeiler et al. (2010)) for G and strided convolutions for D except for the input of G and the last layer of D. We use the single step gradient method as in (Nowozin et al. (2016)), and batch normalization (Ioffe & Szegedy (2015)) was used in each of the generator layers. The different discriminators were trained with varying dropout rates from [0.3, 0.7]. Variations in the discriminators were effected in two ways. We varied the architecture by varying the number of filters in the discriminator layers (reduced by factors of 2, 4 and so on), as well as varying dropout rates. Secondly we also decorrelated the samples that the disriminators were training on by splitting the minibatch across the discriminators. The code was written in Tensorflow (Abadi et al. (2016)) and run on Nvidia GTX 980 GPUs. Code to reproduce experiments and plots is at https://github.com/iDurugkar/GMAN. Specifics for the MNIST architecture and training are:
• Generator latent variables z ∼ U (−1, 1)100
• Generator convolution transpose layers: (4, 4, 128) , (8, 8, 64) , (16, 16, 32) , (32, 32, 1) • Base Discriminator architecture: (32, 32, 1) , (16, 16, 32) , (8, 8, 64) , (4, 4, 128). • Variants have either convolution 3 (4, 4, 128) removed or all the filter sizes
are divided by 2 or 4. That is, (32, 32, 1) , (16, 16, 16) , (8, 8, 32) , (4, 4, 64) or (32, 32, 1) , (16, 16, 8) , (8, 8, 16) , (4, 4, 32).
• ReLu activations for all the hidden units. Tanh activation at the output units of the generator. Sigmoid at the output of the Discriminator.
• Training was performed with Adam (Kingma & Ba (2014)) (lr = 2× 10−4, β1 = 0.5). • MNIST was trained for 20 epochs with a minibatch of size 100. • CelebA and CIFAR were trained over 24000 iterations with a minibatch of size 100.

The paper extends the GAN framework to accommodate multiple discriminators. The authors motivate this from two points of view:
(1) Having multiple discriminators tackle the task is equivalent to optimizing the value function using random restarts, which can potentially help optimization given the nonconvexity of the value function.
(2) Having multiple discriminators can help overcome the optimization problems arising when a discriminator is too harsh a critic. A generator receiving signal from multiple discriminators is less likely to be receiving poor gradient signal from all discriminators.
The paper's main idea looks straightforward to implement in practice and makes for a good addition to the GAN training toolbelt.
I am not very convinced by the GAM (and by extension the GMAM) evaluation metric. Without evidence that the GAN game is converging (even approximately), it is hard to make the case that the discriminators tell something meaningful about the generators with respect to the data distribution. In particular, it does not inform on mode coverage or probability mass misallocation.
The learning curves (Figure 3) look more convincing to me: they provide good evidence that increasing the number of discriminators has a stabilizing effect on the learning dynamics. However, it seems like this figure along with Figure 4 also show that the unmodified generator objective is more stable even with only one discriminator. In that case, is it even necessary to have more than one discriminator to train the generator using an unmodified objective?
Overall, I think the ideas presented in this paper show good potential, but I would like to see an extended analysis in the line of Figures 3 and 4 for more datasets before I think it is ready for publication.
UPDATE: The rating has been revised to a 7 following discussion with the authors.


DEEP VARIATIONAL BAYES FILTERS: UNSUPERVISED LEARNING OF STATE SPACE MODELS FROM RAW DATA
We introduce Deep Variational Bayes Filters (DVBF), a new method for unsupervised learning and identification of latent Markovian state space models. Leveraging recent advances in Stochastic Gradient Variational Bayes, DVBF can overcome intractable inference distributions via variational inference. Thus, it can handle highly nonlinear input data with temporal and spatial dependencies such as image sequences without domain knowledge. Our experiments show that enabling backpropagation through transitions enforces state space assumptions and significantly improves information content of the latent embedding. This also enables realistic long-term prediction.
We introduce Deep Variational Bayes Filters (DVBF), a new method for unsupervised learning and identification of latent Markovian state space models. Leveraging recent advances in Stochastic Gradient Variational Bayes, DVBF can overcome intractable inference distributions via variational inference. Thus, it can handle highly nonlinear input data with temporal and spatial dependencies such as image sequences without domain knowledge. Our experiments show that enabling backpropagation through transitions enforces state space assumptions and significantly improves information content of the latent embedding. This also enables realistic long-term prediction.
1 INTRODUCTION
Estimating probabilistic models for sequential data is central to many domains, such as audio, natural language or physical plants, Graves (2013); Watter et al. (2015); Chung et al. (2015); Deisenroth & Rasmussen (2011); Ko & Fox (2011). The goal is to obtain a model p(x1:T ) that best reflects a data set of observed sequences x1:T . Recent advances in deep learning have paved the way to powerful models capable of representing high-dimensional sequences with temporal dependencies, e.g., Graves (2013); Watter et al. (2015); Chung et al. (2015); Bayer & Osendorfer (2014).
Time series for dynamic systems have been studied extensively in systems theory, cf. McGoff et al. (2015) and sources therein. In particular, state space models have shown to be a powerful tool to analyze and control the dynamics. Two tasks remain a significant challenge to this day: Can we identify the governing system from data only? And can we perform inference from observables to the latent system variables? These two tasks are competing: A more powerful representation of system requires more computationally demanding inference, and efficient inference, such as the well-known Kalman filters, Kalman & Bucy (1961), can prohibit sufficiently complex system classes.
Leveraging a recently proposed estimator based on variational inference, stochastic gradient variational Bayes (SGVB, Kingma & Welling (2013); Rezende et al. (2014)), approximate inference of latent variables becomes tractable. Extensions to time series have been shown in Bayer & Osendorfer (2014); Chung et al. (2015). Empirically, they showed considerable improvements in marginal data likelihood, i.e., compression, but lack full-information latent states, which prohibits, e.g., long-term sampling. Yet, in a wide range of applications, full-information latent states should be valued over compression. This is crucial if the latent spaces are used in downstream applications.
Our contribution is, to our knowledge, the first model that (i) enforces the latent state-space model assumptions, allowing for reliable system identification, and plausible long-term prediction of the observable system, (ii) provides the corresponding inference mechanism with rich dependencies, (iii) inherits the merit of neural architectures to be trainable on raw data such as images or other sensory inputs, and (iv) scales to large data due to optimization of parameters based on stochastic gradient descent, Bottou (2010). Hence, our model has the potential to exploit systems theory methodology for downstream tasks, e.g., control or model-based reinforcement learning, Sutton (1996).
2 BACKGROUND AND RELATED WORK
2.1 PROBABILISTIC MODELING AND FILTERING OF DYNAMICAL SYSTEMS
We consider non-linear dynamical systems with observations xt ∈ X ⊂ Rnx , depending on control inputs (or actions) ut ∈ U ⊂ Rnu . Elements of X can be high-dimensional sensory data, e.g., raw images. In particular they may exhibit complex non-Markovian transitions. Corresponding timediscrete sequences of length T are denoted as x1:T = (x1,x2, . . . ,xT ) and u1:T = (u1,u2, . . . ,uT ).
We are interested in a probabilistic model1 p(x1:T | u1:T ). Formally, we assume the graphical model p(x1:T | u1:T ) = ∫ p(x1:T | z1:T ,u1:T ) p(z1:T | u1:T ) dz1:T , (1)
where z1:T , zt ∈ Z ⊂ Rnz , denotes the corresponding latent sequence. That is, we assume a generative model with an underlying latent dynamical system with emission model p(x1:T | z1:T ,u1:T ) and transition model p(z1:T | u1:T ). We want to learn both components, i.e., we want to perform latent system identification. In order to be able to apply the identified system in downstream tasks, we need to find efficient posterior inference distributions p(z1:T | x1:T ). Three common examples are prediction, filtering, and smoothing: inference of zt from x1:t−1, x1:t, or x1:T , respectively. Accurate identification and efficient inference are generally competing tasks, as a wider generative model class typically leads to more difficult or even intractable inference.
The transition model is imperative for achieving good long-term results: a bad transition model can lead to divergence of the latent state. Accordingly, we put special emphasis on it through a Bayesian treatment. Assuming that the transitions may differ for each time step, we impose a regularizing prior distribution on a set of transition parameters β1:T :
(1) = ∫∫
p(x1:T | z1:T ,u1:T )p(z1:T | β1:T ,u1:T ) p(β1:T ) dβ1:T dz1:T (2)
To obtain state-space models, we impose assumptions on emission and state transition model,
p(x1:T | z1:T ,u1:T ) = T∏ t=1 p(xt | zt), (3)
p(z1:T | β1:T ,u1:T ) = T−1∏ t=0 p(zt+1 | zt,ut,βt). (4)
Equations (3) and (4) assume that the current state zt contains all necessary information about the current observation xt, as well as the next state zt+1 (given the current control input ut and transition parameters βt). That is, in contrast to observations, zt exhibits Markovian behavior.
A typical example of these assumptions are Linear Gaussian Models (LGMs), i.e., both state transition and emission model are affine transformations with Gaussian offset noise,
zt+1 = Ftzt +Btut +wt wt ∼ N (0,Qt), (5) xt = Htzt + yt yt ∼ N (0,Rt). (6)
Typically, state transition matrix Ft and control-input matrix Bt are assumed to be given, so that βt = wt. Section 3.3 will show that our approach allows other variants such as βt = (Ft,Bt,wt). Under the strong assumptions (5) and (6) of LGMs, inference is provably solved optimally by the well-known Kalman filters. While extensions of Kalman filters to nonlinear dynamical systems exist, Julier & Uhlmann (1997), and are successfully applied in many areas, they suffer from two major drawbacks: firstly, its assumptions are restrictive and are violated in practical applications, leading to suboptimal results. Secondly, parameters such as Ft and Bt have to be known in order to perform posterior inference. There have been efforts to learn such system dynamics, cf. Ghahramani & Hinton (1996); Honkela et al. (2010) based on the expectation maximization (EM) algorithm or Valpola & Karhunen (2002), which uses neural networks. However, these algorithms are not applicable in cases
1Throughout this paper, we consider u1:T as given. The case without any control inputs can be recovered by setting U = ∅, i.e., not conditioning on control inputs.
where the true posterior distribution is intractable. This is the case if, e.g., image sequences are used, since the posterior is then highly nonlinear—typical mean-field assumptions on the approximate posterior are too simplified. Our new approach will tackle both issues, and moreover learn both identification and inference jointly by exploiting Stochastic Gradient Variational Bayes.
2.2 STOCHASTIC GRADIENT VARIATIONAL BAYES (SGVB) FOR TIME SERIES DISTRIBUTIONS
Replacing the bottleneck layer of a deterministic auto-encoder with stochastic units z, the variational auto-encoder (VAE, Kingma & Welling (2013); Rezende et al. (2014)) learns complex marginal data distributions on x in an unsupervised fashion from simpler distributions via the graphical model
p(x) = ∫ p(x, z) dz = ∫ p(x | z)p(z) dz.
In VAEs, p(x | z) ≡ pθ(x | z) is typically parametrized by a neural network with parameters θ. Within this framework, models are trained by maximizing a lower bound to the marginal data log-likelihood via stochastic gradients:
ln p(x) ≥ Eqφ(z|x)[ln pθ(x | z)]−KL(qφ(z | x) || p(z)) =: LSGVB(x, φ, θ) (7) This is provably equivalent to minimizing the KL-divergence between the approximate posterior or recognition model qφ(z | x) and the true, but usually intractable posterior distribution p(z | x). qφ is parametrized by a neural network with parameters φ.
The principle of VAEs has been transferred to time series, Bayer & Osendorfer (2014); Chung et al. (2015). Both employ nonlinear state transitions in latent space, but violate eq. (4): Observations are directly included in the transition process. Empirically, reconstruction and compression work well. The state space Z , however, does not reflect all information available, which prohibits plausible generative long-term prediction. Such phenomena with generative models have been explained in Theis et al. (2015).
In Krishnan et al. (2015), the state-space assumptions (3) and (4) are softly encoded in the Deep Kalman Filter (DKF) model. Despite that, experiments, cf. section 4, show that their model fails to extract information such as velocity (and in general time derivatives), which leads to similar problems with prediction.
Johnson et al. (2016) give an algorithm for general graphical model variational inference, not tailored to dynamical systems. In contrast to previously discussed methods, it does not violate eq. (4). The approaches differ in that the recognition model outputs node potentials in combination with message passing to infer the latent state. Our approach focuses on learning dynamical systems for controlrelated tasks and therefore uses a neural network for inferring the latent state directly instead of an inference subroutine.
Others have been specifically interested in applying variational inference for controlled dynamical systems. In Watter et al. (2015) (Embed to Control—E2C), a VAE is used to learn the mappings to and from latent space. The regularization is clearly motivated by eq. (7). Still, it fails to be a mathematically correct lower bound to the marginal data likelihood. More significantly, their recognition model requires all observations that contain information w.r.t. the current state. This is nothing short of an additional temporal i.i.d. assumption on data: Multiple raw samples need to be stacked into one training sample such that all latent factors (in particular all time derivatives) are present within one sample. The task is thus greatly simplified, because instead of time-series, we learn a static auto-encoder on the processed data.
A pattern emerges: good prediction should boost compression. Still, previous methods empirically excel at compression, while prediction will not work. We conjecture that this is caused by previous methods trying to fit the latent dynamics to a latent state that is beneficial for reconstruction. This encourages learning of a stationary auto-encoder with focus of extracting as much from a single observation as possible. Importantly, it is not necessary to know the entire sequence for excellent reconstruction of single time steps. Once the latent states are set, it is hard to adjust the transition to them. This would require changing the latent states slightly, and that comes at a cost of decreasing the reconstruction (temporarily). The learning algorithm is stuck in a local optimum with good reconstruction and hence good compression only. Intriguingly, E2C bypasses this problem with its data augmentation.
This leads to a key contribution of this paper: We force the latent space to fit the transition—reversing the direction, and thus achieving the state-space model assumptions and full information in the latent states.
3 DEEP VARIATIONAL BAYES FILTERS
3.1 REPARAMETRIZING THE TRANSITION
The central problem for learning latent states system dynamics is efficient inference of a latent space that obeys state-space model assumptions. If the latter are fulfilled, the latent space must contain all information. Previous approaches emphasized good reconstruction, so that the space only contains information necessary for reconstruction of one time step. To overcome this, we establish gradient paths through transitions over time so that the transition becomes the driving factor for shaping the latent space, rather than adjusting the transition to the recognition model’s latent space. The key is to prevent the recognition model qφ(z1:T | x1:T ) from directly drawing the latent state zt. Similar to the reparametrization trick from Kingma & Welling (2013); Rezende et al. (2014) for making the Monte Carlo estimate differentiable w.r.t. the parameters, we make the transition differentiable w.r.t. the last state and its parameters:
zt+1 = f(zt,ut,βt) (8) Given the stochastic parameters βt, the state transition is deterministic (which in turn means that by marginalizing βt, we still have a stochastic transition). The immediate and crucial consequence is that errors in reconstruction of xt from zt are backpropagated directly through time.
This reparametrization has a couple of other important implications: the recognition model no longer infers latent states zt, but transition parameters βt. In particular, the gradient ∂zt+1/∂zt is well-defined from (8)—gradient information can be backpropagated through the transition.
This is different from the method used in Krishnan et al. (2015), where the transition only occurs in the KL-divergence term of their loss function (a variant of eq. (7)). No gradient from the generative model is backpropagated through the transitions.
Much like in eq. (5), the stochastic parameters includes a corrective offset term wt, which emphasizes the notion of the recognition model as a filter. In theory, the learning algorithm could still learn the transition as zt+1 = wt. However, the introduction of βt also enables us to regularize the transition with meaningful priors, which not only prevents overfitting the recognition model, but also enforces meaningful manifolds in the latent space via transition priors. Ignoring the potential of the transition over time yields large penalties from these priors. Thus, the problems outlined in Section 2 are overcome by construction.
To install such transition priors, we split βt = (wt,vt). The interpretation of wt is a sample-specific process noise which can be inferred from incoming data, like in eq. (5). On the other hand, vt
are universal transition parameters, which are sample-independent (and are only inferred from data during training). This corresponds to the idea of weight uncertainty in Hinton & Van Camp (1993). This interpretation leads to a natural factorization assumption on the recognition model:
qφ(β1:T | x1:T ) = qφ(w1:T | x1:T ) qφ(v1:T ) (9)
When using the fully trained model for generative sampling, i.e., sampling without input, the universal state transition parameters can still be drawn from qφ(v1:T ), whereas w1:T is drawn from the prior in the absence of input data.
Figure 1 shows the underlying graphical model and the inference procedure. Figure 2a shows a generic view on our new computational architecture. An example of a locally linear transition parametrization will be given in section 3.3.
3.2 THE LOWER BOUND OBJECTIVE FUNCTION
In analogy to eq. (7), we now derive a lower bound to the marginal likelihood p(x1:T | u1:T ). After reflecting the Markov assumptions (3) and (4) in the factorized likelihood (2), we have:
p(x1:T | u1:T ) = ∫∫ p(β1:T ) T∏ t=1 pθ(xt | zt) T−1∏ t=0 p(zt+1 | zt,ut,βt) dβ1:T dz1:T
Due to the deterministic transition given βt+1, the last term is a product of Dirac distributions and the overall distribution simplifies greatly:
p(x1:T | u1:T ) = ∫ p(β1:T ) T∏ t=1 pθ(xt | zt) ∣∣∣ zt=f(zt−1,ut−1,βt−1)
dβ1:T( = ∫ p(β1:T )pθ(x1:T | z1:T ) dβ1:T )
The last formulation is for notational brevity: the term pθ(x1:T | z1:T ) is not independent of β1:T and u1:T . We now derive the objective function, a lower bound to the data likelihood:
ln p(x1:T | u1:T ) = ln ∫ p(β1:T )pθ(x1:T | z1:T )
qφ(β1:T | x1:T ,u1:T ) qφ(β1:T | x1:T ,u1:T ) dβ1:T
≥ ∫ qφ(β1:T | x1:T ,u1:T ) ln ( pθ(x1:T | z1:T )
p(β1:T )
qφ(β1:T | x1:T ,u1:T )
) dβ1:T
= Eqφ [ln pθ(x1:T | z1:T )− ln qφ(β1:T | x1:T ,u1:T ) + ln p(β1:T )] (10) = Eqφ [ln pθ(x1:T | z1:T )]−KL(qφ(β1:T | x1:T ,u1:T ) || p(β1:T )) (11) =: LDVBF(x1:T , θ, φ | u1:T )
Our experiments show that an annealed version of (10) is beneficial to the overall performance:
(10′) = Eqφ [ci ln pθ(x1:T | z1:T )− ln qφ(β1:T | x1:T ,u1:T ) + ci ln p(w1:T ) + ln p(v1:T )] Here, ci = max(1, 0.01 + i/TA) is an inverse temperature that increases linearly in the number of gradient updates i until reaching 1 after TA annealing iterations. Similar annealing schedules have been applied in, e.g., Ghahramani & Hinton (2000); Mandt et al. (2016); Rezende & Mohamed (2015), where it is shown that they smooth the typically highly non-convex error landscape. Additionally, the transition prior p(v1:T ) was estimated during optimization, i.e., through an empirical Bayes approach. In all experiments, we used isotropic Gaussian priors.
3.3 EXAMPLE: LOCALLY LINEAR TRANSITIONS
We have derived a learning algorithm for time series with particular focus on general transitions in latent space. Inspired by Watter et al. (2015), this section will show how to learn a particular instance: locally linear state transitions. That is, we set eq. (8) to
zt+1 = Atzt +Btut +Ctwt, t = 1, . . . , T, (12)
where wt is a stochastic sample from the recognition model and At,Bt, and Ct are matrices of matching dimensions. They are stochastic functions of zt and ut (thus local linearity). We draw
vt = { A (i) t ,B (i) t ,C (i) t | i = 1, . . . ,M } ,
from qφ(vt), i.e.,M triplets of matrices, each corresponding to data-independent, but learned globally linear system. These can be learned as point estimates. We employed a Bayesian treatment as in Blundell et al. (2015). We yield At,Bt, and Ct as state- and control-dependent linear combinations:
At = M∑ i=1 α (i) t A (i) t
αt = fψ(zt,ut) ∈ RM
Bt = M∑ i=1 α (i) t B (i) t
Ct = M∑ i=1 α (i) t C (i) t
The computation is depicted in fig. 2b. The function fψ can be, e.g., a (deterministic) neural network with weights ψ. As a subset of the generative parameters θ, ψ is part of the trainable parameters of our model. The weight vector αt is shared between the three matrices. There is a correspondence to eq. (5): At and Ft, Bt and Bt, as well as CtC>t and Qt are related.
We used this parametrization of the state transition model for our experiments. It is important that the parametrization is up to the user and the respective application.
4 EXPERIMENTS AND RESULTS
In this section we validate that DVBF with locally linear transitions (DVBF-LL) (section 3.3) outperforms Deep Kalman Filters (DKF, Krishnan et al. (2015)) in recovering latent spaces with full information. 2 We focus on environments that can be simulated with full knowledge of the
2We do not include E2C, Watter et al. (2015), due to the need for data modification and its inability to provide a correct lower bound as mentioned in section 2.2.
ground truth latent dynamical system. The experimental setup is described in the Supplementary Material. We published the code for DVBF and a link will be made available at https://brml. org/projects/dvbf.
4.1 DYNAMIC PENDULUM
In order to test our algorithm on truly non-Markovian observations of a dynamical system, we simulated a dynamic torque-controlled pendulum governed by the differential equation
ml2ϕ̈(t) = −µϕ̇(t) +mgl sinϕ(t) + u(t),
m = l = 1, µ = 0.5, g = 9.81, via numerical integration, and then converted the ground-truth angle ϕ into an image observation in X . The one-dimensional control corresponds to angle acceleration (which is proportional to joint torque). Angle and angular velocity fully describe the system.
Figure 3 shows the latent spaces for identical input data learned by DVBF-LL and DKF, respectively, colored with the ground truth in the top row. It should be noted that latent samples are shown, not means of posterior distributions. The state-space model was allowed to use three latent dimensions. As we can see in fig. 3a, DVBF-LL learned a two-dimensional manifold embedding, i.e., it encoded the angle in polar coordinates (thus circumventing the discontinuity of angles modulo 2π). The bottom row shows ordinary least-squares regressions (OLS) underlining the performance: there exists a high correlation between latent states and ground-truth angle and angular velocity for DVBF-LL. On the contrary, fig. 3b verifies our prediction that DKF is equally capable of learning the angle, but extracts little to no information on angular velocity.
The OLS regression results shown in table 1 validate this observation.3 Predicting sin(ϕ) and cos(ϕ), i.e., polar coordinates of the ground-truth angle ϕ, works almost equally well for DVBF-LL and DKF, with DVBF-LL slightly outperforming DKF. For predicting the ground truth velocity ϕ̇, DVBF-LL
3Linear regression is a natural choice: after transforming the ground truth to polar coordinates, an affine transformation should be a good fit for predicting ground truth from latent states. We also tried nonlinear regression with vanilla neural networks. While not being shown here, the results underlined the same conclusion.
Table 1: Results for pendulum OLS regressions of all latent states on respective dependent variable.
Dependent ground truth variable
DVBF-LL DKF Log-Likelihood R2 Log-Likelihood R2
sin(ϕ) 3990.8 0.961 1737.6 0.929 cos(ϕ) 7231.1 0.982 6614.2 0.979 ϕ̇ −11139 0.916 −20289 0.035
(a) Generative latent walk. (b) Reconstructive latent walk.
...
...
... 51 10 15 20 40 45
(c) Ground truth (top), reconstructions (middle), generative samples (bottom) from identical initial latent state.
Figure 4: (a) Latent space walk in generative mode. (b) Latent space walk in filtering mode. (c) Ground truth and samples from recognition and generative model. The reconstruction sampling has access to observation sequence and performs filtering. The generative samples only get access to the observations once for creating the initial state while all subsequent samples are predicted from this single initial state. The red bar indicates the length of training sequences. Samples beyond show the generalization capabilities for sequences longer than during training. The complete sequence can be found in the Appendix in fig. 7.
shows remarkable performance. DKF, instead, contains hardly any information, resulting in a very low goodness-of-fit score of R2 = 0.035.
Figure 4 shows that the strong relation between ground truth and latent state is beneficial for generative sampling. All plots show 100 time steps of a pendulum starting from the exact same latent state and not being actuated. The top row plots show a purely generative walk in the latent space on the left, and a walk in latent space that is corrected by filtering observations on the right. We can see that both follow a similar trajectory to an attractor. The generative model is more prone to noise when approaching the attractor.
The bottom plot shows the first 45 steps of the corresponding observations (top row), reconstructions (middle row), and generative samples (without correcting from observations). Interestingly, DVBF works very well even though the sequence is much longer than all training sequences (indicated by the red line).
Table (2) shows values of the lower bound to the marginal data likelihood (for DVBF-LL, this corresponds to eq. (11)). We see that DVBF-LL outperforms DKF in terms of compression, but only
Table 2: Average test set objective function values for pendulum experiment.
Lower Bound = Reconstruction Error − KL divergence DVBF-LL 798.56 802.06 3.50
DKF 784.70 788.58 3.88
(a) Latent walk of bouncing ball. (b) Latent space velocities.
Figure 5: (a) Two dimensions of 4D bouncing ball latent space. Ground truth x and y coordinates are combined into a regular 3×3 checkerboard coloring. This checkerboard is correctly extracted by the embedding. (b) Remaining two latent dimensions. Same latent samples, colored with ball velocities in x and y direction (left and right image, respectively). The smooth, perpendicular coloring indicates that the ground truth value is stored in the latent dimension.
with a slight margin, which does not reflect the better generative sampling as Theis et al. (2015) argue.
4.2 BOUNCING BALL
The bouncing ball experiment features a ball rolling within a bounding box in a plane. The system has a two-dimensional control input, added to the directed velocity of the ball. If the ball hits the wall, it bounces off, so that the true dynamics are highly dependent on the current position and velocity of the ball. The system’s state is four-dimensional, two dimensions each for position and velocity.
Consequently, we use a DVBF-LL with four latent dimensions. Figure 5 shows that DVBF again captures the entire system dynamics in the latent space. The checkerboard is quite a remarkable result: the ground truth position of the ball lies within the 2D unit square, the bounding box. In order to visualize how ground truth reappears in the learned latent states, we show the warping of the ground truth bounding box into the latent space. To this end, we partitioned (discretized) the ground truth unit square into a regular 3x3 checkerboard with respective coloring. We observed that DVBF learned to extract the 2D position from the 256 pixels, and aligned them in two dimensions of the latent space in strong correspondence to the physical system. The algorithm does the exact same pixel-to-2D inference that a human observer automatically does when looking at the image.
4.3 TWO BOUNCING BALLS
Another more complex environment4 features two balls in a bounding box. We used a 10-dimensional latent space to fully capture the position and velocity information of the balls. Reconstruction and generative samples are shown in fig. 6. Same as in the pendulum example we get a generative model with stable predictions beyond training data sequence length.
5 CONCLUSION
We have proposed Deep Variational Bayes Filters (DVBF), a new method to learn state space models from raw non-Markovian sequence data. DVBFs perform latent dynamic system identification, and subsequently overcome intractable inference. As DVBFs make use of stochastic gradient variational Bayes they naturally scale to large data sets. In a series of vision-based experiments we demonstrated that latent states can be recovered which identify the underlying physical quantities. The generative model showed stable long-term predictions far beyond the sequence length used during training.
ACKNOWLEDGEMENTS
Part of this work was conducted at Chair of Robotics and Embedded Systems, Department of Informatics, Technische Universität München, Germany, and supported by the TACMAN project, EC Grant agreement no. 610967, within the FP7 framework programme.
We would like to thank Jost Tobias Springenberg, Adam Kosiorek, Moritz Münst, and anonymous reviewers for valuable input.
A SUPPLEMENTARY TO LOWER BOUND
A.1 ANNEALED KL-DIVERGENCE
We used the analytical solution of the annealed KL-divergence in eq. (10) for optimization:
Eqφ [− ln qφ(w1:T | x1:T ,u1:T ) + ci ln p(w1:T )] =
ci 1
2 ln(2πσ2p)−
1 2 ln(2πσ2q ) + ci
σ2q + (µq − µp)2
2σ2p − 1 2
B SUPPLEMENTARY TO IMPLEMENTATION
B.1 EXPERIMENTAL SETUP
In all our experiments, we use sequences of 15 raw images of the respective system with 16×16 pixels each, i.e., observation space X ⊂ R256, as well as control inputs of varying dimension and interpretation depending on the experiment. We used training, validation and test sets with 500 sequences each. Control input sequences were drawn randomly (“motor babbling”). Additional details about the implementation can be found in the published code at https://brml.org/ projects/dvbf.
B.2 ADDITIONAL EXPERIMENT PLOTS
B.3 IMPLEMENTATION DETAILS FOR DVBF IN PENDULUM EXPERIMENT
• Input: 15 timesteps of 162 observation dimensions and 1 action dimension
• Latent Space: 3 dimensions
• Observation Network p(xt|zt) = N (xt;µ(zt), σ): 128 ReLU + 162 identity output
• Recognition Model: 128 ReLU + 6 identity output
q(wt|zt,xt+1,ut) = N (wt;µ, σ), (µ, σ) = f(zt,xt+1,ut)
• Transition Network αt(zt): 16 softmax output
• Initial Network w1 ∼ p(x1:T ): Fast Dropout BiRNN with: 128 ReLU + 3 identity output
• Initial Transition z1(w1): 128 ReLU + 3 identity output
• Optimizer: adadelta, 0.1 step rate
• Inverse temperature: c0 = 0.01, updated every 250th gradient update, TA = 105 iterations
• Batch-size: 500
B.4 IMPLEMENTATION DETAILS FOR DVBF IN BOUNCING BALL EXPERIMENT
• Input: 15 timesteps of 162 observation dimensions and 2 action dimension • Latent Space: 4 dimensions • Observation Network p(xt|zt) = N (xt;µ(zt), σ): 128 ReLU + 162 identity output • Recognition Model: 128 ReLU + 8 identity output
q(wt|zt,xt+1,ut) = N (wt;µ, σ), (µ, σ) = f(zt,xt+1,ut)
• Transition Network αt(zt): 16 softmax output • Initial Network w1 ∼ p(x1:T ): Fast Dropout BiRNN with: 128 ReLU + 4 identity output • Initial Transition z1(w1): 128 ReLU + 4 identity output • Optimizer: adadelta, 0.1 step rate • Inverse temperature: c0 = 0.01, updated every 250th gradient update, TA = 105 iterations • Batch-size: 500
B.5 IMPLEMENTATION DETAILS FOR DVBF IN TWO BOUNCING BALLS EXPERIMENT
• Input: 15 timesteps of 202 observation dimensions and 2000 samples • Latent Space: 10 dimensions • Observation Network p(xt|zt) = N (xt;µ(zt), σ): 128 ReLU + 202 sigmoid output • Recognition Model: 128 ReLU + 20 identity output
q(wt|zt,xt+1,ut) = N (wt;µ, σ), (µ, σ) = f(zt,xt+1,ut)
• Transition Network αt(zt): 64 softmax output • Initial Network w1 ∼ p(x1:T ): MLP with: 128 ReLU + 10 identity output • Initial Transition z1(w1): 128 ReLU + 10 identity output • Optimizer: adam, 0.001 step rate • Inverse temperature: c0 = 0.01, updated every gradient update, TA = 2 105 iterations • Batch-size: 80
B.6 IMPLEMENTATION DETAILS FOR DKF IN PENDULUM EXPERIMENT
• Input: 15 timesteps of 162 observation dimensions and 1 action dimension • Latent Space: 3 dimensions • Observation Network p(xt|zt) = N (xt;µ(zt), σ(zt)): 128 Sigmoid + 128 Sigmoid + 2 162
identity output • Recognition Model: Fast Dropout BiRNN 128 Sigmoid + 128 Sigmoid + 3 identity output • Transition Network p(zt|zt−1,ut−1): 128 Sigmoid + 128 Sigmoid + 6 output • Optimizer: adam, 0.001 step rate • Inverse temperature: c0 = 0.01, updated every 25th gradient update, TA = 2000 iterations • Batch-size: 500

This paper presents a variational inference based method for learning nonlinear dynamical systems. Unlike the deep Kalman filter, the proposed method learns a state space model, which forces the latent state to maintain all of the information relevant to predictions, rather than leaving it implicit in the observations. Experiments show the proposed method is better able to learn meaningful representations of sequence data.
The proposed DVBF is well motivated, and for the most part the presentation is clear. The experiments show interesting results on illustrative toy examples. I think the contribution is interesting and potentially useful, so I’d recommend acceptance.
The SVAE method of Johnson et al. (2016) deserves more discussion than the two sentences devoted to it, since the method seems pretty closely related. Like the DVBF, the SVAE imposes a Markovianity assumption, and it is able to handle similar kinds of problems. From what I understand, the most important algorithmic difference is that the SVAE q network predicts potentials, whereas the DVBF q network predicts innovations. What are the tradeoffs between the two?  Section 2.2 says they do the latter in the interest of solving control-related tasks, but I’m not clear why this follows.
Is there a reason SVAEs don’t meet all the desiderata mentioned at the end of the Introduction?
Since the SVAE code is publicly available, one could probably compare against it in the experiments.
I’m a bit confused about the role of uncertainty about v. In principle, one could estimate the transition parameters by maximum likelihood (i.e. fitting a point estimate of v), but this isn’t what’s done. Instead, v is integrated out as part of the marginal likelihood, which I interpret as giving the flexibility to model different dynamics for different sequences. But if this is the case, then shouldn’t the q distribution for v depend on the data, rather than being data-independent as in Eqn. (9)?