A new variational autoencoder (VAE) model is proposed that learns a succinct common representation of two correlated data variables for conditional and joint generation tasks. The proposed Wyner VAE model is based on two information theoretic problems-distributed simulation and channel synthesis-in which Wyner's common information arises as the fundamental limit of the succinctness of the common representation. The Wyner VAE decomposes a pair of correlated data variables into their common representation (e.g., a shared concept) and local representations that capture the remaining randomness (e.g., texture and style) in respective data variables by imposing the mutual information between the data variables and the common representation as a regularization term. The utility of the proposed approach is demonstrated through experiments for joint and conditional generation with and without style control using synthetic data and real images. Experimental results show that learning a succinct common representation achieves better generative performance and that the proposed model outperforms existing VAE variants and the variational information bottleneck method.
shared common randomness. (See Fig. 1 (c,d).) In this sense, the joint distribution q(x, y) and the conditional distributions q(y|x), q(x|y) have the same common information structure characterized by the optimization problem (1), which involves learning the joint distribution in its nature. We call the joint encoder q(z|x, y) (or equivalently, the corresponding random variable Z) as the common representation of (X, Y), and the mutual information I q (X, Y; Z) then can be viewed as a measure of the complexity of Z.
The goal of this paper is to propose a new probabilistic model based on these information theoretic observations, to achieve a good performance in joint and conditional generation tasks. We apply the idea of learning succinct common representation to design a new generative model for a pair of correlated variables: seeking a succinct representation Z in learning the underlying distribution based on its sample may also help reduce the burden on the decoder's side and thereby achieve a better generative performance.
The rest of the paper gradually develops our framework as follows. We first define a probabilistic model based on the motivating problems, which we aim to train and use for generation tasks (Section 2.1), and then establish a general principle for learning the model based on the optimization problem (1) (Section 2.2). We propose one instantiation of the principle with a standard variational technique by introducing additional encoder distributions (Section 2.3). The proposed model with its training method can be viewed as a variant of variational autoencoders (VAEs) Rezende et al., 2014), and is thus called Wyner VAE. (See Appendix A for a brief introduction on VAEs.) The new encoder components introduced in Wyner VAE allow us to decompose a data vector into the common representation and the local representation, which can be used for sampling with style manipulation (Section 2.4). We carefully compare our model and show its advantages over the existing VAE variants (Vedantam et al., 2018;Suzuki et al., 2016;Sohn et al., 2015;Wang et al., 2016) and the information bottleneck (IB) principle (Tishby et al., 1999) (Section 3), which is a well-known information theoretic principle in representation learning. In the experiments, we empirically show the utility of our model in various sampling tasks and its superiority over existing models and that learning a succinct common representation achieves better generative performance in generation tasks (Section 4).
This paper presents a method for learning latent-variable models of paired data that decomposes the latent representation into common and local representations. The approach is motivated by Wyner’s common information, and is made tractable through a variational formulation. On experiments with MoG, MNIST, SVHN, and CelebA, the paper shows that penalizing the complexity of the common representation can improve style-content disentanglement and conditional sampling.
While I found the formulation and motivation from Wyner’s common information and Cuff’s channel synthesis interesting, the resulting model and experiments were unconvincing. There are many terms in the Wyner model proposed, and there are no ablations demonstrating which terms are important and which are not (e.g. do you need both L_xx and L_xy?). On the toy MoG experiment, the common information is known but trained models do not recover the right amount of information. For representation learning, accuracy decreases as the penalty on the common information increases, and for NLL, the joint and conditional NLL is often similar to existing work (CVAE and JVAE). The main win appears to be for style-content disentanglement, but the results there are qualitative and often change the content when only style is changed. It’s also puzzling as to why CVAE overfits so severely in a subset of the experiments. Without a more thorough evaluation of what terms in the loss matter, and showing that the technique recovers something like Wyner’s common information (by extracting the right information on a toy model), I cannot recommend this paper for acceptance.
Minor comments:
* Eqn 1: the constraint “Subject to X-Z-Y” is confusing. Do you mean X-Z-Y is a Markov network (undirected) and not a Markov chain (directed), in which case X -> Z <- Y does not correspond to X-Z-Y? (This is addressed in Eqn 3-4, but should be fixed here)
* How is Wyner’s common information related to the multivaraite mutual information I(X; Z; Y)?
* When describing distributed simulation, please include the variables and objective like you do for common information
* s/Markovity condition/independence assumption?
* Why are both losses in Eqn 4 and Eqn 5 needed? Couldn’t you use either to train q(z|x)?
* Eqn 10 (and most of your objectives) are still intractable due to the q(x)/q(x,y) terms, you should note that it is constant and dropped from the objective
* “Style control”: could you define what you mean by style in this context? Prior work could likely also do “style control” by e.g. interpolating subsets of dimensions.
* Related work: https://openreview.net/forum?id=rkVOXhAqY7 (CEB) that may result in a similar objective as Wyner’s common info
* When comparing to JVAE/JMVAE, it seems like the main difference is suing a latent-variable in the decoder, but the framework is still the same. It’d be useful to spend more time comparing/contrasting with this prior work.
* Fig 4: would be useful to have a picture of the samples (right now they’re just in appendix)
* Fig 4: in the data generating process, I believe I((X, Y); Z) = ln(5) = 1.6 nats, but none of your models converge to rates around there. Why not?
* Unlike other approaches (JVAE, CVAE) you have additional terms in your loss for conditional prediction at training time. It seems like these may be giving you gains, and it’d be useful to perform ablations over the terms in Eqn 15 to figure out what is impacting performance in Fig 4. E.g. if you set \alpha_{x -> y} and \alpha{y -> x} to 0
* Why do the CVAE models overfit? Have you tried optimizing the variational parameters on the test set if it’s due to amortization gap?
* Fig 5b: what’s the variance you’re plotting? In representation learning, we often just care about downstream accuracy, and it looks like \lam = 0  performs best there.
* Table 3: isn’t this showing that \lam = 0 works the best for joint NLL and close to best for conditional NLL on the MoG task? What’s the discrepancy with Fig 5 where conditional NLL is highly dependent on \lambda for MNIST?
* Fig 6 (a1-f1): for this MNIST add 1 dataset, the only common info should be the label. But it looks like the labels do change for non-zero values of lambda (b1 there’s a 2 -> 7), c1 1 -> 9)
* Table 4: would be useful to train CVAE yourself, as the small differences in numbers could just be due to tuning / experimental setup. You also should bold everything that’s within stderr (i.e. \lambda = 0 and \lambda = 0.15 are equally good)
* Fig 9: would be useful to include CVAE, and Wyner VAE in this plot, i.e. is it only using the mode information in the latent variable when doing conditional sampling?
* CelebA results look interesting, but there’s been other work on generation from attributes that presents more visually compelling results, e.g. https://arxiv.org/abs/1706.00409