We present a neural network architecture based upon the Autoencoder (AE) and Generative Adversarial Network (GAN) that promotes a convex latent distribution by training adversarially on latent space interpolations. By using an AE as both the generator and discriminator of a GAN, we pass a pixel-wise error function across the discriminator, yielding an AE which produces sharp samples that match both high-and low-level features of the original images.
Update:
I’d like to thank the authors for their thoroughness in responding to the issues I raised. I will echo my fellow reviewers in saying that I would encourage the authors to submit to another venue, given the substantial modifications made to the original submission.
The updated version provides a clearer context for the proposed approach (phychophysical experimentation) and avoids mischaracterizing GAIA as a generative model.
Despite more emphasis being put on mentioning the existence of bidirectional variants of GANs, I still feel that the paper does not adequately address the following question: “What does GAIA offer that is not already achievable by models such as ALI, BiGAN, ALICE, and IAE, which equip GANs with an inference mechanism and can be used to perform interpolations between data points and produce sharp interpolates?” To be clear, I do think that the above models are inadequate for the paper’s intended use (because their reconstructions tend to be semantically similar but noticeably different perceptually), but I believe this is a question that is likely to be raised by many readers.
To answer the authors’ questions:
- Flow-based generative models such as RealNVP relate to gaussian latent spaces in that they learn to map from the data distribution to a simple base distribution (usually a Gaussian distribution) in a way that is invertible (and which makes the computation of the Jacobian’s determinant tractable). The base distribution can be seen as a Gaussian latent space which has the same dimensionality as the data space.
- Papers on building more flexible approximate posteriors in VAEs: in addition to the inverse autoregressive flow paper already cited in the submission, I would point the authors to Rezende and Mohamed’s “Variational Inference with Normalizing Flows”, Huang et al.’s “Neural Autoregressive Flows”, and van den Berg et al.’s “Sylvester Normalizing Flows for Variational Inference”.
-----
The paper title summarizes the main claim of the paper: "adversarial training on latent space interpolations encourage[s] convex latent distributions". A convex latent space is defined as a space in which a linear interpolation between latent codes obtained by encoding a pair of points from some data distribution yields latent codes whose decoding also belongs to the same data distribution. The authors argue that current leading approaches fall short of producing convex latent spaces while preserving the "high-dimensional structure of the original distribution". They propose a GAN-AE hybrid, called GAIA, which they claim addresses this issue. The proposed approach turns the GAN generator and discriminator into autoencoders, and the adversarial game is framed in terms of minimizing/maximizing the discriminator’s reconstruction error. In addition to that, interpolations between pairs of data points are computed in the generator’s latent space, and the interpolations are decoded and treated as generator samples. A regularization term is introduced to encourage distances between pairs of data points to be mirrored by their representation in the generator’s latent space. The proposed approach is evaluated through qualitative inspection of latent space interpolations, attribute manipulations, attribute vectors, and generator reconstructions.
Overall I feel like the problem presented in the paper is well-justified, but the paper itself does not build a sufficiently strong argument in favor of the proposed approach for me to recommend its acceptance. I do think there is a case to be made for a model which exhibits sharp reconstructions and which allows realistic latent space manipulations -- and this is in some ways put forward in the introduction -- but I don’t feel that the way in which the paper is currently cast highlights this very well. Here is a detailed breakdown of why, and where I think it should be improved, roughly ordered by importance:
- The main reason for my reluctance to accept the paper is the fact that its main subject is convex latent spaces, yet I don’t see that reflected in the evaluation. The authors do not address how to evaluate (quantitatively or qualitatively) whether a certain model exhibits a convex latent space, and how to compare competing approaches with respect to latent space convexity. Figure 2 does present latent space interpolations which help get a sense of the extent to which interpolates also belong to the data distribution, however in the absence of a comparison to competing approaches it’s impossible for me to tell whether the proposed approach yields more convex latent spaces.
- I don’t agree with the premise that current approaches are insufficient. The authors claim that autoencoders produce blurry reconstructions; while this may be true for factorized decoders, autoregressive decoders should alleviate this issue. They also claim that GANs lack bidirectionality but fail to mention the existing line of work in that direction (ALI, BiGAN, ALICE, and more recently Implicit Autoencoders). Finally, although flow-based generative models are mentioned later in the paper, they are not discussed in Section 1.2 when potential approaches to building convex latent spaces are enumerated and declared insufficient. As a result, the paper feels a little disconnected from the current state of the generative modeling literature.
- The necessity for latent spaces to "respect the high-dimensional structure of the [data] distribution" is stated as a fact but not well-justified. How do we determine whether a marginal posterior is "a suboptimal representation of the high-dimensional dataset"? I think a more nuanced statement would be necessary. For instance, many recent approaches have been proposed to build more flexible approximate posteriors in VAEs; would that go some way towards embedding the data distribution in a more natural way?
- I also question whether latent space convexity is a property that should always hold. In the case of face images a reasonable argument can be made, but in a dataset such as CIFAR10 how should we linearly interpolate between a horse and a car?
- The proposed model is presented in the abstract as an "AE which produces non-blurry samples", but it’s not clear to me how one would sample from such a model. The generator is defined as a mapping from data points to their reconstruction; does this mean that the sampling procedure requires access to training examples? Alternatively one could fit a prior distribution on top of the latent codes and their interpolations, but as far as I can tell this is not discussed in the paper. I would like to see a more thorough discussion on the subject.
- When comparing reconstructions with competing approaches there are several confounding factors, like the resolution at which the models were trained and the fact that they all reconstruct different inputs. Removing those confounding factors by comparing models trained at the same resolution and reconstructing the same inputs would help a great deal in comparing each approach.
- The structure-preserving regularization term compares distances in X and Z space, but I doubt that pixelwise Euclidian distances are good at capturing an intuitive notion of distance: for example, if we translate an image by a few pixels the result is perceptually very similar but its Euclidian distance to the original image is likely to be high. As far as I can tell, the paper does not present evidence backing up the claim that the regularization term does indeed preserve local structure.
- Figures 2 and 3 are never referenced in the main text, and I am left to draw my own conclusions as to what claim they are supporting. As far as I can tell they showcase the general capabilities of the proposed approach, but I would have liked to see a discussion of whether and how they improve on results that can be achieved by competing approaches.
- The decision of making the discriminator an autoencoder is briefly justified when discussing related work; I would have liked to see a more upfront and explicit justification when first introducing the model architecture.
- When discussing feature vectors it would be appropriate to also mention Tom White’s paper on Sampling Generative Networks.