IntroVAE: Introspective Variational Autoencoders for Photographic Image Synthesis
We present a novel introspective variational autoencoder (IntroVAE) model for synthesizing high-resolution photographic images. IntroVAE is capable of selfevaluating the quality of its generated samples and improving itself accordingly. Its inference and generator models are jointly trained in an introspective way. On one hand, the generator is required to reconstruct the input images from the noisy outputs of the inference model as normal VAEs. On the other hand, the inference model is encouraged to classify between the generated and real samples while the generator tries to fool it as GANs. These two famous generative frameworks are integrated in a simple yet efficient single-stream architecture that can be trained in a single stage. IntroVAE preserves the advantages of VAEs, such as stable training and nice latent manifold. Unlike most other hybrid models of VAEs and GANs, IntroVAE requires no extra discriminators, because the inference model itself serves as a discriminator to distinguish between the generated and real samples. Experiments demonstrate that our method produces high-resolution photo-realistic images (e.g., CELEBA images at 1024 2 ), which are comparable to or better than the state-of-the-art GANs.
Update:  I raised my score by two points because the rebuttal and reviews/comments revealed more differences that I originally noticed with respect to the AGE work, in particular in terms of the use of the KL divergence as a discriminator per example, and because the authors promised to discuss the connection to AGE and potentially expand the experimental section.  I remain concerned that the resulting model is not a variational auto-encoder anymore despite the naming of the model (but rather closer to a GAN where the discriminator is based on the KL divergence), and about the experimental section, which reveals that the method works well, but does not provide a rich analysis for the proposed improvements.  ---  The paper proposes an approach for adversarial learning in the context of variational auto-encoders. Rather than using a separate discriminator network, the work proposes a learning objective which encourages the encoder to discriminate between real data and generated data: it guides the approximate posterior to be close to the prior in the real data case and far from the prior otherwise. The approach is illustrated on the task of synthesizing high-resolution images, trained on the CelebA-HQ dataset.  First, high-quality image generation remains an important area of research, and as a result, the paper's topic is relevant to the community. Furthermore, the results are very good, comparable to state of the art, despite not using a discriminator network.  However, I believe the paper does not position itself strongly with respect to the cited work, AGE [34]. The authors highlight the following differences: "design of inference models, the training objects, and the divergence computations," but without detail. To me, the approaches seem quite similar: in both cases, the objective relies on decreasing the divergence between the posterior and the prior for real data, and increasing it for generated data; in both cases the chosen divergence measure is a KL divergence, and the reference (prior) distribution is a Gaussian; etc.. There are slight differences such as the use of an upper bound (m) on the divergence between the generated data posterior and the prior, but a discussion of why they might be needed is missing. Could you discuss this in your rebuttal?  A major point made in the write-up is that previous methods based on encoder-decoder architectures (without discriminators), including AGE, have obtained weaker results and that the submission is the first to generate high-resolution images. This result is promising, but I think it should come with an understanding of the differences to the previous works, highlighting which modifications have caused the improvement. Otherwise, the improvement could be attributed to better hyper-parameter tuning. Instead, an empirical comparison with the cited work, AGE, is not provided at all.  Other comments: * In lines 127-132, the use of E(x) to denote the KL divergence loss, as well as referring to E as the inference (encoder) model creates confusion. Same with E* later.  Summary: The current work presents good image synthesis results for an adversarially trained VAE architecture with no discriminator. However, the approach appears very similar to previous work, making it less relevant for the NIPS community.