Generative modeling of high-dimensional data is a key problem in machine learning. Successful approaches include latent variable models and autoregressive models. The complementary strengths of these approaches, to model global and local image statistics respectively, suggest hybrid models combining the strengths of both models. Our contribution is to train such hybrid models using an auxiliary loss function that controls which information is captured by the latent variables and what is left to the autoregressive decoder. In contrast, prior work on such hybrid models needed to limit the capacity of the autoregressive decoder to prevent degenerate models that ignore the latent variables and only rely on autoregressive modeling. Our approach results in models with meaningful latent variable representations, and which rely on powerful autoregressive decoders to model image details. Our model generates qualitatively convincing samples, and yields stateof-the-art quantitative results.
Summary:
This paper attempts to solve the problem of meaningfully combining variational autoencoders (VAEs) and PixelCNNs. It proposes to do this by simultaneously optimizing a VAE with PixelCNN++ decoder, and a VAE with factorial decoder. The model is evaluated in terms of log-likelihood (with no improvement over a PixelCNN++) and the visual appearance of samples and reconstructions.
Review:
Combining density networks (like VAEs) and autoregressive models is an unsolved problem and potentially very useful. To me, the most interesting bit of information in this paper was the realization that you can weight the reconstruction and KL terms of a VAE and interpret it as variational inference in a generative model with multiple copies of pixels (below Equation 7). Unfortunately the authors were unable to make any good use of this insight, and I will explain below why I don’t see any evidence of an improved generative model in this paper.
As the paper is written now, it is not clear what the goal of the authors is. Is it density estimation? Then the addition of the VAE had no measurable effect on the PixelCNN++’s performance, i.e., it seems like a bad idea due to the added complexity and loss of tractability. Is it representation learning? Then the paper is missing experiments to support the idea that the learned representations are in any way an improvement. Is it image synthesis (not a real application by itself), then the paper should have demonstrated the usefulness of the model on a real task and probably involve human subjects in a quantitative evaluation.
Much of the authors’ analysis is based on a qualitative evaluation of samples. However, samples can be very misleading. A lookup table storing the training data generates samples containing objects and perfect details, but obviously has not learned anything about either objects or the low-level statistics of natural images. 
In contrast to the authors, I fail to see a meaningful difference between the groups of samples in Figure 1.
The VAE samples in Figure 3b) look quite smooth. Was independent Gaussian noise added to the VAE samples or are those (as is sometimes done) sampled means? If the former, what was sigma and how was it chosen?
On page 7, the authors conclude that “the pixelCNN clearly takes into account the output of the VAE decoder” based on the samples. Being a mixture model, a PixelCNN++ could easily represent the following mixture:
p(x | z) = 0.01 \prod_i p(x_i | x_{<i}) + 0.99 \prod_i p(x_i | z)
The first term is just like a regular PixelCNN++, ignoring the latent variables. The second term is just like a variational autoencoder with factorial decoder. The samples in this case would be dominated by the VAE, which depends on the latent state. The log-likelihood would be dominated by the first term and would be minimally effected (see Theis et al., 2016). Note that I am not saying that this is exactly what the model has learned. I am merely providing a possible counter example to the notion that the PixelCNN++ has learned to use of the latent representation in a meaningful way.
What happens if the KL term is simply downweighted but the factorial decoder is not included? This seems like it would be a useful control to include.
The paper is well written and clear.