Variational Auto-Encoders (VAEs) are designed to capture compressible information about a dataset. As a consequence the information stored in the latent space is seldom sufficient to reconstruct a particular image. To help understand the type of information stored in the latent space we train a GAN-style decoder constrained to produce images that the VAE encoder will map to the same region of latent space. This allows us to "imagine" the information captured in the latent space. We argue that this is necessary to make a VAE into a truly generative model. We use our GAN to visualise the latent space of a standard VAE and of a β-VAE.
Summary:
This paper proposes a hybrid VAE-GAN model, called the latent space renderer-GAN (LSR-GAN), with the goal to “imagine” the latent space of a VAE, and to improve the decoding and sampling quality of a VAE. First, a VAE-like model is trained, after which the encoder weights are frozen, and the decoder is trained as the generator of a GAN (together with an auxiliary discriminator). The generator loss also contains a reconstruction-like term in the latent space, described by the negative log density of the encoding distribution of the original latent conditioned on the output of the generator: -log q(z|g(z)). 
Decision: reject
This paper contains incorrect claims. Leaving those mistakes aside, the experiments don’t lead to new insights and no comparison against other VAE-GAN hybrids is made. 
Supporting arguments for decision:
In the introduction the authors state “This is a consequence of the well known evidence lower bound or ELBO objective function consisting of a negative log-probability of generating the original image from the latent representation (this is often implemented as a mean squared error between the image and the reconstruction, although as we argue in Appendix A this term should be proportional to the logarithm of the mean squared error)...”. This statement is surprising and I don’t see how it can be correct. If -log p(x|z) is something like the log of the mean squared error, then the density p(x|z) should be a squared error function, which is not even a valid distribution.
To be more precise, according to the authors, if one takes a Gaussian p(x|z), the reconstruction error should be modeled with  
-log p(x|z) = log 1/N sum (x_i - \hat x_i)^2  + const 					(1)
where \hat x_i is a function of z. Note the logarithm on the right-hand side here. In appendix A, the authors observe that most other implementations instead optimize 
-log p(x|z) = sum_{i=1}^N (x_i -\hat x_i)^2/(2\sigma^2) + N/2 log(2 pi sigma^2) 		(2)
Where often sigma is set to ½ (so that the last term on the lhs of (2) drops out when gradients are taken). The authors claim the latter is incorrect because sigma should be equal to the (biased) empirical variance. They then insert sigma=empirical variance into (2), take derivatives while ignoring the dependence of sigma on \hat x_i, and then arrive at something like eq. 1. They also argue that those implementations that use sigma=½ are actually optimizing a beta-VAE because of this “incorrect” prefactor of the reconstruction error. 
I’m confident that the above claims are not correct. 
As an example, using a Gaussian decoder distribution requires parameterizing the mean and variance of that Gaussian distribution as a function of z. Here, setting sigma = ½ (as is commonly done in other literature) is valid, contrary to the above claim. One is always allowed to just set the variance to a constant and ignore its dependence on z. This just leaves you with a less flexible distribution to model p(x|z).
The authors appear to use (1) as the reconstruction error term in the ELBO during VAE optimization, and then play with different prefactors of the KL term similar to beta-VAEs. The combination of the reconstruction error (1) and the prefactor no longer makes this a VAE, but more like a regularized auto-encoder with an unconventional reconstruction error that should not be interpreted as coming from the negative of a log density. Due to the logarithm in front of the mean squared error, the loss function is less sensitive to large errors in reconstruction. It is surprising that the reconstruction error does not cause overflow as log (0) --> - infinity. 
Comments on experiments and related work: 
- Experiments only show images of reconstructions of a VAE and LSR-GAN, a mean squared error plot of reconstruction errors and an accuracy plot of a classifier evaluated on the reconstructions produced by both models. In terms of MSE the LSR-GAN actually performs worse than their VAE baseline. The authors also train a classifier on cifar-10 and measure its accuracy using the ground truth labels and the reconstructed images of a VAE and LSR-GAN. Here the LSR-GAN performs marginally better than the VAE, but the overall accuracy is very poor.
- No experimental comparison is made against other VAE-GAN hybrids. Related work on hybrid VAE-GANS is discussed in the appendix, not in the main paper.
Other sections:
- Section 3 contains a very long interpretation of the minimum description length (MDL). It is unclear what the goal of this section is, as it does not lead to any results, nor does it help bring the point across as to why the proposed model is good at imagining the latent space of a VAE.
Additional feedback to improve the paper (not part of decision assessment):
The additional loss term of the generator is very similar to what is used in reweighted wake sleep, although this is not mentioned. It could be worth making a connection here. 
In the conclusion the authors state “VAEs are often taken to be a pauper’s GAN’. This is not a very scientific statement and can be perceived as insulting for various reasons. Please rephrase this.