There have been multiple attempts with variational auto-encoders (VAE) to learn powerful global representations of complex data using a combination of latent stochastic variables and an autoregressive model over the dimensions of the data. However, for the most challenging natural image tasks the purely autoregressive model with stochastic variables still outperform the combined stochasticautoregressive models. In this paper, we present simple additions to the VAE framework that generalize to natural images by embedding spatial information in the stochastic layers. We significantly improve the state-of-the-art results on MNIST, OMNIGLOT, CIFAR10 and ImageNet when the feature map parameterization of the stochastic variables are combined with the autoregressive PixelCNN approach. Interestingly, we also observe close to state-of-the-art results without the autoregressive part. This opens the possibility for high quality image generation with only one forward-pass.
The paper combines several recent advances on generative modelling including a ladder variational posterior and a PixelCNN decoder together with the proposed convolutional stochastic layers to boost the NLL results of the current VAEs. The numbers in the tables are good but I have several comments on the motivation, originality and experiments.
Most parts of the paper provide a detailed review of the literature. However, the resulting model is quite like a combination of the existing advances and the main contribution of the paper, i.e. the convolution stochastic layer, is not well discussed. Why should we introduce the convolution stochastic layers? Could the layers encode the spatial information better than a deterministic convolutional layer with the same architecture? What's the exact challenge of training VAEs addressed by the convolution stochastic layer? Please strengthen the motivation and originality of the paper.
Though the results are good, I still wonder what is the exact contribution of the convolutional stochastic layers to the NLL results?  Can the authors provide some results without the ladder variational posterior and the PixelCNN decoder on both the gray-scaled and the natural images?
According to the experimental setting in the Section 3 (Page 5 Paragraph 2), "In case of gray-scaled images the stochastic latent layers are dense with sizes 64, 32, 16, 8, 4 (equivalent to Sønderby et al. (2016)) and for the natural images they are spatial (cf. Table 1). There was no significant difference when using feature maps (as compared to dense layers) for modelling gray-scaled images." there is no stochastic convolutional layer.  Then is there anything new in FAME on the gray images? Furthermore, how could FAME advance the previous state-of-the-art? It seems because of other factors instead of the stochastic convolutional layer. 
The results on the natural images are not complete. Please present the generation results on the ImageNet dataset and the reconstruction results on both the CIFAR10 and ImageNet datasets. The quality of the samples on the CIFAR10 dataset seems not competitive to the baseline papers listed in the table. Though the visual quality does not necessarily agree with the NLL results but such large gap is still strange. Besides, why FAME can obtain both good NLL and generation results on the MNIST and OMNIGLOT datasets when there is no stochastic convolutional layer? Meanwhile, why FAME cannot obtain good generation results on the CIFAR10 dataset? Is it because there is a lot randomness in the stochastic convolutional layer? It is better to provide further analysis and it is not safe to say that the stochastic convolutional layer helps learn better latent representations based on only the NNL results.
Minor things:
Please rewrite the sentence "When performing reconstructions during training ... while also using the stochastic latent variables z = z 1 , ..., z L." in the caption of Figure 1.