Towards Conceptual Compression
We introduce convolutional DRAW, a homogeneous deep generative model achieving state-of-the-art performance in latent variable image modeling. The algorithm naturally stratifies information into higher and lower level details, creating abstract features and as such addressing one of the fundamentally desired properties of representation learning. Furthermore, the hierarchical ordering of its latents creates the opportunity to selectively store global information about an image, yielding a high quality 'conceptual compression' framework.
This paper introduces a new approach to unsupervised representation learning for images based on recurrent neural nets. The main innovation is to alter the cost function to encourage modeling of high-level features rather than fine details. This leads to an effective framework for lossy compression of images. 
This is obviously a high quality paper with solid technical contributions and impressive results. The main question for a paper like this one is whether it is award-worthy or not. This is a weighty question because awards function not just to highlight the best results, but also as exemplars of the highest standards in scientific research. From this point of view, I heartily support acceptance of this paper, but vote against a best paper award for reasons detailed below.   Best features of this paper: 1. Compression  To me, the focus on compression and the high level insights that emerge from this focus are the key feature of the paper. Research in unsupervised learning is often focused on perplexity/log likelihood scores to the exclusion of all else. While in principle these things are intimately related (through arithmetic coding, as the authors point out), the focus on a more operational task, lossy compression, reveals a different way of thinking about unsupervised learning. The same LL scores may, nevertheless, lead to drastically different results from the point of view of "conceptual compression". Even if this notion is only specified informally in this paper, I think it is a useful contribution to the field.   It seems to me that compression has made a comeback (or never left?) with many papers talking about factorial codes as a motivation for unsupervised learning. (As in the recent InfoGAN paper, arxiv:1606.03657, which I mention again below.) The standard way of thinking is that in an independent or disentangled basis, (lossless) compression would become trivial. While that is an attractive notion, this paper points to a different path that doesn't appear to necessarily require any explicit disentangling. The point that very little information is contained at the conceptual level, with most contained at the detail level, is intuitive but difficult to express in the factorial coding point of view. What I find most exciting here is that there appears qualitatively to be a difference between building a latent representation based on the factorial idea (infoGAN) and the "conceptual" approach here. This makes for an exciting time for unsupervised learning.   2. Results I appreciate the effort to find and report on new standardized benchmarks that are more challenging than MNIST. I found the comparisons with JPEG compression compelling. I expect future developments will require equal parts of innovation for quantifying these types of comparisons, but I think the results here were enough to communicate a new and significant contribution. As a clarification here: the innovation is to use lossy compression in conjunction with a specific type of loss function to encourage and evaluate the learning of abstract details in unsupervised representation learning. There is certainly a great deal of existing work on lossy compression for images, and I think it would be appropriate to at least cite a review on that topic.    The following critiques are, as mentioned above, geared towards justifying a high, but not top, score for this paper.   3. Reproducibility Compared to some disciplines, you could say that the work presented here is highly reproducible. On the other hand, computer science can be held to a higher standard and I believe it should be. I did not see any mention of code availability (even if the link would have to be omitted for anonymity). For comparison, a contemporaneous paper, the infoGAN paper, provides documented code with running examples that reproduce paper figures. Without any expertise implementing deep nets, I was able to get the code from that paper running in a few minutes. The implementation consisted of a few hundred lines of code, the main ingredients of which I could easily recognize. If, on the other hand, I had to reproduce this code based on the description in the paper, it would take, at best, months of work and, at worst, I would find that there were relevant details about initialization or hyper-parameters that were essential but incompletely described. For the long-term vitality of research in this field, I believe code availability is an essential consideration for top research, even if it is not (yet) a requirement for publishing.   4. Concepts I appreciated the overall framing of the paper and got a lot out of it. However, I am still dubious about the suggestion that the top latent factors specify a concept like "cat" or "dog", while the subsequent factors fill in detail. It looks to me more like the top factors specify "light, orange, oriented smudge", and the subsequent layers specify details that make it recognizable as a cat or a dog later. I would be surprised if there were consistently a "cat" neuron that appears in this framework, for instance.  The flip side of the "concept" framing is that this model "learns to fill in detail later". It seems to me that the inclusion of convolutional structure is doing the heavy lifting here, as it does in most recent work, and is not an intrinsic advantage of the framework. I guess the test of this would be to see if this assertion continues to hold on non-image data. Regardless, the results are very interesting and further speculation and research on how and what is represented in this framework are warranted.   p.s. The authors might want to check out this paper, http://arxiv.org/pdf/1503.03585.pdf. In particular they discuss various multi-scale approaches and the "dead leaf" images are a good test of this. It might be nice to nod in this direction and clarify how your work differs from these efforts to model images at multiple scales.    edit: Based on the other reviews and the responses, it seems that more technical detail were omitted than I understood. I'm lowering my confidence and clarity score, but still argue for acceptance.