We present an information-theoretic framework for understanding trade-offs in unsupervised learning of deep latent-variables models using variational inference. This framework emphasizes the need to consider latent-variable models along two dimensions: the ability to reconstruct inputs (distortion) and the communication cost (rate). We derive the optimal frontier of generative models in the two-dimensional rate-distortion plane, and show how the standard evidence lower bound objective is insufficient to select between points along this frontier. However, by performing targeted optimization to learn generative models with different rates, we are able to learn many models that can achieve similar generative performance but make vastly different trade-offs in terms of the usage of the latent variable. Through experiments on MNIST and Omniglot with a variety of architectures, we show how our framework sheds light on many recent proposed extensions to the variational autoencoder family.
EDIT:  I have reviewed the authors revisions and still recommend acceptance. 
Summary
This paper proposes assessing VAEs via two quantities: rate R (E[ KLD[q(z|x) || p(z)] ]) and distortion D (E[ log p(x|z) ]), which can be used to bound the mutual information (MI) I(x,z) from above and below respectively (i.e. H[x] - D <= I(x,z) <= R).  This fact then implies the inequality H[x] <= R + D, where H[x] is the entropy of the true data distribution, and allows for the construction of a phase diagram (Figure 1) with R and D on the x and y axis respectively.  Models can be plotted on the diagram to show the degree to which they favor reconstruction (D) or sampling diversity (R).  The paper then reports several experiments, the first being a simulation to show that a VAE trained with vanilla ELBO cannot recover the true rate in even a 1D example.  For the second experiment, 12 models are trained by varying the encoder/decoder strength (CNN vs autoregressive) and prior (fact. Gauss vs autoregressive vs VampPrior).  Plots of the D vs R and ELBO vs R are shown for the models, revealing that the same ELBO value can be decomposed into drastically different R and D values.  The point is further made through qualitative results in Figure 4.   
Evaluation
Pros:  While no one facet of the paper is particularly novel (as similar observations and discussion has been made by [1-4]), the paper, as far as I’m aware, is the first to formally decompose the ELBO into the R vs D tradeoff, which is natural.  As someone who works with VAEs, I didn’t find the conclusions surprising, but I imagine the paper would be valuable to someone learning about VAEs.  Moreover, it’s nice to have a clear reference for the unutilized-latent-space-behavior mentioned in various other VAE papers.  The most impressive aspect of the paper is the number of models trained for the empirical investigation.  Placing such varied models (CNN vs autoregressive vs VampPrior etc) onto the same plot from comparison (Figure 3) is a valuable contribution.       
Cons:  As mentioned above, I didn’t find the paper conceptually novel, but this isn’t a significant detraction as its value (at least for VAE researchers) is primarily in the experiments (Figure 3).  I do think the paper---as the ‘Discussion and Further Work’ section is only two sentences long---could be improved by providing a better summary of the findings and recommendations moving forward.  Should generative modeling papers be reporting final R and D values in addition to marginal likelihood?  How should an author demonstrate that their method isn’t doing auto-decoding?  The conclusion claims that “[The rate-distortion tradeoff] provides new methods for training VAE-type models which can hopefully advance the state of the art in unsupervised representation learning.”  Is this referring to the constrained optimization problem given in Equation #4?  It seems to me that the optimal R-vs-D tradeoff is application dependant; is this not always true?      
Miscellaneous / minor comments:  Figure 3 would be easier to read if the dots better reflected their corresponding tuple (although I realize representing all tuple combinations in terms of color, shape, etc is hard).  I had to keep referring to the legend, losing my place in the scatter plot.  I found sections 1 and 2 rather verbose; I think some text could be cut to make room for more final discussion / recommendations.  For example, I think the first two whole paragraphs could be cut or at least condensed and moved to the related works section, as they are just summarizing research history/trends.  The paper’s purpose clearly starts at the 3rd paragraph (“We are interested in understanding…”).  The references need cleaned-up.  There are several conference publications that are cited via ArXiv instead of the conference (IWAE should be ICLR, Bowman et al. should be CoNLL, Lossy VAE should be ICLR, Stick-Breaking VAE should be ICLR, ADAM should be ICLR, Inv Autoregressive flow should be NIPS, Normalizing Flows should be ICML, etc.), and two different versions of the VAE paper are cited (ArXiv and ICLR).  
Conclusions
I found this paper to present valuable analysis of the ELBO objective and how it relates to representation learning in VAEs.  I recommend the paper be accepted, although it could be substantially improved by including more discussion at the end.  
1.  S. Zhao, J. Song, and S. Ermon.  “InfoVAE: Information Maximizing Variational Autoencoders.”  ArXiv 2017.
2.  X. Chen, D. Kingma, T. Salimans, Y. Duan, P. Dhariwal, J. Shulman, I. Sutskever, and P. Abbeel.  “Variational Lossy Autoencoder.”  ICLR 2017.
3.  I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner. “Beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework.”  ICLR 2017
4.  S. Bowman, L. Vilnis, O. Vinyas,  A. Dai, R. Jozefowicz, and S. Bengio.  “Generating Sentences from a Continuous Space.”  CoNLL 2016.