Deep generative models have achieved remarkable progress in recent years. Despite this progress, quantitative evaluation and comparison of generative models remains as one of the important challenges. One of the most popular metrics for evaluating generative models is the log-likelihood. While the direct computation of log-likelihood can be intractable, it has been recently shown that the loglikelihood of some of the most interesting generative models such as variational autoencoders (VAE) or generative adversarial networks (GAN) can be efficiently estimated using annealed importance sampling (AIS). In this work, we argue that the log-likelihood metric by itself cannot represent all the different performance characteristics of generative models, and propose to use rate distortion curves to evaluate and compare deep generative models. We show that we can approximate the entire rate distortion curve using one single run of AIS for roughly the same computational cost as a single log-likelihood estimate. We evaluate lossy compression rates of different deep generative models such as VAEs, GANs (and its variants) and adversarial autoencoders (AAE) on MNIST and CIFAR10, and arrive at a number of insights not obtainable from log-likelihoods alone.
This paper presents a method for evaluating latent-variable generative models in terms of the rate-distortion curve that compares the number of bits needed to encode the representation with how well you can reconstruct an input under some distortion measure. To estimate this curve, the author’s use AIS and show how intermediate distributions in AIS can be used to bound and estimate rate and distortion. They apply their evaluation to GANs, VAEs, and AAEs trained on MNIST and CIFAR-10.
I found this paper well written, with a number of interesting technical contributions, particularly how to leverage AIS to compute R-D curves for an individual model. However, the utility and interpretation of these R-D curves for single models remains confusing to me, and there is insufficient discussion and comparison to other joint diversity/sample quality metrics proposed in the GAN literature. The compute time required for evaluation may also limit the applicability: 4-7 hours for 50 images on MNIST, and 7 days for CIFAR-10. 
 
Major comments:
* How should we interpret rate-prior distortion for an individual model vs. rate-distortion where models are optimized for each rate? Past work in learned compression and generative models (Theis et al. 2016, Balle et al. 2016, Alemi et al., 2018) show that models must adjust their decoder (and prior) as a function of rate to be optimal in terms of distortion. For a fixed decoder, optimizing the prior may still be required to achieve low rate. Given that many of the models you compare are trained to do well at one point on the R-D curve, why does it make sense to evaluate them at other points? Additionally, you only evaluate models with deterministic decoders and many of the experimental conclusions are highly specific to this setting but not noted. 
* As you focus on general distortion metrics instead of NLL alone, it'd be interesting to compare curves under different distortion measures, e.g. MS-SSIM for images or L1 vs. L2. Right now there's not much experimental novelty vs. prior work that looked at rate-distortion curves with NLL distortion and Gaussian observation models.
* It’d be useful to include experiments comparing Rate-Prior distortion curves and Rate-distortion curves where you a) optimize over the prior, b) optimize the decoder, fixing the prior, and c) optimize both the prior and decoder. 
* There’s no comparison of other approaches to generate the rate-prior distortion curve. For example, you could just use an amortized inference network like in VAE w/ a flexible variational family and anneal beta over time.
* There are several related papers which should be discussed and contrasted, in particular https://arxiv.org/abs/1901.07821 which looks at rate-distortion-perception tradeoffs, and https://arxiv.org/abs/1806.00035 which presents precision-recall curves for diversity/quality metrics applied to implicit models. How do the insights gained from the rate-distortion curves relate to precision/recall and why should one be preferred over the other? https://arxiv.org/abs/1611.02163 also looked at distortion as a metric for GANs (equivalent to beta -> infinity in your framework).
Minor comments:
* Missing discussion of several related works: that presents a complexity measure for the latent space of GANs: https://arxiv.org/abs/1802.04874
* “Wasserstein distance remains difficult to approximate…” - see https://openreview.net/forum?id=HkxKH2AcFm that advocates for evaluation with Wasserstein
* Tightness of bound on simulated data (what BDMC provides) may not correspond to tightness of bound on real data (what you care about in practice). 
* The treatment of VAEs as implicit models only makes sense with location scale family p(x|z), thus the entire framework proposed here doesn’t make sense with e.g. autoregressive p(x|z), as used in PixelVAE and others.
* Why focus on fixed prior p(z)? An alternative would be to optimize p(z), q(z|x) and fix p(x|z). How would this change the resulting rate-prior distortion curves?
* “We can compute the R-D curve by sweeping over \beta rather than by sweeping over D” - this is not the case when the R-D curve has linear segments, see e.g. Rezende & Viola 2018
* Many of the properties and discussion around rate-prior distortion functions (especially w/NLL distortion) are also in Alemi et al. 2018 as their definition of “rate” is identical to your definition of “rate-prior”. Also many of these properties are specific to continuous latents which isn’t noted.
* Should clarify that q_k(z|x) correspond to points along R_p(D)
* The results in Eqn 14-17 showing you can tractably estimate distortion and get an upper bound on rate using the AIS-derived distributions are very cool!
* “AIS variance is proportional to 1/MK” - this is for variance in the partition function? How does this translate to variance in estimates of rate/distortion?
* “In the case of probabilistic decoders, …” -> need the caveat this is with NLL distortion
* Validation on the linear VAE is great!It looks like some of the points for AIS at low distortion are below the analytic rate, but the proofs indicate the estimated rate should be greater than the analytic rate. Is this just noise?
* Fig 4 and 5: hard to see difference between the dashed lines
* VAE results would change drastically if you targeted them to different regimes (e.g. beta-VAE or constrained optimization like GECO)
* Statements like “VAE is trained with the ELBO objective, which encourages good reconstructions” only make sense when the decoder is location-scale. VAEs w/rich autoregressive decoders typically do a horrible job reconstructing.
* How robust are model differences across random initialization? It’d be great to add error bars to all these plots, especially given that GAN training can stochastically succeed.
* Eqn 18/Fig6a: depending on the dataset, you could easily notice the difference of 4.6 nats and log-likelihood could still tell these two models apart. It’d be useful to add a line at beta=1 to show that the likelihood would be the same but the R-D curves are different.
========================
Update after rebuttal
Thank you to the authors for addressing a number of my concerns, adding additional text and experiments. However, my main concern remains: if rate-distortion in an individual model is a useful method for evaluating generative models, you should compare it empirically with other metrics that have been proposed for this purpose (e.g. precision-recall). Additionally, while the theoretical novelty of getting the full R-D curve from a single AIS run is very cool, I'm skeptical of the practical utility as a metric for generative models due to the computational costs of AIS (4-7 hours for 50 images on MNIST). The simple baseline I suggested of training an amortized inference network with beta annealed over time would not require training a separate encoder for each point in R-D, you could just start beta large, and anneal beta in steps to 0 over time, tracing out an R-D curve. Given the current experiments, it's not obvious if the win of AIS in terms of accurate posterior inference is worth the increased computational cost over a simple VI baseline.