Evaluating Lossy Compression Rates of Deep Generative Models
Deep generative models have achieved remarkable progress in recent years. Despite this progress, quantitative evaluation and comparison of generative models remains as one of the important challenges. One of the most popular metrics for evaluating generative models is the log-likelihood. While the direct computation of log-likelihood can be intractable, it has been recently shown that the log-likelihood of some of the most interesting generative models such as variational autoencoders (VAE) or generative adversarial networks (GAN) can be efficiently estimated using annealed importance sampling (AIS). In this work, we argue that the log-likelihood metric by itself cannot represent all the different performance characteristics of generative models, and propose to use rate distortion curves to evaluate and compare deep generative models. We show that we can approximate the entire rate distortion curve using one single run of AIS for roughly the same computational cost as a single log-likelihood estimate. We evaluate lossy compression rates of different deep generative models such as VAEs, GANs (and its variants) and adversarial autoencoders (AAE) on MNIST and CIFAR10, and arrive at a number of insights not obtainable from log-likelihoods alone.
This paper considers the rate-distortion tradeoffs of deep generative models such as variational autoencoders (VAEs) and generative adversarial networks (GANs). The authors propose an annealed importance sampling (AIS) method to compute the rate-distortion curve efficiently. In experiments, the authors compare the rate-distortion curves for VAEs and GANs and discuss the properties of rate-distortion curves. The method for computing the rate-distortion curves of deep generative models is interesting and the rate-prior distortion curve is promising as a performance measure. However, the main technical contribution of this work is the estimated AIS rate-prior distortion curve and it is based on a straight-forward application of AIS. In fact, Sections 2 and 3 discuss already known result in literature although summarizing them in a paper is nice for readers. Although the findings in the experiments are interesting and insightful, they are still preliminary and further investigations are desirable. In Section 5, the authors mention the consistency of their framework with Shannon’s rate distortion theorem. This seems to be a little overstatement because the authors discuss little about the optimization of the prior p(z).
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Evaluating Lossy Compression Rates of Deep Generative Models
Deep generative models have achieved remarkable progress in recent years. Despite this progress, quantitative evaluation and comparison of generative models remains as one of the important challenges. One of the most popular metrics for evaluating generative models is the log-likelihood. While the direct computation of log-likelihood can be intractable, it has been recently shown that the log-likelihood of some of the most interesting generative models such as variational autoencoders (VAE) or generative adversarial networks (GAN) can be efficiently estimated using annealed importance sampling (AIS). In this work, we argue that the log-likelihood metric by itself cannot represent all the different performance characteristics of generative models, and propose to use rate distortion curves to evaluate and compare deep generative models. We show that we can approximate the entire rate distortion curve using one single run of AIS for roughly the same computational cost as a single log-likelihood estimate. We evaluate lossy compression rates of different deep generative models such as VAEs, GANs (and its variants) and adversarial autoencoders (AAE) on MNIST and CIFAR10, and arrive at a number of insights not obtainable from log-likelihoods alone.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper considers the rate-distortion tradeoffs of deep generative models such as variational autoencoders (VAEs) and generative adversarial networks (GANs). The authors propose an annealed importance sampling (AIS) method to compute the rate-distortion curve efficiently. In experiments, the authors compare the rate-distortion curves for VAEs and GANs and discuss the properties of rate-distortion curves. The method for computing the rate-distortion curves of deep generative models is interesting and the rate-prior distortion curve is promising as a performance measure. However, the main technical contribution of this work is the estimated AIS rate-prior distortion curve and it is based on a straight-forward application of AIS. In fact, Sections 2 and 3 discuss already known result in literature although summarizing them in a paper is nice for readers. Although the findings in the experiments are interesting and insightful, they are still preliminary and further investigations are desirable. In Section 5, the authors mention the consistency of their framework with Shannon’s rate distortion theorem. This seems to be a little overstatement because the authors discuss little about the optimization of the prior p(z).
assistant
This paper considers the rate-distortion tradeoffs of deep generative models such as variational autoencoders (VAEs) and generative adversarial networks (GANs). The authors propose an annealed importance sampling (AIS) method to compute the rate-distortion curve efficiently. In experiments, the authors compare the rate-distortion curves for VAEs and GANs and discuss the properties of rate-distortion curves. The method for computing the rate-distortion curves of deep generative models is interesting and the rate-prior distortion curve is promising as a performance measure. However, the main technical contribution of this work is the estimated AIS rate-prior distortion curve and it is based on a straight-forward application of AIS. In fact, Sections 2 and 3 discuss already known result in literature although summarizing them in a paper is nice for readers. Although the findings in the experiments are interesting and insightful, they are still preliminary and further investigations are desirable. In Section 5, the authors mention the consistency of their framework with Shannon’s rate distortion theorem. This seems to be a little overstatement because the authors discuss little about the optimization of the prior p(z).