Deep Generative Models for Distribution-Preserving Lossy Compression
We propose and study the problem of distribution-preserving lossy compression. Motivated by recent advances in extreme image compression which allow to maintain artifact-free reconstructions even at very low bitrates, we propose to optimize the rate-distortion tradeoff under the constraint that the reconstructed samples follow the distribution of the training data. The resulting compression system recovers both ends of the spectrum: On one hand, at zero bitrate it learns a generative model of the data, and at high enough bitrates it achieves perfect reconstruction. Furthermore, for intermediate bitrates it smoothly interpolates between learning a generative model of the training data and perfectly reconstructing the training samples. We study several methods to approximately solve the proposed optimization problem, including a novel combination of Wasserstein GAN and Wasserstein Autoencoder, and present an extensive theoretical and empirical characterization of the proposed compression systems.
This works uses generative models for the task of compression. The premise behind this paper is that distribution preserving lossy compression should be possible at all bitrates. At low bitrates, the scheme should be able to recover data from the underlying distribution, while at higher bitrates it produces data (images) that  reconstructs the input samples.  That the scheme works as advertised can be seen rather convincingly in section 5 containing the evaluations.   The paper is well written and constitutes a very interesting exercise.  The compression setup consists of an encoder to compress the input image E, and a stochastic component B, and a generator G recovers the input. Together, they are trained to minimize a distortion measure. The objective function is formulated as a Lagrangian consisting of the distortion measure between input and reconstruction and a divergence term, implemented as a GAN which takes effect at low bitrates so that it recovers images from the underlying distribution.   Several other generative compressive setups are compared against. But the closest comparison (in terms of approach) seems to be the work "Generative Adversarial Networks for learned image compression" (termed GC in the results). In this connection, they point out that if one were simply to minimize the distortion measure as done in GC, it results in the setup largely ignoring the noise injected except at very low bitrates where it is the only factor involved. I think I can intuitively understand this part, but a little clarification would be useful.  The reasoning in section 3, on why one cannot simply replace the distortion metric using a Wasserstein distance seems to be that the encoder is rate constrained and therefore optimizing this metric would not result in optimizing over all couplings. They then go on to decompose the distortion measure (equation (4)) as the sum of two terms, one of which is the wasserstein distance and the other, a term containing the distortion rate which goes to zero at large values.. By doing so (the generator is learnt separately) they can replace the distortion measure with the said Wasserstein term and optimize it.   The connection where they jointly train the setup to function essentially as a VAE-GAN by combining the two terms in the objective - the distortion as the primal, and the divergence as the dual is a most interesting trick for future use.   One aspect that I could not piece together (it is likely that I am missing it) is how the Wasserstein++ algorithm manages to enforce the 1-Lipshitz constraint for the GAN part of the objective. The 1-Lipshitz requirement (as one would expect) is explicitly mentioned in equation 11 for the GAN term. As far as I can see, weight clipping/gradient penalty does not seem to be mentioned in Wasserstein++.   As regards the evaluation studies produced in section 5,  I am curious about use of the Frechet Inception Distance as a metric for use in GANs.   On the whole, I am very impressed with this work. It presents a way to attain generative compression at both low and high bitrates, with reconstructions getting better at higher bitrates and the model recovering distribution images (in GAN like fashion) at low bitrates. The use of Wasserstein autoencoders is also quite interesting.  Answers to specific review criteria below: 1) Quality: Excellent.  2) Clarity: Very good. The mathematical details could perhaps be worked out as an extended tutorial. 3) Originality: Very good. The material presents Wasserstein autoencoders and GANs (combined as Wasserstein++) as the approach to effect compression. This is rather unique.  4) Significance: Very good. Generative compression is a significant area of research.