AMBIENTGAN: GENERATIVE MODELS FROM LOSSY MEASUREMENTS
Generative models provide a way to model structure in complex distributions and have been shown to be useful for many tasks of practical interest. However, current techniques for training generative models require access to fully-observed samples. In many settings, it is expensive or even impossible to obtain fullyobserved samples, but economical to obtain partial, noisy observations. We consider the task of learning an implicit generative model given only lossy measurements of samples from the distribution of interest. We show that the true underlying distribution can be provably recovered even in the presence of per-sample information loss for a class of measurement models. Based on this, we propose a new method of training Generative Adversarial Networks (GANs) which we call AmbientGAN. On three benchmark datasets, and for various measurement models, we demonstrate substantial qualitative and quantitative improvements. Generative models trained with our method can obtain 2-4x higher inception scores than the baselines.
The output of the generator is passed through a simulated random measurement function f Θ . The discriminator must decide if a measurement is real or generated. models of the data structure. Recent work has shown that generative models can be particularly effective for easier sensing [Bora et al. (2017); Mardani et al. (2017)]-but if sensing is expensive in the first place, how can we collect enough data to train a generative model to start with?
This work solves this chicken-and-egg problem by training a generative model directly from noisy or incomplete samples. We show that our observations can be even projections or more general measurements of different types and the unknown distribution is still provably recoverable. A critical assumption for our framework and theory to work is that the measurement process is known and satisfies certain technical conditions. We present several measurement processes for which it is possible to learn a generative model from a dataset of measured samples, both in theory and in practice. Our approach uses a new way of training GANs, which we call AmbientGAN. The idea is simple: rather than distinguish a real image from a generated image as in a traditional GAN, our discriminator must distinguish a real measurement from a simulated measurement of a generated image; see Figure 1. We empirically demonstrate the effectiveness of our approach on three datasets and a variety of measurement models. Our method is able to construct good generative models from extremely noisy observations and even from low dimensional projections with drastic per-sample information loss. We show this qualitatively by exhibiting samples with good visual quality, and quantitatively by comparing inception scores [Salimans et al. (2016)] to baseline methods.
Theoretical results. We first consider measurements that are noisy, blurred versions of the desired images. That is, we consider convolving the original image with a Gaussian kernel and adding independent Gaussian noise to each pixel (our actual theorem applies to more general kernels and noise distributions). Because of the noise, this process is not invertible for a single image. However, we show that the distribution of measured images uniquely determines the distribution of original images. This implies that a pure Nash equilibrium for the GAN game must find a generative model that matches the true distribution. We show similar results for a dropout measurement model, where each pixel is set to zero with some probability p, and a random projection measurement model, where we observe the inner product of the image with a random Gaussian vector.
Empirical results. Our empirical work also considers measurement models for which we do not have provable guarantees. We present results on some of our models now and defer the full exploration to Section 8.
In Fig. 2, we consider the celebA dataset of celebrity faces [Liu et al. (2015)] under randomly placed occlusions, where a randomly placed square containing 1/4 of the pixels is set to zero. It is hard to inpaint individual images, so cleaning up the data by inpainting and then learning a GAN on the result yields significant artifacts. By incorporating the measurement process into the GAN training, we can produce much better samples.
In Fig. 3a we consider learning from noisy, blurred version of images from the celebA dataset. Each image is convolved with a Gaussian kernel and then IID Gaussian noise is added to each pixel.  Learning a GAN on images denoised by Wiener deconvolution leads to poor sample quality while our models are able to produce cleaner samples.
In Fig. 3b, we consider learning a generative model on the 2D images in the MNIST handwritten digit dataset [LeCun et al. (1998)] from pairs of 1D projections. That is, measurements consist of picking two random lines and projecting the image onto each line, so the observed value along the line is the sum of all pixels that project to that point. We consider two variants: in the first, the choice of line is forgotten, while in the second the measurement includes the choice of line. We find for both variants that AmbientGAN recovers a lot of the underlying structure, although the first variant cannot identify the distribution up to rotation or reflection.
The paper proposes an approach to train generators within a GAN framework, in the setting where one has access only to degraded / imperfect measurements of real samples, rather than the samples themselves. Broadly, the approach is to have a generator produce the "full" real data, pass it through a simulated model of the measurement process, and then train the discriminator to distinguish between these simulated measurements of generated samples, and true measurements of real samples. By this mechanism, the proposed method is able to train GANs to generate high-quality samples from only imperfect measurements.
The paper is largely well-written and well-motivated, the overall setup is interesting (I find the authors' practical use cases convincing---where one only has access to imperfect data in the first place), and the empirical results are convincing. The theoretical proofs do make strong assumptions (in particular, the fact that the true distribution must be uniquely constrained by its marginal along the measurement). However, in most theoretical analysis of GANs and neural networks in general, I view proofs as a means of gaining intuition rather than being strong guarantees---and to that end, I found the analysis in this paper to be informative.
I would make a  suggestions for possible further experimental analysis: it would be nice to see how robust the approach is to systematic mismatches between the true and modeled measurement functions (for instance, slight differences in the blur kernels, noise variance, etc.). Especially in the kind of settings the paper considers, I imagine it may sometimes also be hard to accurately model the measurement function of a device (or it may be necessary to use a computationally cheaper approximation for training). I think a study of how such mismatches affect the training procedure would be instructive (perhaps more so than some of the quantitative evaluation given that they at best only approximately measure sample quality).