On GANs and GMMs
A longstanding problem in machine learning is to find unsupervised methods that can learn the statistical structure of high dimensional signals. In recent years, GANs have gained much attention as a possible solution to the problem, and in particular have shown the ability to generate remarkably realistic high resolution sampled images. At the same time, many authors have pointed out that GANs may fail to model the full distribution ("mode collapse") and that using the learned models for anything other than generating samples may be very difficult. In this paper, we examine the utility of GANs in learning statistical models of images by comparing them to perhaps the simplest statistical model, the Gaussian Mixture Model. First, we present a simple method to evaluate generative models based on relative proportions of samples that fall into predetermined bins. Unlike previous automatic methods for evaluating models, our method does not rely on an additional neural network nor does it require approximating intractable computations. Second, we compare the performance of GANs to GMMs trained on the same datasets. While GMMs have previously been shown to be successful in modeling small patches of images, we show how to train them on full sized images despite the high dimensionality. Our results show that GMMs can generate realistic samples (although less sharp than those of GANs) but also capture the full distribution, which GANs fail to do. Furthermore, GMMs allow efficient inference and explicit representation of the underlying statistical structure. Finally, we discuss how GMMs can be used to generate sharp images.
Summary  The authors compare the performance of Gaussian mixture models (GMMs) and deep generative models (mainly GANs, but also VAEs) for image generation. They use a mixture of factor analysers (MFA) model and train it on several image data sets, and show that this model can faithfully learn the distribution of images. They also propose a new evaluation scheme for generative models, based on Voronoi cells. They use this scheme to compare MFAs, GANs, and VAEs: MFAs perform pretty well, and avoid mode collapse but produce somewhat blurry images. They propose a scheme based on pix2pix to obtain sharper, "GAN-like" images. Globally, the paper is rather thought-provoking and carries an interesting message.  Quality  Overall, the technical quality of the paper is rather good. Regarding the new evaluation method, I think it makes sense. However, a theoretical assessment (at least superficial) of this method would be welcomed. It is not clear what kind of discrepancy the proposed method measures, while some other popular techniques have clearer interpretations (like the Wasserstein distance, or the test log-likelihood).  The MFA part is rather classical but interesting and natural. More details should be added regarding the choice of the dimension of the latent space and the number of components. For example, would it make sense to use different dimensions for the different components.  Regarding the experiments, I think the authors do a good job at showing that their metric is a good "mode collapse detector". I also really like that they show how to interpret the latent variables of the MFA model. However, I think that a crucial experiment would be welcomed: the comparison of (both training and test) log-likelihoods for their models and the competitors that allow likelihood evaluation . It is possible to evaluate the likelihood of VAEs using importance sampling, and there are other deep generative models of images, like RNVP (Density estimation using Real NVP, Dinh, Sohl-Dickstein, and Bengio, ICLR 2017) that allow direct likelihood evaluation. Since their model is also trained using maximum likelihood, such experiments would be very insightful.  Clarity  I think that the paper reads quite well. A few technical points should be clarified (at least in the supplementary material): - What is precisely meant by "a GMM can approximate any density" (l. 120)? With what metric? A citation should back this claim as well as the one stated just after that ("it is easy to construct densities for which the number of gaussian required grows exponentially"). - How is in-painting done exactly? - How is SGD implemented? The authors say that each component is estimated separately (l. 155), could they explain precisely what this means? What is the learning rate? - The relevance of using L2 Voronoi cells should be discussed more.  Originality  The evaluation method is, to the best of my knowledge, novel. The MFA model is very well known, but using it for image generation is somewhat original (in particular the experiments of Fig. 6b). The pix2pix part introduces also an idea that I've never seen before: first learn roughly the distribution with a technique immune to mode collapse, then refine the samples to produce sharp images.   Significance  I really like the general message of this paper, which, albeit not surprising, is thought provoking. However, I think that such an empirical paper should have much stronger experiments. In particular, I think that some comparisons with the existing metrics (like FID, test log-likelihood, or estimated Wasserstein distance) should be present in the paper. In particular, it is very surprising not to see likelihood comparisons with other likelihood-based models. Moreover, the sharp samples obtained with pix2pix could be compared to GANs samples under the FID or the Wasserstein metric.   Minor comments  - In-painting is sometimes spelled in-painting and sometimes inpainting  - The title is quite misleading, and the analysis should focus more generally on deep generative models (not only GANs)   ------ Post-rebuttal edit ------  I have read the rebuttal, that clarified a few points. My main concern remains the absence of likelihood evaluation for the other techniques. In particular, I think that the likelihood of the VAE trained by the authors should really be compared to the one of the GMM in the final version (for example in the supplementary material). This can be rather easily done using importance sampling, as in many VAE papers. Beyond that, I still think that the overall message is very valuable and that the paper should be accepted.