Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
In this paper we introduce a generative parametric model capable of producing high quality samples of natural images. Our approach uses a cascade of convolutional networks within a Laplacian pyramid framework to generate images in a coarse-to-fine fashion. At each level of the pyramid, a separate generative convnet model is trained using the Generative Adversarial Nets (GAN) approach [11]. Samples drawn from our model are of significantly higher quality than alternate approaches. In a quantitative assessment by human evaluators, our CIFAR10 samples were mistaken for real images around 40% of the time, compared to 10% for samples drawn from a GAN baseline model. We also show samples from models trained on the higher resolution images of the LSUN scene dataset.
As in the summary -- the paper implements a sensible idea, it is clearly written, and the generated images are beautiful. I have a concern that the quantitative results are misleading and/or wrong.I have a suspicion that the model may be performing similarly to a computer graphics engine ... where it generates very good naturalistic images, but where most real images would be assigned extremely low probability. This could make for a fine paper, but the results would need to be presented in a way that makes this clear.Detailed comments as follow:45 - 'indicating a better density model...' don't think this part follows57-61 - Should note these aren't actually generative models as typically defined. Though p(visible|hidden) is straightforward, p(hidden) is complex and difficult to sample from.72 - 'much more ambitious', and/or addresses a different goal93 - 'their respective' -> 'both'148 - prob. move footnote to main text, following a comma at the end of the sentence186 - 'can directly' -> 'can be directly'191-193 - Is this true? I would imagine fitting a model to each scale would just provide more parameters with which to overfit.220 - I wonder if it will be difficult for these variables to represent something interpretable about the image, since there is only one noise variable per pixel, and the noise variables are interpreted convolutionally which will largely constrain them all to play a very similar role.226 - On a validation set?231 - Good idea + interesting to visualize variation captured.239 - 'nature image' -> 'natural image'Table 1: - Give units. Nats/image?- If the units are nats/image, then I believe this performance would be state of the art if you compare against lossless image compression of 24-bit full size images. This would be impressive! But it also makes me extremely skeptical, because my understanding is that Parzen window techniques badly undershoot the true log likelihood, and 50,000 samples is a very small number of samples to fill the space of 32x32 images -- or even to fill a state space of dimensionality (32x32*3 - 16x16*3), after the conditioning. (e.g., assuming pixel values are integers, you would need order exp(1799 * (32*32*3 - 16*16*3) / (32*32*3)) samples to accurately capture the distribution using your Parzen window technique ... which is so much larger than 50000 that it leads to a floating point overflow when I try to compute it)- Did you add uniform noise in [-0.5, 0.5] to each pixel value in your training and test sets? If not, it may be that scale 0 is learning to capture something about the integer quantization of pixel intensities. This is cheating, in that you can get infinite likelihood without learning anything about natural images by modeling the data as delta functions at integer pixel intensities. This could matter even for the multiscale Parzen approach, since it may be possible to exactly predict pixel values at scale 0 conditioned on scale 1.- I'm especially suspicious that Parzen is being performed incorrectly, because you are also using non-comparable Parzen variants to compare GAN and LAPGAN. See Appendix A comment below.- (A really neat followup project, if the numbers do turn out to be real, would be to use something similar to this method for image or video compression. I think the multi-scale structure would make for a straightforward encoding and automatically get you smooth performance degradation with limited bandwidth.)- There are other models you could compare against here. [24] provides a bound on log likelihood for CIFAR-10. (and I believe also a deep learning workshop paper at this year's ICML -- will try to go back and look this up)(Update: I realized from reviewer conversation that these log likelihoods are for 28x28 pixel images, not 32x32. That does explain the difficulty in direct comparison, and makes the log likelihood about 25% more believable. It would be good to note the image size in Table 1.)Figure 5. - This is both interesting and difficult to know how to interpret. It seems as if image generation is quite stereotyped given initial 4x4 seeds. That would be consistent with the noise variables entering into later layers in a possibly difficult to utilize way (line 220). One interpretation might be that the top layer GAN creates most of the diversity in the generated samples, and that the conditional layers fill in consistent but inflexible details.Section 5.3: - It's very difficult to know how subjects will choose to label something as "real" or "not real". For instance, if an image appears out of focus, or pixelated, or like an animal that you can't quite discern, do they still label it real? If you want to do a fair comparison against natural images, it would be much, much better to force subjects to make a choice between generated images and natural images. (this would be a "two alternative forced choice" experiment, and is extremely common)- Who were the volunteers? What instructions did they receive? Did they know when they were looking at GAN vs. LAPGAN images, or what type of responses the experimenters hoped for in each session? Was the experiment single blind? Double blind? Did the subjects have previous experience with all three datasets? Strangers on mechanical turk, and a double blind experiment, would be better.Some suggestions for two possible improved psychophysics experiments: 1) Do generated images look real: Show two images next to each other, and ask the user to select which one is 'real'. As they approach chance performance, this will be a strong indication that the images produced by the model appear realistic. 2) Are generated images diverse: Generating realistic images doesn't mean that you've captured the distribution over real images. For instance, maybe all the images you generate look very similar. One experiment that could significantly address this is to show real and generated images next to each other. Then show a third image, either real or generated, below, and ask which of the first two the third image is most similar to. The degree to which generated images are chosen to be similar to each other at above chance level is a measure of their diversity relative to natural images.- If you are interested in the effect of exposure time, you should display a mask image (e.g. checkerboard or white noise) after the stimulus, rather than a gray screen. Otherwise your subject will have additional time to view the afterimage.- You should provide a fixation target (e.g. a small cross) before each trial, and instruct the subject to fixate on it.- This experiment seems obviously harmless, but you should be *very careful* about doing human experimentation without approval from an institutional review board. Running human experiments without review by an ethics board can get you in a lot of trouble, especially in a university setting. Many institutions have short online forms and a very easy expedited process for obviously harmless experiments like this one. Getting approval should be pretty lightweight, but is strongly recommended.403 - 'previous scales' -> 'more than one previous scale' (assuming I've understood what's going on correctly)Appendix A: This is a neat idea! Nice.Parzen window estimates tend to be biased low because they require an exponential number of samples to fully explore a high dimensional space. Your method instead requires samples to explore a sequence of lower dimensional spaces. This will make it much easier for the samples to explore the state space, and can be expected to greatly increase the Parzen log likelihood, which is great.You measure the GAN log likelihood using a Parzen estimate in the full space, and the LAPGAN log likelihood using your modified Parzen estimate. This isn't a fair comparison, since it is impossible to tell how much of the improvement is due to the improved Parzen estimation method, and how much is due to better performance of the model.I strongly encourage you to include a third row in Table 1, where you give the LAPGAN log likelihood computed with the same Parzen estimator as the GAN row. Then it will be possible to tease apart model improvement vs. measurement technique improvement.417 - notation, should probably make this q_0(h | l)--After reading the comments from Assigned_Reviewer_2, I share their concern that the number of latent variables is smaller than the number of free data dimensions in the conditional GAN layers (by a factor of 3 I believe). I would also like to request clarification on this.