In this paper we show strategies to easily identify fake samples generated with the Generative Adversarial Network framework. One strategy is based on the statistical analysis and comparison of raw pixel values and features extracted from them. The other strategy learns formal specifications from the real data and shows that fake samples violate the specifications of the real data. We show that fake samples produced with GANs have a universal signature that can be used to identify fake samples. We provide results on MNIST, CIFAR10, music and speech data.
The primary purpose of this paper, from what I understand, is to show that fake samples created with common generative adversarial network (GAN) implementations are easily identified using various statistical techniques. This can potentially be useful in helping to identify artificial samples in the real world.
I think that the reviewers did an excellent job of probing into different statistical perspectives, such as looking at the continuity of the distribution of pixel intensities and the various higher moments of spectral data. I also must applaud the fact that they did not relegate themselves to image data, but branched out to speech and music data as well.
One of the first findings is that, with MNIST and CIFAR, the pixel intensities of fake samples are noticeably different when viewed from the perspective of a Kolgomorov-Smirnov Test or Jensen-Shannon Divergence comparison. This is an interesting observation, but less so than it would be if compared to something such as a variational autoencoder (VAE), which fits a KL distribution explicitly. IWGAN and LSGAN are using different metrics in their loss functions (such as Wassertein and least squares), and thus the result is not surprising or novel. I think if the authors had somehow shown how they worked their metrics into IWGAN or LSGAN to achieve better results, this could have been interesting.
Another observation the authors make is about the smoothness of the GAN distributions. This may not be so easily wrapped into the loss function, but it seems easily remedied as a post-processing step, or perhaps even a smoothing layer in the network itself. Nevertheless, this is an observation that I have not seen discussed in the literature so there is merit to at least noting the difference. It is confusing that on page 4, the authors state that they hypothesized that the smoothness was due to the pixel values themselves, and chose to alter the distribution of the original pixels in [0,1]. However, they state that in Figure 5, the smoothness remained "as expected." Did the authors misspeak here?
I found the music and speech experiments very interesting. The authors note that the synthetic Bach chorales, for instance, introduce many transitions that are not seen in the training and testing set of real Bach chorales. This, again, is interesting to note, but not surprising as the authors are judging the synthetic chorales on criteria for which they were not explicitly optimized. I do not believe these observations to be paper-worthy by themselves. However, the authors I believe have a good start on creating papers in which they specifically address these issues, showing how they can create better synthetic samples by incorporating their observations.
As to the writing style, there are many places where the writing is not quite clear. I would suggest getting an additional party to help proofread to avoid grammatical mistakes. I do not believe that the mistakes are so egregious as to impede understanding. However, it could distract from the importance on the authors future innovations if not corrected.
One last note. The title of the paper is "TequilaGAN: How to Easily Identify GAN Samples." This makes it seem as if the authors were introducing another type of GAN, like LSGAN or DCGAN. However, they are not. As a matter of fact, nowhere else in the paper is the word "TequilaGAN" mentioned. This title seems a bit sensational and misleading.
In the end, although I did find this paper to be an interesting read, I cannot recommend it for publication in ICLR.
----
Edit - November 29, 2018: Increasing my rating from a 4 to a 5 after discussion with the authors. Though their insights are not unknown, I think the authors are right in the fact that this is not explicitly discussed, at least not in the peer-review research with which I am familiar. But I don't think this by itself merits an ICLR publication.