Published as a conference paper at ICLR 2018 DO GANS LEARN THE DISTRIBUTION? SOME THEORY AND EMPIRICS
This paper proposes a clever new test based on the birthday paradox for measuring diversity in generated samples. The main goal is to quantify mode collapse in state-of-the-art generative models. The authors also provide a specific theoretical construction that shows bidirectional GANs cannot escape specific cases of mode collapse.
Using the birthday paradox test, the experiments show that GANs can learn and consistently reproduce the same examples, which are not necessarily exactly the same as training data (eg. the triplets in Figure 1).
The results are interpreted to mean that mode collapse is strong in a number of state-of-the-art generative models.
Bidirectional models (ALI, BiGANs) however demonstrate significantly higher diversity that DCGANs and MIX+DCGANs.
Finally, the authors verify empirically the hypothesis that diversity grows linearly with the size of the discriminator.
This is a very interesting area and exciting work. The main idea behind the proposed test is very insightful. The main theoretical contribution stimulates and motivates much needed further research in the area. In my opinion both contributions suffer from some significant limitations. However, given how little we know about the behavior of modern generative models, it is a good step in the right direction.
1. The biggest issue with the proposed test is that it conflates mode collapse with non-uniformity. The authors do mention this issue, but do not put much effort into evaluating its implications in practice, or parsing Theorems 1 and 2. My current understanding is that, in practice, when the birthday paradox test gives a collision I have no way of knowing whether it happened because my data distribution is modal, or because my generative model has bad diversity. Anecdotally, real-life distributions are far from uniform, so this should be a common issue. I would still use the test as a part of a suite of measurements, but I would not solely rely on it. I feel that the authors should give a more prominent disclaimer to potential users of the test.
2. Also, given how mode collapse is the main concern, it seems to me that a discussion on coverage is missing. The proposed test is a measure of diversity, not coverage, so it does not discriminate between a generator that produces all of its samples near some mode and another that draws samples from all modes of the true data distribution. As long as they yield collisions at the same rate, these two generative models are ‘equally diverse’. Isn’t coverage of equal importance?
3. The other main contribution of the paper is Theorem 3, which shows—via a very particular construction on the generator and encoder—that bidirectional GANs can also suffer from serious mode collapse. I welcome and are grateful for any theory in the area. This theorem might very well capture the underlying behavior of bidirectional GANs, however, being constructive, it guarantees nothing in practice. In light of this, the statement in the introduction that “encoder-decoder training objectives cannot avoid mode collapse” might need to be qualified. In particular, the current statement seems to obfuscate the understanding that training such an objective would typically not result into the construction of Theorem 3.