While generative adversarial networks (GAN) have been widely adopted in various topics, in this paper we generalize the standard GAN to a new perspective by treating realness as a random variable that can be estimated from multiple angles. In this generalized framework, referred to as RealnessGAN, the discriminator outputs a distribution as the measure of realness. While RealnessGAN shares similar theoretical guarantees with the standard GAN, it provides more insights on adversarial learning. More importantly, compared to multiple baselines, RealnessGAN provides stronger guidance for the generator, achieving improvements on both synthetic and real-world datasets. Moreover, it enables the basic DCGAN (Radford et al., 2015) architecture to generate realistic images at 1024*1024 resolution when trained from scratch.
Post rebuttal: The authors' responses have addressed most of my concerns, and I've raised my rating from 3 to 6.
----------------------------------------
Summary:
This paper extends the discriminator of GAN to use a distributional output (multiple scalars) instead of a single scalar. As a result, the trained GAN becomes robust to the mode collapse.
Pros:
- The proposed method is clearly written and well-justified (e.g., Theorem 2).
- Extension of the relativistic GAN [1] to the proposed setting is interesting.
- The authors demonstrate that vanilla DCGAN architecture can generate high-fidelity (1024x1024) images.
Cons:
1. An ensemble of discriminators?
The authors use multiple scalars to consider diverse factors of the realness. However, it is simply an ensemble of discriminators [2] in a spirit. As each discriminator focus on different factors, it is not surprising that the generator becomes robust to the mode collapse. Also, recent work on mode collapse (e.g., [3]) shows better results on the mixture of gaussian experiments even using a single discriminator. At least, the authors should compare their method with the ensemble methods and claim the advantage over them.
2. Choice of the anchor distributions.
The choice of anchor distributions A_0 and A_1 are not specified. While the authors provide some partial results in Table 2, it would be worthwhile to clarify the experimental details and justify them.
3. Role of each outcome u_i?
The authors claim that each outcome u_i corresponds to the different factors of realness. However, the role of learned u_i is not investigated. Also, one may enforce u_i to learn different factors by promoting diversity of them, e.g., decrease their cosine similarity [4].
Minor comments:
- The word "support" [5] is misused. The support itself means the set of non-zero elements, hence the authors should use the word "outcome" (or "sample") instead of "support".
- The notation is not consistent. For example, the authors may use "x \sim p_data(x)" (specify variable) or "z \sim p_z" (omit variable), but not both.
- Numbering is not consistent. For example, "Tab.4.2." should be changed to "Tab.2." for consistency.
[1] Jolicoeur-Martineau. The relativistic discriminator: a key element missing from standard GAN. ICLR 2019.
[2] Durugkar et al. Generative Multi-Adversarial Networks. ICLR 2017.
[3] Xiao et al. BourGAN: Generative Networks with Metric Embeddings. NeurIPS 2018.
[4] Elfeki et al. GDPP: Learning Diverse Generations Using Determinantal Point Process. ICML 2019.
[5] https://en.wikipedia.org/wiki/Support_(mathematics)