PacGAN: The power of two samples in generative adversarial networks
Generative adversarial networks (GANs) are a technique for learning generative models of complex data distributions from samples. Despite remarkable advances in generating realistic images, a major shortcoming of GANs is the fact that they tend to produce samples with little diversity, even when trained on diverse datasets. This phenomenon, known as mode collapse, has been the focus of much recent work. We study a principled approach to handling mode collapse, which we call packing. The main idea is to modify the discriminator to make decisions based on multiple samples from the same class, either real or artificially generated. We draw analysis tools from binary hypothesis testing-in particular the seminal result of Blackwell [4]-to prove a fundamental connection between packing and mode collapse. We show that packing naturally penalizes generators with mode collapse, thereby favoring generator distributions with less mode collapse during the training process. Numerical experiments on benchmark datasets suggest that packing provides significant improvements.
This paper considers the idea of using a mini-batch of samples to train the discriminator (packing) to mitigate the mode collapse issue in training GANs. This idea has been around in the literature and is known as mini-batch discrimination [1]. As the authors have mentioned, there are many ways for applying this technique. The authors propose a very simple implementation that just uses a fully connected layer in the input to connect the packed samples to the discriminator network. It is surprising and interesting that such a simple implementation works in practice. It seems to be a counter-intuitive structure as it does not preserve the locality of the pixel information in samples of a batch. I do not know if this would be problematic in larger experiments that make use of convolutional units at the input layers of their discriminator. Moreover, such a structure is not necessarily permutation invariant, i.e. does not treat all the samples in the batch symmetrically.  Analysis:  The paper seems to be exaggerating the importance and implications of the theoretical analysis. First of all the analysis only applies to the TV distance. Note that TV distance is not used in practice for training GANs as it is not practical. Moreover, the kind of relationship that is proved between mode collapse and packing in the analysis seems to be objective dependent and might not be true for some plausible objectives. A simple example to see that such a relationship does not hold is when some variant of KL divergence is used as an objective. Note that KL(P^N, Q^N) = N KL(P,Q). Therefore, packing in this case does not seem to have any effect in changing the geometry of the objective. Thus, it is not appropriate to make general claims about the fundamental relationship between packing and mode collapse (see lines 10-11) based on these limited theoretical results.  The statements of the theorems 3 and 4 are very difficult to digest. Unfortunately, no intuition or clarification is provided to make them more understandable. Moreover, from the results of theorem 2 and 3, it is not readily clear that for a given epsilon, delta and tau if the lower bound of Theorem 2 would be greater than the upper-bound in Theorem 3 for some packing order m. Note that this is analytically crucial in order to make sure that a packed discriminator can distinguish mode collapse. This point does not seem to be proven explicitly, but has been mentioned as an implication of theorem 2 and 3; see lines 206-210 and 218-221. It seems that the authors only justify this point using plots in Figure 3 and Figure 11. Note that the epsilon is different for red and blue curves in Figure 3 and 11.   Experiments:  In the experimental section, the authors have done a good job of isolating the effects of packing on mode collapse. The improvements in mode collapse reduction, compared to the baseline methods, seem to be substantial, especially in the stacked MNIST and celebA experiments. However, the set of baseline methods do not include the most stable/state of the art methods/objectives for GANs. For example, the more stable set of methods that use the Wasserstein family of distances with some regularization, e.g. [2], are absent from the experiments. Therefore, it is difficult to confirm that such a large improvement is not due to the instability of the baseline methods in training.  Moreover, no comparison with the prior methods that use mini-batch discrimination is presented. This comparison is crucial because those methods stem from the same idea as packing.  It is also important to include some experiments on larger/more challenging GAN datasets such as CIFAR-10 or ImageNet. Note that the current set of experiments are more focused on evaluating the mode collapse. But the quality/sharpness of the generated samples is also important in generative models. For these larger datasets, the metrics such as inception or FID score could be used to measure the quality of the produced images as well as the mode collapse problems.   The authors also claim that the introduction of packing has little computational overhead. It is not readily obvious that the introduction of packing does not affect the convergence speed of the methods. It would be very useful if some experimental results are presented to prove the above claim.   Presentation:  In terms of clarity and presentation, the paper requires a lot of improvement. The authors keep mentioning the relationship between the ROC curve and mode collapse region as a fundamental and intuitive point in understanding the analysis. If that is really the case, maybe it would be better to reshuffle the content so that such this point is presented in the main body. As it was mentioned above, the theorems are not clearly presented and the reader is left alone to make sense of very sophisticated expressions without any clarifications or intuitions.  Some minor comments:  beta is not defined in Theorem 3. -   In Figure 11, even with different epsilons, at least a packing of order 5 is required to distinguish mode collapse. It would be interesting if the authors can provide a relationship between the required order of packing and the epsilon gap in H1 and H0.  [1] Salimans, Tim, et al. "Improved techniques for training GANs." Advances in Neural Information Processing Systems. 2016.  [2] Gulrajani, Ishaan, et al. "Improved training of Wasserstein GANs." Advances in Neural Information Processing Systems. 2017.