Generative adversarial networks (GANs) are powerful generative models, but usually suffer from instability which may lead to poor generations. Most existing works try to alleviate this problem by focusing on stabilizing the training of the discriminator, which unfortunately ignores the robustness of both generator and discriminator. In this work, we consider the robustness of GANs and propose a novel robust method called robust generative adversarial network (RGAN). Particularly, we design a robust optimization framework where the generator and discriminator compete with each other in a worst-case setting within a small Wasserstein ball. The generator tries to map the worst input distribution (rather than a specific input distribution, typically a Gaussian distribution used in most GANs) to the real data distribution, while the discriminator attempts to distinguish the real and fake distribution with the worst perturbation. We have provided theories showing that the generalization of the new robust framework can be guaranteed. A series of experiments on CIFAR-10, STL-10 and CelebA datasets indicate that our proposed robust framework can improve consistently on four baseline GAN models. We also provide ablation analysis and visualization showing the efficacy of our method on both generator and discriminator quantitatively and qualitatively.
Summary
The present work proposes to combine GANs with adversarial training replacing the original GAN lass with a mixture of the original GAN loss and an adversarial loss that applies an adversarial perturbation to both the input image of the discriminator, and to the input noise of the generator. The resulting algorithm is called robust GAN (RGAN). Existing results of [Goodfellow et al 2014] (characterizing optimal generators and discriminators in terms of the density of the true data) are adapted to the new loss functions and generalization bounds akin to [Arora et al 2017] are proved. Extensive experiments show a small but consistent improvement over a baseline method.
Decision
The authors do a thorough job at characterizing the proposed method using both theoretical analysis and wide ranging experimental studies. My main criticism of the paper in its present form is the lack of motivation for the proposed method. Why, out of the many possible ways to impose additional regularization should one use adversarial training to regularize GANs? While it is remarkable that the experimental results seem to be improving consistently, the improvement is quite small. Similarly, while theoretical results are provided, a discussion of what they mean for the performance of RGAN is sorely lacking, leaving me unconvinced that adversarial training leads to an improvement over GANs when compared with simpler methods of regularization. Therefore I vote to reject the paper in its present form.
Suggestions for improvement on the experiments
My main concern with the experiments is that a similar small improvement over the baseline could be achieved by tuning the hyperparameters in an alternative simpler regularization method. For instance, instead of using an adversarial perturbation, one could simply use a random perturbation applied to both the random noise and the discriminator input at testing time. The former would amount to a variance of the truncation trick [Brock et al 2019], while the latter would amount to using instance noise. These are established methods to improve GAN performance and to make a case for adversarial training of GANs one would need to show improvements compared to these simpler strategies, in my opinion.
My main suggestion for the theoretical part is to make a stronger case of what (if anything) these theoretical results say about the performance of RGAN compared to the usual GAN. In particular, the generalization bound does not seem to depend on lambda, (which interpolates between the original GAN and RGAN). What is to be inferred from these results regarding the performance of RGAN?
Questions to the authors
(1) I assume you perform adversarial training in practice by backpropagating in image/noise space? How does this affect performance? How would the convergence plots look like if wall-clock time, or the number of model evaluations were used on the x-axis?
(2) Did you try investing a similar computational budget to tune hyperparameters for simpler regularization methods as mentioned above and compare the resulting improvement?
(3) Is the value (I presume, standard deviation) given after each the inception score computed for different multiple iterations of the same run or multiple runs with different initialization and random seed?