DON'T LET YOUR DISCRIMINATOR BE FOOLED
Generative Adversarial Networks are one of the leading tools in generative modeling, image editing and content creation. However, they are hard to train as they require a delicate balancing act between two deep networks fighting a never ending duel. Some of the most promising adversarial models today minimize a Wasserstein objective. It is smoother and more stable to optimize. In this paper, we show that the Wasserstein distance is just one out of a large family of objective functions that yield these properties. By making the discriminator of a GAN robust to adversarial attacks we can turn any GAN objective into a smooth and stable loss. We experimentally show that any GAN objective, including Wasserstein GANs, benefit from adversarial robustness both quantitatively and qualitatively. The training additionally becomes more robust to suboptimal choices of hyperparameters, model architectures, or objective functions.
## Overview
This paper proposes a new way to stabilize the training process of GAN by regularizing the Discriminator to be robust to adversarial examples. Specifically, this paper proves that a discriminator which is robust to adversarial attacks also leads to a robust minimax objects. Authors provide theoretical analysis about the how the robustness of the Discriminator affects the properties of the objective function, and the proposed regularization term provides an efficient and effective way to regularize the discriminator to be robust. However, it does not build connection between the robustness of the Discriminator and why it can provide meaningful gradient to the Generator. Experimental results demonstrate the effectiveness of the proposed method. This paper is easy to understand.
## Drawbacks
There are some problems in this paper. First, this paper is not highly motivated and lacks of intuition. I can hardly understand why the robustness can stabilize the training of GAN. Will it solve the problem of gradient vanishing problem or speed up the convergence of GAN? The toy example in Sec. 4.2 shows that it can regularize the Discriminator to provide a meaningful gradient to Generator, but no theoretical analysis is provided. The main gap between them is that the smoothness of D around the generated data points does not imply the effectiveness of gradients. Second, the theoretical analysis is inconsistent with the experimental settings. Theorem 4.3 holds true when f is non-positive, but WGAN’s loss function can be positive and this paper does not give any details about this part. Third, in Sec. 4.2, I can hardly distinguish the difference between robust loss, robust discriminator and regularized objectives.
Besides, there are lots of typos in this paper. In Sec 3, Generative Adversarial Networks part, the notations of x and z are quiet confusing. In Definition 3.2, d which measures the distance between network outputs is not appeared above.
## Summarization
Generally, this paper provides a novel way to stabilize the training of GAN. However, it does not illustrate its motivation clearly and no insight is provided.
## After rebuttal
Some of the issues are addressed. So I change my rating to 6.