In this paper, we investigate the underlying factor that leads to the failure and success in training of GANs. Specifically, we study the property of the optimal discriminative function f * (x) and show that f * (x) in most GANs can only reflect the local densities at x, which means the value of f * (x) for points in the fake distribution (P g ) does not contain any information useful about the location of other points in the real distribution (P r ). Given that the supports of the real and fake distributions are usually disjoint, we argue that such a f * (x) and its gradient tell nothing about "how to pull P g to P r ", which turns out to be the fundamental cause of failure in training of GANs. We further demonstrate that a well-defined distance metric (including the dual form of Wasserstein distance with a compacted constraint) does not necessarily ensure the convergence of GANs. Finally, we propose Lipschitz-continuity condition as a general solution and show that in a large family of GAN objectives, Lipschitz condition is capable of connecting P g and P r through f *
[pros]
- It proposes a general formulation of GAN-type adversarial learning as in (1), which includes the original GAN, WGAN, and IPM-type metrics as special cases.
- It also proposes use of the penalty term in terms of the Lipschitz constant  of the discriminative function.
[cons]
- Some of the arguments on the Wasserstein distance and on WGAN are not sound.
- Theorem 3 does not make sense.
- The proposed scheme is eventually similar to the gradient-penalty-based formulation in Gulrajani et al. (2017).
[Quality]
I found some weaknesses in this paper, so that I judge the quality of this paper not to be high. For example, the criticisms on the Wasserstein distance in Section 2.3 and in Section 4.4, as well as the argument on WGAN at the end of Section 3.1, is not sound. The claim in Theorem 3 does not make sense, if we literally take its statement. All these points are detailed below.
[Clarity]
The main paper is clearly written, whereas in the appendices I noticed several grammatical and spelling errors as well as unclear descriptions.
[Originality]
Despite that the arguments in this paper are interesting, the proposed scheme is somehow eventually similar to the gradient-penalty-based formulation in Gulrajani et al. (2017), with differences being introduction of loss metrics $\phi,\varphi,\psi$ and the form of the gradient penalty, $\max \|\nabla f(x)\|_2^2$ in this paper versus $E[(\|\nabla f(x)\|_2-1)^2]$ in Gulrajani et al. (2017). This fact has made me to think that the originality of this paper is marginal.
[Significance]
This paper is significant in that it would stimulate empirical studies on what objective functions and what types of gradient penalty are efficient in GAN-type adversarial learning.
Detailed comments:
In Section 2.3, the authors criticize use of the Wasserstein distance as the distance function of GANs, but their criticism is off the point. It is indeed a problem not of the Wasserstein distance itself, but of its dual formulation.
It is true mathematically that $f$ in equation (8) does not have to be defined outside the supports of $P_g$ and $P_r$ because it does not affect the expectations in (8). In practice, however, one may regard that $f$ satisfies the condition $f(x)-f(y)\le d(x,y)$ not only on the supports of $P_g$ and $P_r$ but throughout the entire space $\mathbb{R}^n$. It is equivalent to requiring $f$ to satisfy the 1-Lipschitz condition on $\mathbb{R}^n$, and is what WGAN (Arjovsky et al., 2017) tries to do in its implementation of the "critic" $f$ via a multilayer neural network with weight clipping.
One can also argue that, if one defines $f$ only on the supports of $P_g$ and $P_r$, then it should trivially be impossible to obtain gradient information which can change the support of $P_g$. The common practice of requiring the Lipschitz condition throughout $\mathbb{R}^n$ is thus reasonable from this viewpoint. This is therefore not the problem of the Wasserstein distance itself, but the problem regarding how the dual problem is implemented in learning of GANs. In this regard, the discussion in this section, as well as that in Section 4.4, is misleading.
On optimizing $k$, I do not agree with the authors's claim at the end of Section 3.1 that WGAN may not have zero gradient with respect to $f$ even when $P_g=P_r$. Indeed, when $P_g=P_r$, for any measurable function $f$ one trivially has $J_D[f]=E_{x\sim P_g}[f(x)]-E_{x\sim P_r}[f(x)]=0$, so that the functional derivative of $J_D$ with respect to $f$ does vanish identically. 
I do not understand the claim of Theorem 3. I think that the assumption is too strong. If one literally takes "$\forall x \not= y$", then one can exchange $x$ and $y$ in the condition $f(y)-f(x)=k\|x-y\|$ to obtain $f(x)-f(y)=k\|y-x\|$, which together would imply $k=0$, and consequently $f$ is constant. One would be able to prove that if there exists $(x,y)$ with $x \not= y$ such that $f(y)-f(x)=k\|x-y\|$ holds then the gradient of $f$ at $x_t$ is equal to $k(y-x)/\|x-y\|$ under the Lipschitz condition.
Appendix G: Some notations should be made more precise. For example, in the definition of J_D the variable of integration $x$ has been integrated out, so that $J_D$ no longer has $x$ as its variable. The expression $\partial J_D/\partial x$ does not make any sense. Also, $J_D^*(k)$ is defined as "arg min" of $J_D$, implying as if $J_D^*(k)$ were a $k$-Lipschitz function.
Page 5, line 36: $J_D(x)$ appears without explicit definition.
Page 23, lines 34 and 38: Cluttered expression $\frac{\partial [}{\partial 2}]$ makes the statements not understandable. It also appears on page 24 several times.