On the Convergence and Robustness of Training GANs with Regularized Optimal Transport
Generative Adversarial Networks (GANs) are one of the most practical methods for learning data distributions. A popular GAN formulation is based on the use of Wasserstein distance as a metric between probability distributions. Unfortunately, minimizing the Wasserstein distance between the data distribution and the generative model distribution is a computationally challenging problem as its objective is non-convex, non-smooth, and even hard to compute. In this work, we show that obtaining gradient information of the smoothed Wasserstein GAN formulation, which is based on regularized Optimal Transport (OT), is computationally effortless and hence one can apply first order optimization methods to minimize this objective. Consequently, we establish theoretical convergence guarantee to stationarity for a proposed class of GAN optimization algorithms. Unlike the original non-smooth formulation, our algorithm only requires solving the discriminator to approximate optimality. We apply our method to learning MNIST digits as well as CIFAR-10 images. Our experiments show that our method is computationally efficient and generates images comparable to the state of the art algorithms given the same architecture and computational power.
SUMMARY  The authors investigate the task of training a Generative Adversarial Networks model based on optimal transport (OT) loss. They focus on regularized OT losses, and show that approximate gradients of these losses can be obtained by approximately solving regularized OT problem (Thm 4.1). As a consequence, a non-convex stochastic gradient method for minimizing this loss has a provable convergence rate to stationarity (Thm 4.2). The analysis also applies to Sinkhorn losses. The authors then explore numerically the behavior of a practical algorithm where the dual variable are parametrized by neural networks (the theory does not immediately apply because estimating the loss gradient becomes non-convex).  QUALITY I find that the point of view adopted by the authors is relevant. Training GAN is known to suffer from instabilities and here the authors provide theoretical tools to understand how to obtain gradients of the loss with a bounded error. While this relation between regularization and stability is not new in optimization/statistics, it seems to be invoked with proofs in this specific application for the first time. Yet, the authors provide numerical results on simple datasets which are interpretable.   CLARITY The article is well written, well organized, and easy to read.   ORIGINALITY The theoretical results are new in this setting but rely on well known principles. The numerical algorithm is, I believe, a minor variation of previously proposed Wasserstein GAN algorithms.  SIGNIFICANCE The paper does not provide a strikingly new point of view, but make reliable progress in the direction of robust training for GANs.  SPECIFIC REMARKS - l.130 "for practical considerations the dual functions belong to the set of parametric functions": this is likely to considerably change the (geometrical and statistical) properties of the distance and is instrumental. I would be in favor that the community as a whole emphasize more on the differences between these "parametrized Wasserstein" and the classical Wasserstein distances (I just state this as an opinion). - l.167 "although pi may not be a feasible transport plan": it is feasible, as a solution to Eq.(4). - l.174 again, I think the switch to parametric dual function cannot be reduced to a mere computational trick; - theorem 3.1 it seems that there is redundancy in the second and third conditions. The definition of L-Lipschitz smooth could maybe be given? - proof of Th.3.1 (in Appendix C): why is the 1-norm of plans appearing (Jensen's inequality should be explicitly invoked I guess) (l.449 typo  after nabla). - l.200 typo "the its" - l.272 I'd say that neural network falls also in the class of parametric approach. - l.273 I didn't understand why gradients are unbiased, is there any result supporting this claim ? This is a very important claim. - l.292 this is a very (!) strong conjecture... in any case, Lemma 4.3 is not supporting this conjecture since it is just a sufficient stationarity condition. - l.299 again, it is not predicted by Lemma 4.3 which does not characterize stationary points in general.  [UPDATE AFTER REBUTTAL I have read the reviews and authors' response. As reviewer #2, I thought that the setting was that of arbitrary probability measures (including "continuous" marginals with densities), as implied by the notations of the paper. I am not entirely convinced by the comment in author's response stating that "discrete distributions are the case in practice where distributions are represented by their samples" because the smoothness should hold for the "expected" risk, although the algorithm only uses samples. Looking at the proof, it is true that it lacks some details required to correctly deal with the continuous case (details to apply Danskin’s theorem, choices of topologies). Some technical difficulties might appear although I do not expect the results to be different. The authors in the rebuttal claim that they can adapt the proof to fix these issues. Therefore, I still lean towards acceptance although I am slightly less convinced than before, because of this confusion between continuous and discrete setting.]