ON THE REGULARIZATION OF WASSERSTEIN GANS
Since their invention, generative adversarial networks (GANs) have become a popular approach for learning to model a distribution of real (unlabeled) data. Convergence problems during training are overcome by Wasserstein GANs which minimize the distance between the model and the empirical distribution in terms of a different metric, but thereby introduce a Lipschitz constraint into the optimization problem. A simple way to enforce the Lipschitz constraint on the class of functions, which can be modeled by the neural network, is weight clipping. Augmenting the loss by a regularization term that penalizes the deviation of the gradient norm of the critic (as a function of the network's input) from one, was proposed as an alternative that improves training. We present theoretical arguments why using a weaker regularization term enforcing the Lipschitz constraint is preferable. These arguments are supported by experimental results on several data sets.
This paper proposes a novel regularization scheme for Wasserstein GAN based on a relaxation of the constraints on the Lipschitz constant of 1. The proposed regularization penalize the critic function only when its gradient has a norm larger than one using some kind of squared hinge loss. The reasons for this choice are discussed and linked to theoretical properties of OT. Numerical experiments suggests that the proposed regularization leads to better posed optimization problem and even a slight advantage in terms of inception score on the CIFAR-10 dataset.
The paper is interesting and well written, the proposed regularization makes sens since it is basically a relaxation of the constraints and the numerical experiments also suggest it's a good idea. Still as discussed below the justification do not address a lots of interesting developments and implications of the method and should better discuss the relation with regularized optimal transport.
Discussion:
+ The paper spends a lot of time justifying the proposed method by discussing the limits of the "Improved training of Wasserstein GAN" from Gulrajani et al. (2017). The two limits (sampling from marginals instead of optimal coupling and differentiability of the critic) are interesting and indeed suggest that one can do better but the examples and observations are well known in OT and do not require proof in appendix. The reviewer believes that this space could be better spend discussing the theoretical implication of the proposed regularization (see next).
+ The proposed approach is a relaxation of the constraints on the dual variable for the OT problem. As a matter of fact we can clearly recognize a squared hinge loss is the proposed loss. This approach (relaxing a strong constraint) has been used for years when learning support vector machines and ranking and a small discussion or at least reference to those venerable methods would position the paper on a bigger picture.
+ The paper is rather vague on the reason to go from Eq. (6) to Eq. (7). (gradient approximation between samples to gradient on samples). Does it lead to better stability to choose one or the other? 
 How is it implemented in practice? recent NN toolbox can easily compute the exact gradient and use it for the penalization but this is not clearly discussed even in appendix. Numerical experiments comparing the two implementation or at least a discussion is necessary.
+ The proposed approach has a very strong relations to the recently proposed regularized OT (see [1] for a long list of regularizations) and more precisely to the euclidean regularization. I understand that GANS (and Wasserstein GAN) is a relatively young community and that references list can be short but their is a large number of papers discussing regularized optimal transport and how the resulting problems are easier to solve. A discussion of the links is necessary and will clearly bring more theoretical ground to the method. Note that a square euclidean regularization leads to a regularization term in the dual of the form max(0,f(x)+f(y)-|x-y|)^2 that is very similar to the proposed regularization. In other words the authors propose to do regularized OT (possibly with a new regularization term) and should discuss that.
+ The numerical experiments are encouraging but a bit short. The 2D example seem to work very well and the convergence curves are far better with the proposed regularization. But the real data CIFAR experiments are much less detailed with only a final inception score (very similar to the competing method) and no images even in appendix. The authors should also define (maybe in appendix) the conditional and unconditional inception scores and why they are important (and why only some of them are computed in Table 1).
+ This is more of a suggestion. The comparison of the dual critic to the true Wasserstein distance is very interesting. It would be nice to see the behavior for different values of lambda.
[1] Dessein, A., Papadakis, N., & Rouas, J. L. (2016). Regularized Optimal Transport and the Rot Mover's Distance. arXiv preprint arXiv:1610.06447.
Review update after reply:
The authors have responded to most of my concerns and I think the paper is much stronger now and discuss the relation with regularized OT. I change the rating to Accept.