Published as a conference paper at ICLR 2018 SOBOLEV GAN
We propose a new Integral Probability Metric (IPM) between distributions: the Sobolev IPM. The Sobolev IPM compares the mean discrepancy of two distributions for functions (critic) restricted to a Sobolev ball defined with respect to a dominant measure µ. We show that the Sobolev IPM compares two distributions in high dimensions based on weighted conditional Cumulative Distribution Functions (CDF) of each coordinate on a leave one out basis. The Dominant measure µ plays a crucial role as it defines the support on which conditional CDFs are compared. Sobolev IPM can be seen as an extension of the one dimensional VonMises Cramér statistics to high dimensional distributions. We show how Sobolev IPM can be used to train Generative Adversarial Networks (GANs). We then exploit the intrinsic conditioning implied by Sobolev IPM in text generation. Finally we show that a variant of Sobolev GAN achieves competitive results in semisupervised learning on CIFAR-10, thanks to the smoothness enforced on the critic by Sobolev GAN which relates to Laplacian regularization.
The paper deals with the increasingly popular GAN approach to constructing generative models.  Following the first formulation of GANs in 2014, it was soon realized that the training dynamics was highly unstable, leading to significant difficulties in achieving stable results. The paper by Arjovsky et al (2017) provided a framework based on the Wasserstein distance, a distance measure between probability distributions belonging to the class of so-called Integral Probability Metrics (IPMs). This approach solved the stability issues of GANs and demonstrated improved empirical results. Several other works were then developed to deal with these stability issues, specifically the Fisher IPM. Both these methods relied on discriminating between distributions P and Q based on computing a function f, belonging to an appropriate function class {\cal F}, that maximizes the deviation E_{x~P}f(x)-E_{x~Q}f(x). The main issue relates to the choice of the class {\cal F}. For the Wasserstein distance this was the class of L_1 Lipschitz functions, while for the Fisher distance it was the class of square integrable functions. The present paper introduces a new notion of distance, where {\cal F} is the defined through the Sobolev norm, based on the L_2 norm of the gradient of f(x), with respect to a measure \mu(x), where the latter can be freely chosen under certain assumptions. 
The authors prove a theorem related to the properties of the Sobolev norm, and express it in terms of the component-wise conditional distributions. Moreover, they show that the optimal critic f is obtained by solving a PDE subject to zero boundary conditions. They then use their suggested metric in order to develop a GAN algorithm, and present experimental results demonstrating its utility. The Sobolev IPM has two nice features. First, it is based on the component-wise conditional distribution of the CDFs, and, second, its relation to the Laplacian regularizer from manifold learning. Its 1D version also relates to the well-known von Mises Cramer statistics used in hypothesis testing. 
The paper belongs to a class of recent papers attempting to suggest improvements to the original GAN algorithm, relying on the KL divergence. It is well conceived and articulated, and provides an interesting and potentially powerful new direction to improve GANs in practice. However, it is somewhat difficult to follow the paper, and would urge the authors to improve and augment their presentation of the following issues. 
1)	One often poses regularization schemes based on optimality criteria. Is there any optimality principle under which the Sobolev IPM is a desired choice? 
2)	The authors argue that their approach is especially well suited for discrete sequential data. This issue was not clear to me, and it would be good if the authors could expand on this issue and provide a clearer explanation. 
3)	How would the Sobolev norm behave under a change of coordinates or a homeomorphism of the space? Would it make sense to require some invariance in this respect? 
4)	The Lagrangian in eq. (9) contains both a Lagrange constraint on the Sobolev norm and a penalty term. Why are both needed? Why do the updates of  \lambda and p in Algorithm 1 used different schemes (SGD and ADAM, respectively). 
5)	Table 2, p. 13 – it would be nice to see a comparison to the recently introduced gradient penalty approach, Gulrajani et al., Improved training of wasserstein gans. arXiv preprint arXiv:1704.00028, 2017.
6)	The integral defining F_p(x) on p. 3 has x as an argument on the LHS and as an integrand of the RHS. Please correct this. Also specify that x=(x_1,\ldots,x_d).