On gradient regularizers for MMD GANs
We propose a principled method for gradient-based regularization of the critic of GAN-like models trained by adversarially optimizing the kernel of a Maximum Mean Discrepancy (MMD). We show that controlling the gradient of the critic is vital to having a sensible loss function, and devise a method to enforce exact, analytical gradient constraints at no additional cost compared to existing approximate techniques based on additive regularizers. The new loss function is provably continuous, and experiments show that it stabilizes and accelerates training, giving image generation models that outperform state-of-the art methods on 160 ⇥ 160 CelebA and 64 ⇥ 64 unconditional ImageNet.
however, when the MMD kernel is not based directly on image pixels, but on learned features of images. Wasserstein-inspired gradient regularization approaches can be used on the MMD critic when learning these features: [27] uses weight clipping [3], and [5,7] use a gradient penalty [18].
The recent Sobolev GAN [33] uses a similar constraint on the expected gradient norm, but phrases it as estimating a Sobolev IPM rather than loosely approximating Wasserstein. This expectation can be taken over the same distribution as [18], but other measures are also proposed, such as (P + Q ✓ ) /2. A second recent approach, the spectrally normalized GAN [32], controls the Lipschitz constant of the critic by enforcing the spectral norms of the weight matrices to be 1. Gradient penalties also benefit GANs based on f -divergences [37]: for instance, the spectral normalization technique of [32] can be applied to the critic network of an f -GAN. Alternatively, a gradient penalty can be defined to approximate the effect of blurring P and Q ✓ with noise [40], which addresses the problem of non-overlapping support [2]. This approach has recently been shown to yield locally convergent optimization in some cases with non-continuous distributions, where the original GAN does not [30].
In this paper, we introduce a novel regularization for the MMD GAN critic of [5,7,27], which directly targets generator performance, rather than adopting regularization methods intended to approximate Wasserstein distances [3,18]. The new MMD regularizer derives from an approach widely used in semi-supervised learning [10,Section 2], where the aim is to define a classification function f which is positive on P (the positive class) and negative on Q ✓ (negative class), in the absence of labels on many of the samples. The decision boundary between the classes is assumed to be in a region of low density for both P and Q ✓ : f should therefore be flat where P and Q ✓ have support (areas with constant label), and have a larger slope in regions of low density. Bousquet et al. [10] propose as their regularizer on f a sum of the variance and a density-weighted gradient norm.
We adopt a related penalty on the MMD critic, with the difference that we only apply the penalty on P: thus, the critic is flatter where P has high mass, but does not vanish on the generator samples from Q ✓ (which we optimize). In excluding Q ✓ from the critic function constraint, we also avoid the concern raised by [32] that a critic depending on Q ✓ will change with the current minibatch -potentially leading to less stable learning. The resulting discrepancy is no longer an integral probability metric: it is asymmetric, and the critic function class depends on the target P being approximated.
We first discuss in Section 2 how MMD-based losses can be used to learn implicit generative models, and how a naive approach could fail. This motivates our new discrepancies, introduced in Section 3. Section 4 demonstrates that these losses outperform state-of-the-art models for image generation.
Summary:  The paper presents a gradient constrained MMD that generalizes  the Sobolev discrepancy  introduced in Sobolev GAN by adding an L_2 regularizer to the gradient regularizer (the kernel case was also discussed in the aforementioned paper). Computing this regularized  MMD is computationally expensive authors propose a bound on the this discrepancy , and optimize with the new scaled MMD , that changes the radius of the ball in the RKHS according to the kernel and its derivative statistics on the target distribution in this paper.  The paper then combines the Scaled MMD with the recently introduced spectral normalization in order to train GAN.   Comments:  - Line 88: phi_psi: X to R^s.  It seems throughout the paper that s is considered to be equal 1. It would be good to mention this early on. Have you tried s >1?  - Line 117 ||f||_{L} if this is the Lipchitz norm this is not correct one need to consider the definition with sup_{x,y} f(x)- f(y)/d(x,y), the one given is a bound.   - Line 208 the gaussian kernel is considered always with sigma =1?  - Table (2 b) SMMDGAN without SN is not given, it  would been nice to have it as well to tease apart the contribution of SN and the scaled MMD. Line 247 and appendix, I think some more ablation study is needed to understand how this scaling and the spectral normalization interact.( see point below on  tracking the RKHS norm and singular values with and without SN might help)  - Have you tracked the cost function in the training process? How does the numerator and denominator behave? It would be great to see how the radius in the RKHS behave in the training and how this relates to the plots of singular values you give in Figure 2 with SN and w/ SN.    - Line 242, on Sobolev GAN it seems that on CIFAR 10 there is an issue in the hyperparameter selection or the implementation  or the choice of mu as it is set in this paper to mu =P. In the reviewer experience Sobolev GAN behaves at least close to WGAN-GP on CIFAR 10.      Minor:   - The figure captions are too sparse, and one needs to jump to the text to know what is going on. It would be good to have better descriptive captions it helps in the readability of the paper.   -Figure 6 (a) and (b) are the same  - would be good to have on a toy data like the mixture of gaussians  to train the GAN with the original cost function given in proposition 4.