MMD GAN: Towards Deeper Understanding of Moment Matching Network
Generative moment matching network (GMMN) is a deep generative model that differs from Generative Adversarial Network (GAN) by replacing the discriminator in GAN with a two-sample test based on kernel maximum mean discrepancy (MMD). Although some theoretical guarantees of MMD have been studied, the empirical performance of GMMN is still not as competitive as that of GAN on challenging and large benchmark datasets. The computational efficiency of GMMN is also less desirable in comparison with GAN, partially due to its requirement for a rather large batch size during the training. In this paper, we propose to improve both the model expressiveness of GMMN and its computational efficiency by introducing adversarial kernel learning techniques, as the replacement of a fixed Gaussian kernel in the original GMMN. The new approach combines the key ideas in both GMMN and GAN, hence we name it MMD GAN. The new distance measure in MMD GAN is a meaningful loss that enjoys the advantage of weak ⇤ topology and can be optimized via gradient descent with relatively small batch sizes. In our evaluation on multiple benchmark datasets, including MNIST, CIFAR-10, CelebA and LSUN, the performance of MMD GAN significantly outperforms GMMN, and is competitive with other representative GAN works.
This paper presents the MMD-GAN, a generative architecture which combines the benefits of generative adversarial nets (GANs) and generative moment matching nets (GMMNs). The architecture is based on MMD, but with a kernel function learned by an adversary trying to maximize MMD. There's a bit of theory showing non-degeneracy, and experiments demonstrate that the generated samples are competitive with W-GANs.
Overall, I think this is a strong paper which presents a simple idea and executes it well. The proposed method is intuitively appealing. The modeling choices are well motivated, the writing is clear, the theoretical justification is nice, and relationships to other methods are clearly highlighted. The experiments compare against strong baselines on challenging datasets. Being able to make GMMNs competitive with WGANs is impressive, since to the best of my knowledge there was still a large performance gap between them. 
While the results seem significantly better than the baselines, the paper could use more analysis of where this improvement comes from. How does the MMD-GAN eliminate the need for large mini-batches?  (Don't the samples in each mini-batch still need to essentially tile the space in order to achieve low MMD?)  Is there an explanation for why it outperforms the WGAN?
Minor comments:
- Section 3 describes the method in relation to the GAN, but isn't the WGAN a more direct match?  If I understand right, the MMD-GAN is essentially a kernelized W-GAN. 
- "Also, the gradient has to be bounded, which can be done by clipping phi." Is this the same as the Lipschitz constraint from WGANs?  If so, why not use the regularization method from the "improved WGAN" paper, rather than the original version (which the authors found was less effective)?
- If the claim is that the proposed method prevents mode dropping, estimating log-likelihoods is probably a stronger quantitative way to test it. E.g., see Wu et al., 2017.
- The experiment of Section 5.3 doesn't seem to be measuring stability. (I assume this refers to avoiding the degenerate solutions that regular GANs fall into?)