We propose a novel supervised learning method to optimize the kernel in maximum mean discrepancy generative adversarial networks (MMD GANs). Specifically, we characterize a distributionally robust optimization problem to compute a good distribution for the random feature model of Rahimi and Recht to approximate a good kernel function. Due to the fact that the distributional optimization is infinite dimensional, we consider a Monte-Carlo sample average approximation (SAA) to obtain a more tractable finite dimensional optimization problem. We subsequently leverage a particle stochastic gradient descent (SGD) method to solve the derived finite dimensional optimization problem. Based on a mean-field analysis, we then prove that the empirical distribution of the interactive particles system at each iteration of the SGD follows the path of the gradient descent flow on the Wasserstein manifold. We also establish the non-asymptotic consistency of the finite sample estimator. Our empirical evaluation on synthetic data-set as well as MNIST and CIFAR-10 benchmark data-sets indicates that our proposed MMD GAN model with kernel learning indeed attains higher inception scores well as Frèchet inception distances and generates better images compared to the generative moment matching network (GMMN) and MMD GAN with untrained kernels.
This paper proposes to learn a kernel for training MMD-GAN by optimizing over the probability distribution that defines the kernel by means of random features. This is unlike the usual setting of MMD-GAN where the kernel is parametrized by composing a fixed top-kernel with a discriminator network that is optimized during training. The main motivation for this approach is to avoid having to 'manually' fix some parameters of the top-level kernel like the bandwidth of the RBF kernel. They provide an algorithm to achieve such optimization in probability space along with some consistency results and perform experiments on MNIST and Cifar10 to demonstrate empirically the advantage of such an approach over those that fix the top-level kernel.
Theory:
	Theorem 4.1 provides a convergence result of an oracle finite-sample estimator: that is the one obtained by exactly solving the optimization problem in 19b. In that case, they show the consistency of the proposed estimator. The result is somehow expected but the proof relies on nice duality results for measures and is very technical.	
The clarity of the proof could be improved:
- Currently, the structure of the main proof mixes direct lemma (lemma B.7) with less obvious ones (lemma B.6). Also, some concepts are introduced in the main proof but not necessary for its understanding: The notion of Orlicz norm is in introduced in Definition B.3 on the fly to state lemma B.6, but only equations 50 and 51 are used in this lemma which does not make use of the notion of Orlicz norm at all.
	Theorem 4.1 doesn't say anything about the consistency of the algorithm itself. To partly address this, the authors show in theorem 4.2 that as the number of particles grows the empirical process converges to a McKean Vlasov PDE (equation 22). This means that the proposed algorithm is approximating some gradient flow in metric space (25). 
- However, this gradient flow is a non-convex optimization problem and there is no guarantee that a global solution is reached.  Recent work provides cases when global convergence occurs [Chizat2018] but it is not the case in general. Some further clarification about the connection between Thm 4.1 and Thm 4.2 would be therefore useful.
	
Thm 4.2 is also curious in the sense that the process defined by equation 16, which is noisy since it relies on one sample from the data, would converge towards (22) which is a Mc-valsov equation with a drift only (no diffusion or other noise).  What happened to the noise coming from sampling from P_v  and P_w in equation 16? Wouldn't there be some sort of diffusion term as in [Hu2018]?
	
More generally it would be nice to have a discussion of the assumptions and results in the paper as they seem to rely on methods that people in the machine learning community are not totally familiar with.
	
Experiments:  The experiments are not convincing for several reasons:
 - The comparison with the other methods is somehow unfair since the bandwidth is manually tuned for the competing methods. It is easy to adaptively learn the bandwidth as well:  in this case, it will be just an additional parameter of a discriminator network. This was done in [Arbel2018] where a single gaussian kernel is used and a regularization of the critic allows to learn the bandwidth without manual tuning. Does the proposed method offer an additional advantage compared to those?
- In practice and for a scaling parameter alpha=1, isn't the algorithm strictly equivalent to considering an MMD-GAN with a dot product kernel and a discriminator given by the feature \phi(x,\zeta)?
- Mnist and Cifar10 are somehow very simple, what would happen on more complicated datasets (CelebA or imagenet)?
- I also think there is an ablation that is missing: If the auto-encoder also needs to be optimized then does it also help to optimize over the particles as well or is optimizing the auto-encoder discriminator enough to achieve a similar performance? In other words, does optimizing the auto-encoder compensate for the need to learn the distribution mu? Of course, this would depend on how the auto-encoder is parametrized but I don't see why it wouldn't in many cases.
Overall, I'm not convinced that the proposed approach would lead to any substantial improvement for MMD-GAN in practice, and the experiments are not really convincing as they are now. However, I find the theoretical results interesting and might be used to better analyse the dynamics of GAN's. But as the paper is currently framed, it is hard to put these theoretical results in perspective.