Conditional Generative Moment-Matching Networks
Maximum mean discrepancy (MMD) has been successfully applied to learn deep generative models for characterizing a joint distribution of variables via kernel mean embedding. In this paper, we present conditional generative momentmatching networks (CGMMN), which learn a conditional distribution given some input variables based on a conditional maximum mean discrepancy (CMMD) criterion. The learning is performed by stochastic gradient descent with the gradient calculated by back-propagation. We evaluate CGMMN on a wide range of tasks, including predictive modeling, contextual generation, and Bayesian dark knowledge, which distills knowledge from a Bayesian model by learning a relatively small CGMMN student network. Our results demonstrate competitive performance in all the tasks.
This paper presents conditional generative moment matching networks, an extension of GMMNs to conditional generation / prediction applications.  The key to the proposed method is the kernel embedding of conditional distributions and conditional MMD metric for measuring discrepancy of conditional distributions.  The proposed CGMMN was tested on classification, conditional generation and distilling Bayesian models.
The naive approach of extending GMMNs to conditional setting is to estimate a GMMN for each conditional distribution, and all these conditional distributions share parameters through the use of the same neural network.  The problem of this approach is that each conditional distribution only has very few examples, and in the case of continuous domain for the conditioning variables, each conditional distribution may only have one single example, causing data sparsity problem.  The proposed approach treats all the conditional distributions as a family and tries to match the model with the conditional embedding operator directly rather than matching each individual conditional distributions.  The advantage of the proposed approach seems clear, but in some cases I can still see the naive approach do a reasonable job, for example in conditional generation where the conditioning variable takes one of 10 values as in MNIST.  It would be interesting to compare to such a naive approach as a baseline.  It is nice to see that the conditional MMD objective has a nice formulation, and can be easily computed almost in the same way as the unconditional MMD as pointed out in Remark 1.  All the K matrices in equation 2 are independent from the model parameters, and backpropagation can be done in the same way as before.  One concern about the proposed approach is that the CMMD objective potentially requires big minibatches, otherwise the stochasticity of the minibatches may dominate the error, on the other hand big minibatches makes the computation of the objective expensive, especially the matrix inverses could be prohibitively expensive to compute.  Overall I think the CMMD provides a nice way to model conditional distributions.  The experiment results on classification and distillation of Bayesian models are also interesting.  The presentation in section 2.3 and 3.1 wasn’t easy to follow.  I have to dive into the referenced papers to understand more about what is actually going on.  There are a couple minor problems:  (1) on line 77, it is mentioned that the MMD estimator is unbiased, which is not true. (2) on line 90, the conditional embedding operator was not consistent with the one given in reference [29].  To get the form used in this paper it seems we need a different definition of the cross covariance operator.