We consider the problem of training generative models with deep neural networks as generators, i.e. to map latent codes to data points. Whereas the dominant paradigm combines simple priors over codes with complex deterministic models, we argue that it might be advantageous to use more flexible code distributions. We demonstrate how these distributions can be induced directly from the data. The benefits include: more powerful generative models, better modeling of latent structure and explicit control of the degree of generalization.
The paper proposes, under the GAN setting, mapping real data points back to the latent space via the "generator reversal" procedure on a sample-by-sample basis (hence without the need of a shared recognition network) and then using this induced empirical distribution as the "ideal" prior targeting which yet another GAN network might be trained to produce a better prior for the original GAN.
I find this idea potentially interesting but am more concerned with the poorly explained motivation as well as some technical issues in how this idea is implemented, as detailed below.
1. Actually I find the entire notion of an "ideal" prior under the GAN setting a bit strange. To start with, GAN is already training the generator G to match the induced P_G(x) (from P(z)) with P_d(x), and hence by definition, under the generator G, there should be no better prior than P(z) itself (because any change of P(z) would then induce a different P_G(x) and hence only move away from the learning target).
I get it that maybe under different P(z) the difficulty of learning a good generator G can be different, and therefore one may wish to iterate between updating G (under the current P(z)) and updating P(z) (under the current G), and hopefully this process might converge to a better solution. But I feel this sounds like a new angle and not the one that is adopted by the authors in this paper.
2. I think the discussions around Eq. (1) are not well grounded. Just as you said right before presenting Eq. (1), typically the goal of learning a DGM is just to match Q_x with the true data distrubution P_x. It is **not** however to match Q(x,z) with P(x,z). And btw, don't you need to put E_z[ ... ] around the 2nd term on the r.h.s. ?
3. I find the paper mingles notions from GAN and VAE sometimes and misrepresents some of the key differences between the two.
E.g. in the beginning of the 2nd paragraph in Introduction, the authors write "Generative models like GANs, VAEs and others typically define a generative model via a deterministic generative mechanism or generator ...". While I think the use of a **deterministic** generator is probably one of the unique features of GAN, and that is certainly not the case with VAE, where typically people still need to specify an explicit probabilistic generative model.
And for this same reason, I find the multiple references of "a generative model P(x|z)" in this paper inaccurate and a bit misleading.
4. I'm not sure whether it makes good sense to apply an SVD decomposition to the \hat{z} vectors. It seems to me the variances \nu^2_i shall be directly estimated from \hat{z} as is. Otherwise, the reference "ideal" distribution would be modeling a **rotated** version of the \hat{z} samples, which imo only introduces unnecessary discrepancies.
5. I don't quite agree with the asserted "multi-modal structure" in Figure 2. Let's assume a 2d latent space, where each quadrant represents one MNIST digit (e.g. 1,2,3,4). You may observe a similar structure in this latent space yet still learn a good generator under even a standard 2d Gaussian prior. I guess my point is, a seemingly well-partitioned latent space doesn't bear an obvious correlation with a multi-modal distribution in it.
6. The generator reversal procedure needs to be carried out once for each data point separately, and also when the generator has been updated, which seems to be introducing a potentially significant bottleneck into the training process.