OPTIMAL TRANSPORT MAPS FOR DISTRIBUTION PRESERVING OPERATIONS ON LATENT SPACES OF GENERATIVE MODELS
Generative models such as Variational Auto Encoders (VAEs) and Generative Adversarial Networks (GANs) are typically trained for a fixed prior distribution in the latent space, such as uniform or Gaussian. After a trained model is obtained, one can sample the Generator in various forms for exploration and understanding, such as interpolating between two samples, sampling in the vicinity of a sample or exploring differences between a pair of samples applied to a third sample. However, the latent space operations commonly used in the literature so far induce a distribution mismatch between the resulting outputs and the prior distribution the model was trained on. Previous works have attempted to reduce this mismatch with heuristic modification to the operations or by changing the latent distribution and re-training models. In this paper, we propose a framework for modifying the latent space operations such that the distribution mismatch is fully eliminated. Our approach is based on optimal transport maps, which adapt the latent space operations such that they fully match the prior distribution, while minimally modifying the original operation. Our matched operations are readily obtained for the commonly used operations and distributions and require no adjustment to the training procedure.
Noticing that widely used latent code interpolations for exploring the generative capabilities of VAEs and GANs have distribution mismatch problems, this paper proposes to utilize monotone transport map to exactly eliminate the distribution mismatch between modified interpolated codes and a prior distribution, assuming I.I.D. code components and a L1 code distance. More precisely, a transformation of the latent space operation is learnt with the objective that the distribution of the transformed variable match the prior distribution used in training the generative models. Optimal transport is used as a measure to minimize the two distributions. By restricting the class of cost functions used in the optimal transport formulation, the solution to the optimal transport problem (and hence the transformation function) has been shown to take a simple form (closed form in cases where cdf has a analytical form). Experiments on CIFAR-10, LLD-icon, LSUN, CelebA datasets show that, the minimally modified interpolated codes for several different interpolations produce samples with higher Inception Scores and better visual effects under an improved Wasserstein GAN than the original interpolated codes.
This paper is well written, the studied problem is highly important, and the approach presented has potentially wide applications. 
However, there are some concerns about the experimental evaluations,
1. Although the quantitative evaluations for 2-point and 4-point interpolations are important, it is hard to assess these interpolations in a semantically meaningful way. Extensive quantitative (FID and IS) and qualitative evaluations should be conducted for analogy interpolations. For example, adding glasses, adding mustache, and many others. It is much easier to assess the quality of the generated images from the minimally modified interpolated code for this category in a meaningful way.
2. Another concern is that how big the effect of the transformation function inducing on the latent space operations will be. For example, a linear interpolation is no longer linear after getting transformed. So, are there transformations that drastically transform the original latent space operations? In that case, will the transformed variable make any sense with respect to the original latent space operations? Extensive experiments for analogy interpolations are required to answer these questions.
3. Experiments have been shown only on GAN architectures, however, the framework can be easily extended to VAEs. Experiments on VAEs will be informative.
Minor:
Section 1.1, in the second paragraph, (SLERP) should be moved a correct position.
Figure 2: it's better to use a different color for midpoint linear other than blue
Problem 1, f* ---> f*: