Published as a conference paper at ICLR 2018 SEMANTIC INTERPOLATION IN IMPLICIT MODELS
In implicit models, one often interpolates between sampled points in latent space. As we show in this paper, care needs to be taken to match-up the distributional assumptions on code vectors with the geometry of the interpolating paths. Otherwise, typical assumptions about the quality and semantics of in-between points may not be justified. Based on our analysis we propose to modify the prior code distribution to put significantly more probability mass closer to the origin. As a result, linear interpolation paths are not only shortest paths, but they are also guaranteed to pass through high-density regions, irrespective of the dimensionality of the latent space. Experiments on standard benchmark image datasets demonstrate clear visual improvements in the quality of the generated samples and exhibit more meaningful interpolation paths.
The paper concerns distributions used for the code space in implicit models, e.g. VAEs and GANs. The authors analyze the relation between the latent space dimension and the normal distribution which is commonly used for the latent distribution. The well-known fact that probability mass concentrates in a shell of hyperspheres as the dimensionality grows is used to argue for the normal distribution being sub-optimal when interpolating between points in the latent space with straight lines. To correct this, the authors propose to use a Gamma-distribution for the norm of the latent space (and uniform angle distribution). This results in more mass closer to the origin, and the authors show both that the midpoint distribution is natural in terms of the KL divergence to the data points, and experimentally that the method gives visually appealing interpolations.
While the contribution of using a standard family of distributions in a standard implicit model setup is limited, the paper does make interesting observations, analyses and an attempt to correct the interpolation issue. The paper is clearly written and presents the theory and experimental results nicely. I find that the paper can be accepted but the incremental nature of the contribution prevents a higher score.