Published as a conference paper at ICLR 2018 LATENT SPACE ODDITY: ON THE CURVATURE OF DEEP GENERATIVE MODELS
Deep generative models provide a systematic way to learn nonlinear data distributions through a set of latent variables and a nonlinear "generator" function that maps latent points into the input space. The nonlinearity of the generator implies that the latent space gives a distorted view of the input space. Under mild conditions, we show that this distortion can be characterized by a stochastic Riemannian metric, and we demonstrate that distances and interpolants are significantly improved under this metric. This in turn improves probability distributions, sampling algorithms and clustering in the latent space. Our geometric analysis further reveals that current generators provide poor variance estimates and we propose a new generator architecture with vastly improved variance estimates. Results are demonstrated on convolutional and fully connected variational autoencoders, but the formalism easily generalizes to other deep generative models.
The paper makes an important observation: the generating function of a generative model (deep or not) induces a (stochastic) Riemannian metric tensor on the latent space. This metric might be the correct way to measure distances in the latent space, as opposed to the Euclidean distance.
While this seems obvious, I had actually always thought of the latent space as "unfolding" the data manifold as it exists in the output space. The authors propose a different view which is intriguing; however, they do not, to the best of my understand, give a definitive theoretical reason why the induced Riemannian metric is the correct choice over the Euclidean metric.
The paper correctly identifies an important problem with the way most deep generative models evaluate variance. However the solution proposed seems ad-hoc and not particularly related to the other parts of the paper. While the proposed variance estimation (using RBF networks) might work in some cases, I would love to see (perhaps in future work) a much more rigorous treatment of the subject.
Pros:
1. Interesting observation and mathematical development of a Riemannian metric on the latent space.
2. Good observation about the different roles of the mean and the variance in determining the geodesics: they tend to avoid areas of high variance.
3. Intriguing experiments and a good effort at visualizing and explaining them. I especially appreciate the interpolation and random walk experiments. These are hard to evaluate objectively, but the results to hint at the phenomena the authors describe when comparing Euclidean to Riemannian metrics in the latent space.
Cons:
1. The part of the paper proposing new variance estimators is ad-hoc and is not experimented with rigorously, comparing it to other methods in terms of calibration for example. 
Specific comments:
1. To the best of my understanding eq. (2) does not imply that the natural distance in Z is locally adaptive. I think of eq (2) as *defining* a type of distance on Z, that may or may not be natural. One could equally argue that the Euclidean distance on z is natural, and that this distance is then pushed forward by f to some induced distance over X. 
2. In the definition of paths \gamma, shouldn't they be parametrized by arc-length (also known as unit-speed)? How should we think of the curve \gamma(t^2) for example?
3. In Theorem 2, is the term "input dimension" appropriate? Perhaps "data dimension" is better?
4. I did not fully understand the role of the LAND model. Is this a model fit AFTER fitting the generative model, and is used to cluster Z like a GMM ? I would appreciate a clarification about the context of this model.