DISTRIBUTION-INTERPOLATION TRADE OFF IN GENERATIVE MODELS
We investigate the properties of multidimensional probability distributions in the context of latent space prior distributions of implicit generative models. Our work revolves around the phenomena arising while decoding linear interpolations between two random latent vectors -regions of latent space in close proximity to the origin of the space are oversampled, which restricts the usability of linear interpolations as a tool to analyse the latent space. We show that the distribution mismatch can be eliminated completely by a proper choice of the latent probability distribution or using non-linear interpolations. We prove that there is a trade off between the interpolation being linear, and the latent distribution having even the most basic properties required for stable training, such as finite mean. We use the multidimensional Cauchy distribution as an example of the prior distribution, and also provide a general method of creating non-linear interpolations, that is easily applicable to a large family of commonly used latent distributions.
The authors study the problem of when the linear interpolant between two random variables follows the same distribution. This is related to the prior distribution of an implicit generative model. In the paper, the authors show that the Cauchy distribution has such a property, however due to the heavy-tails is not particularly useful. In addition, they propose a non-linear interpolation that naturally has this property.
Technically the paper in my opinion is solid. Also, the paper is ok-written, but I think it needs improvements (see comments).
Comments:
#1) In my opinion the motivation is not very clear and should be improved. In the paper is mentioned that the goal of shortest path interpolation is to get smooth transformations. So, in principle, I am really skeptical when the linear interpolant is utilized as the shortest path. Even then, what is the actual benefit of having the property that the linear interpolants follow the same distribution as the prior? How this is related to smoother transformations? What I understand is that, if we interpolate between several random samples, we will get less samples near the origin, and additionally, these samples will follow the prior? But how this induces smoothness in the overall transformation? I think this should be explained properly in the text i.e. why is it interesting to solve the proposed problem.
#2) From Observation 2.2. we should realize that the distribution matching property holds if the distribution has infinite mean? I think that this is implicitly mentioned in Section 2.2. paragraph 1, but I believe that it should be explicitly stated.
#3) Fig.1 does not show something interesting, and if it does it is not explained. In Fig. 2 I think that interpolations between the same images should be provided such that to have a direct comparison. Also, in Fig. 3 the norm of Z can be shown in order to be clear that the Cauchy distribution has the desired property. 
#4) Section 2.2. paragraph 6, first sentence. Here it is stated that the distribution "must be trivial or heavy-tailed". This refers only to the Cauchy distribution? Since earlier the condition was the infinite mean. How these are related? Needs clarification in the text.
#4) In Figure 4, I believe that the norms of the interpolants should be presented as well, such that to show if the desired property is true. Also in Figure 5, what we should see? What are the improvements when using the proposed non-linear interpolation?
Minor comments:
#1) Section 1.2. paragraph 2. For each trained model the latent space usually has different structure e.g. different untrained regions. So I believe that interpolations is not the proper way to compare different models.
#2) Section 1.3 paragraph 1, in my opinion the term "pathological" should be explained precisely here. So it makes clear to the reader what he should expect.
#3) Section 2.2. paragraph 2. The coordinate-wise implies that some Z_i are near zero and some others significantly larger? 
In generally, I like the presented analysis. However, I do not fully understand the motivation. I think that choosing the shortest path guarantees smooth transformations. I do not see why the distribution matching property provides smoother transformations. To my understanding, this is simply a way to generate less samples near the origin, but this does not directly means smoother transformations of the generated images. I believe that the motivation and the actual implications of the discussed property have to be explained better.