The goal of generative models is to model the underlying data distribution of a sample based dataset. Our intuition is that an accurate model should in principle also include the sample based dataset as part of its induced probability distribution. To investigate this, we look at fully trained generative models using the Generative Adversarial Networks (GAN) framework and analyze the resulting generator on its ability to memorize the dataset. Further, we show that the size of the initial latent space is paramount to allow for an accurate reconstruction of the training data. This gives us a link to compression theory, where Autoencoders (AE) are used to lower bound the reconstruction capabilities of our generative model. Here, we observe similar results to the perception-distortion tradeoff (Blau & Michaeli (2018)). Given a small latent space, the AE produces low quality and the GAN produces high quality outputs from a perceptual viewpoint. In contrast, the distortion error is smaller for the AE. By increasing the dimensionality of the latent space the distortion decreases for both models, but the perceptual quality only increases for the AE.
Impact of the latent space on the ability of GANs to fit the distribution
This paper purports to study the behavior of latent variable generative models by examining how the dimensionality of the latent space affects the ability of said models to reconstruct samples from the dataset. The authors perform experiments with deterministic autoencoders and WGAN-GP on CIFAR, CelebA, and random noise, and measure MSE and FID as a function of the dimensionality of z.
My take: This paper does not offer any especially intriguing insights, and many of the conclusions the authors draw are, in my opinion, not supported by their experiments. The paper is confusingly written and hard to follow—throughout my read I struggled to determine what the authors meant, and it was not clear to me what this paper is supposed to contribute. The potential impact of this paper is very low, and I argue strongly in favor of rejection.
Notes:
My most critical complaint is the central experiment set of the paper: measuring MSE and FID as a function of Dim-Z for two models. First of all, the authors assume that the reduction in MSE as a function of dim Z is indicative of increased memorization in the GAN models. I disagree that this is the case; since the GAN-based reconstruction is done via optimization it is unsurprising that increasing dim Z increases the reconstruction quality, as you are literally giving the model more degrees of freedom with which to fit the individual samples. This is glaringly evident in Figure 8, where increasing dim Z renders the model better able to reconstruct a sample of pure noise, which is almost certainly not in its normal output distribution, (or if it is, is in there with staggeringly low probability). The fact that the higher dim-z models are better able to reconstruct the noise supports the notion that it is merely the number of degrees of freedom that matter in these experiments, rather than what the model actually learns.
Second, it is important to note that FID can be easily gamed by memorization, and for an autoencoder (which has direct access to samples) with an increasingly large bottleneck it is unsurprising that increasing dim-Z tends to decrease the FID, and equally unsurprising that increasing the dim-Z for the GAN does not tend to improve results, since this does not really allow the model increased memorization capacity (not to mention the relationship between performance and dim-Z has been explored before in GAN papers).
Third, the organization of the experimental section makes it very difficult to infer what the authors are trying to conclude from these experiments. The noise experiment is presented, but no insights or conclusions are drawn, other than (a) noting that the model has a harder time reconstructing the noise than training samples and (b) that lower dim models have a harder time reconstructing the noise, both of which are just restatements of the information presented in the figure rather than an actual insight or conclusion.
-I’m not really sure what the experiment in section 3 is supposed to show. This experiment is poorly described and lacking details. First of all, what is the loss function used there? Is this the output of the discriminator or the MSE between the output and a target sample? How is z* found and what does it represent—is it just a randomly sampled latent, or is it the latent that corresponds to the z-value which minimizes some MSE loss for a target sample? If it’s the latter, why is this notation not introduced until section 4? If it’s a latent, why are you calling it a data point? Why are there no axes and no scales on these plots? How is it clear that there is an optimization path from z0 to z*; is that supposed to be inferred from z0 having a higher value than z* or appearing to be directly uphill from z*, because it’s not clear to me that that is the case in Figure 2a. In general I did not find this experiment to support the conclusions the authors draw.  
-Figure 4: It is important to note that FID can be trivially gamed by memorizing the dataset, and an autoencoder is much more well-suited to memorizing the dataset as it has direct access to samples (whereas a GAN must get them through the filter of the discriminator). Authors should test interpolation or hold-out likelihood for the  autoencoder, these models are not directly comparable in this manner.
-The presentation of this paper is, in general, all over the place. The authors should focus on writing such that each point follows the next, building progressively towards their results and insights, and making it easy for a reader to follow their train of thought.
“In this work, we show that by reducing the problem to a compression task, we can give a lower bound on the required capacity and latent space dimensionality of the generator network for the distribution estimation task.” At what point is this lower bound (either in terms of model capacity or latent space dimensionality) specified in the paper? Is Figure 3 supposed to be this lower bound, because to me it only indicates that the autoencoder tends to have a lower MSE, not that it conclusively lower bounds the memorization capacity of the GAN. Wouldn't a method like GLO which directly optimizes for memorization be a better lower bound for this, anyhow?
“We rely on the assumption, that less capacity is needed to reconstruct the training set, than to reconstruct the entire distribution” What does this phrase mean? Are the authors referring to the entire distribution of natural images, of which the training set is assumed to be a subset? Or do they mean the output distribution of the generator? This was not clear to me.
Minor:
-“ style transfer by Karras et al. (2018),”, and “anomaly detection (Shocher 2018).” StyleGAN is not a style transfer paper, and InGAN  is not about anomaly detection. Please do not incorrectly summarize papers.
-“Trained GAN newtworks” While amusing, this is a typo. Please thoroughly read your paper and correct all typos and grammatical mistakes, like “combiniation.”
-“…that an accurate reconstruction of the generator manifold is possible works using first order methods”  The word “works” seems to be out of place here. Again, please thoroughly proofread your paper.
-The legend in Figure 2 has a white background, making the white x corresponding to z0 invisible. Please fix this, and add appropriate axes to this plot.
-Figure 7 and 8 may in fact have error bars, but they are not described (are they 1 std or another interval?) or referenced, and in Figure 8 (if these are error bars) they are nearly invisible.