This paper shows that deep learning (DL) representations of data produced by generative adversarial nets (GANs) are random vectors which fall within the class of so-called concentrated random vectors. Further exploiting the fact that Gram matrices, of the type G = X X with X = [x 1 , . . . , x n ] ∈ R p×n and x i independent concentrated random vectors from a mixture model, behave asymptotically (as n, p → ∞) as if the x i were drawn from a Gaussian mixture, suggests that DL representations of GAN-data can be fully described by their first two statistical moments for a wide range of standard classifiers. Our theoretical findings are validated by generating images with the BigGAN model and across different popular deep representation networks.
This paper provides a formal proof that the data produced by a GAN are concentrated vectors. The proof is rather intuitive: a GAN essentially consists of a Lipschitz function that takes as input a latent vector drawn from a Gaussian distribution and therefore one can use Lipschitz concentration bounds to prove that the transformed output satisfies a concentration property. Overall, I think this is a very intuitive idea and I fail to see the added value of such an observation. I provide detailed feedback below, I would really like to get a sensible answer regarding the novelty aspect.
Gaussian assumption
It is of course very common to sample the latent vectors from a Gaussian distribution, although one could potentially use a different distribution (uniform, gamma, …). To what extent could these results extend to other distributions?
Prior work
The paper does not appropriately discuss prior work that impose a Lipschitz constraint while training, see e.g.:
Gulrajani, Ishaan, et al. "Improved training of wasserstein gans." Advances in neural information processing systems. 2017.
Roth, Kevin, et al. "Stabilizing training of generative adversarial networks through regularization." Advances in neural information processing systems. 2017.
Theorem 3.3
I would like further clarifications regarding this theorem. This theorem is essentially a concentration bound for the resolvent of the gram matrix G around its mean. The expectation is already computed in (Louart & Couillet, 2019) so the contribution in this theorem seems to be in showing that the result only depends on the first and second order statistics. You claim this is a surprising result, although it does not seem so surprising to me given that this is an asymptotic result. Could you comment on deriving a non-asymptotic result instead?
Added value
As mentioned earlier, I do not see what particular insight is this paper bringing. There are some obvious connections that one could make, e.g. implications such as robustness to adversarial samples depending on the Lipschitz constant, or perhaps improve generalization property, but some of these connections are already made in prior work (see comment above). What particular insight do we get from your analysis?
Experiments
I think a more detailed study regarding the effect of various regularization schemes would be valuable. One could for instance compare various networks with/without batchnorm, resnet connections, ...