GILBO: One Metric to Measure Them All
We propose a simple, tractable lower bound on the mutual information contained in the joint generative density of any latent variable generative model: the GILBO (Generative Information Lower BOund). It offers a data-independent measure of the complexity of the learned latent variable description, giving the log of the effective description length. It is well-defined for both VAEs and GANs. We compute the GILBO for 800 GANs and VAEs each trained on four datasets (MNIST, FashionMNIST, CIFAR-10 and CelebA) and discuss the results.
[Response to rebuttal: I am happy and thoroughly satisfied with the author's response to the reviews, which is why I have increased my rating. I only want to touch upon one aspect once more: Reviewer 2 has raised an interesting issue (and in my view the authors have responded well and convincingly), namely the problem that the continuous MI might diverge for a differentiable deterministic model. The paper might benefit from explicitly including and addressing this issue, at least in the appendix. The issue is trivially resolved by acknowledging the fact that we perform computation with limited bit-precision, so the authors might even choose to make this explicit by replacing integrals with sums in their equations. For the continuous case, the problem remains when using a deterministic, reversible generator and I think it is interesting to comment on this in the paper and discuss how (and potentially under what circumstances) typical Relu networks tend to produce non-reversible functions ("many-to-one" mapping) while still being (mostly) differentiable. A full investigation of this is beyond the scope of the paper, but it might spark interest to look into this in some future work and thoroughly investigate what kind of compression GAN decoders perform (in the continuous limit).]  The paper introduces a new metric to quantitatively characterize the complexity of a generative latent-variable model with the ultimate goal of providing a performance indicator to quantitatively characterize generative models obtained from both GANs and VAEs. The aim is to quantify the mutual information between the latent variable z and the generated data x. This quantity is interesting since it provides a training-data independent measure, is invariant under re-parametrization and has interesting and well-known information-theoretic properties and interpretations. Since the computation of the mutual information involves the intractable posterior p(x|z), the paper proposes to instead consider a lower bound (GILBO) based on a variational approximation of the posterior instead. The measure is demonstrated in an exhaustive experimental study, where 800 models (VAEs and GANs) are compared on four different datasets. The authors find that the proposed measure provides important insight into the complexity of the generative model which is not captured by previously proposed data-dependent and sample-quality focused measures such as the FID score. The GILBO allows to qualitatively divide trained models into different regimes such as “data memorization” or “failure to capture all classes in latent space”. Additionally the GILBO highlights some qualitative differences between GANs and VAEs. The paper concludes by addressing some potential shortcomings of the method: reproducibility of quantitative values, tightness of lower bound, quality of variational approximation and consistency with qualitative “behavior” of the generative model.  The recent literature reports several attempts at objectively comparing the quality of complex generative models obtained from GANs or VAEs. While many methods so far focused on capturing the match between the true and the generated data-distribution or quantitative measures that correlate well with visual quality of generated samples, these measures seem to be insufficient to fully capture all important aspects of a generative model. Reporting the GILBO as an additional quantity that characterizes complexity of the generative model is a novel and original proposal. The paper is generally well written (except for some hiccups at the beginning of section 2) and the experimental section is impressive (certainly in terms of invested computation). Main points of criticism regarding the method are already addressed in the paper. For these reasons, I argue for accepting the paper. My comments below are meant as suggestions to authors for further increasing the quality of the paper.  (1 Intro of the GILBO in Sec. 2) Section 2 (lines 23 to 48) could use some polishing and additional details, particularly for readers unfamiliar with standard notation in generative latent-variable models. In particular: -) please explain what the variables X and Z correspond to in the context of generative models, ideally even mention that such a generative model consists of a prior over the latent variable and a (deterministic) generator p(x|z) which is a deep neural network in GAN and VAE. -) line(27): I guess to be precise, it is the reduction in the number of questions to guess X if you observe Z and know p(X). -) line (32): I think it would not hurt if you (in-line) add the equation for marginalization to make it obvious why p(x) is intractable. Also consider giving the equation for the posterior p(z|x) as you have it in line 34 before mentioning p(x), since p(x) does not appear in Eq. (1) but p(z|x) does. This might help readers follow the reasoning more easily. -) Eq (2): perhaps consider being explicit about the fact that GILBO is a function which depends on the parameters of e(z|x), for instance by writing GILBO(\theta)=… and then using e_\theta(z|x) or similar. -) line (42 – 44): “maximizing the likelihood…” – it could be helpful to write down the same as an equation, just to make it precise for readers.  (2 Proofs) There are two arguments that you might want to back up with a proof (e.g. in the supplementary material). Though the proofs should not be too difficult, some readers might appreciate them and they are actually central statements for the appealing properties of the GILBO. Additionally it might be beneficial because it would require being more specific about the technical conditions that need to be fulfilled to get the appealing properties of GILBO.  -) line 46: show that the GILBO is actually a lower bound to the mutual information. -)line 48: show that failure to converge, etc. simply makes the bound looser, i.e. show that Eq. (2) cannot diverge due to some ill-parameterized e(z|x).  (3 Discussion) Three more issues that you might want to include in your discussion. -) Do you think it would be possible/computationally feasible to replace the variational approximation of the posterior with some sampling-approximation, which would in principle allow for arbitrary precision and thus guarantee convergence to the actual mutual information (while computationally expensive, this could be used to compute a gold-standard compared to the variational approximation)? -) How sensitive is the GILBO to different settings of the optimization process (optimizer, hyper-parameters)? Ideally, this answer is backed up by some simulations. This would be of practical importance, since comparing GILBOs of different models could in principle require following very closely the same procedure (and of course encoder architecture) across papers. -) Besides qualitative “sanity-checks” (as in Fig. 2: which regime does my generative model fall into), how do you propose to use the GILBO to quantitatively compare models. Say I have two models with different GILBO (but in the same qualitative regime) and different FID score – which model is favorable, higher GILBO, higher FID, …?  Misc: -) Footnote 1, last sentence: … it could also achieve a looser bound -) Figure 1a: Any specific reason for a different color-map here, otherwise please use the same color-map as in the other plots. -) Fig. 7 caption vs. line 133: did you use 128 or 127 samples? -) Fig. 8 caption vs. line 165/166: did you use 16 or 15 samples? -) citations: I quite like the author, year citations (even though they “cost” extra space in a conference paper) – please check that all your citations have the year properly set (see line 194, where it is missing) -) line 207/208: Reporting the GILBO as an additional measure could be a good idea, please be a bit more specific that this would require fixing the encoder architecture and optimization procedure as well and comment on practical aspects such as whether you would use a different encoder architecture per data-set or the same across many data-sets, etc.