INFORMATION THEORETIC LOWER BOUNDS ON NEGATIVE LOG LIKELIHOOD
In this article we use rate-distortion theory, a branch of information theory devoted to the problem of lossy compression, to shed light on an important problem in latent variable modeling of data: is there room to improve the model? One way to address this question is to find an upper bound on the probability (equivalently a lower bound on the negative log likelihood) that the model can assign to some data as one varies the prior and/or the likelihood function in a latent variable model. The core of our contribution is to formally show that the problem of optimizing priors in latent variable models is exactly an instance of the variational optimization problem that information theorists solve when computing ratedistortion functions, and then to use this to derive a lower bound on negative log likelihood. Moreover, we will show that if changing the prior can improve the log likelihood, then there is a way to change the likelihood function instead and attain the same log likelihood, and thus rate-distortion theory is of relevance to both optimizing priors as well as optimizing likelihood functions. We will experimentally argue for the usefulness of quantities derived from rate-distortion theory in latent variable modeling by applying them to a problem in image modeling.
(Please find my response to the rebuttal and updated version in a comment below)
The paper analyses latent-variable modeling from a rate-distortion point-of-view in a novel and interesting fashion, highlighting important fundamental connections. In particular, the paper presents a novel theorem (inspired by how the rate-distortion function is computed) that gives a lower bound on the negative log likelihood. This lower bound allows to quantify by how much a latent-variable model could be improved by either modifying the prior or the likelihood function. The latter is important, since the paper shows a duality between improving one while keeping the other fixed and vice versa. Finally, the paper derives a practical implementation/approximation (founded on solid theoretical analysis) of quantifying the improvement potential of a latent-variable model. These, so called “glossy statistics” are quantitatively analyzed in a set of experiments with different variational autoencoder architectures (various likelihood models and priors) on a number of datasets.
The main contribution of this paper is to provide novel proofs and theoretical analysis that connect latent-variable modeling with rate-distortion on a very fundamental level. While similar attempts have been reported in the recent literature (perhaps a bit more focused on the empirical aspects), the analysis and results in the paper follow a very fundamental treatment of rate-distortion theory and in particular of computation of the rate-distortion function. The central idea underlying rate-distortion, i.e. lossy compression by discarding irrelevant information, seems very suitable as a guiding principle for representation learning. In particular, learning representations that generalize well is essentially another instance of a lossy compression problem. The paper thus addresses an important and timely topic which should be of broad interest to the representation learning community. The paper is well written and mathematically rigorous. I have checked most parts of the proofs, though there still is a chance that I missed something. I am not entirely convinced by the practical impact of the experimental section of the paper (though the experiments are beyond toy-level and I do not doubt the results), but I also believe that this is not the main contribution of the paper, which is rather laying the mathematical groundwork for future work. I vote and argue for accepting the paper for presentation at the conference. My criticism below is aimed at giving some pointers for potentially improving the paper.
1) As the paper acknowledges, there is a risk of overfitting when improving likelihood functions under fixed priors (and vice versa). While the glossy statistics certainly allow making approximate statements of whether the model can be improved further or not, there is no “threshold value” or other guideline that would indicate a modeling expert that they are entering an over-fitting regime if the model-class is further enriched. Therefore, I am not sure about the practical impact of the experiments: the glossy statistics seem to be indicative of the margin for improvements in the negative log-likelihood - but whether all of these improvements are really desirable is unclear. To test this, one might resort to tasks other than generative modeling, such that models that overfit can easily be “spotted” (by degrading test-set performance).
2) Rate-Distortion can be “made more robust” against overfitting by different choices of \alpha (essentially limiting channel capacity). Maybe I am missing something, but shouldn’t the \alpha carry over into the computation of the ratios for c(z)? Was it just assumed to be 1? The same question for Theorem 2 and the equation just above Theorem 2 - does the alpha drop (is it absorbed into the likelihood) or was it set to one? It might be interesting to see how the glossy statistics behave if \alpha is considered a hyper-parameter of the model, e.g. under “low capacity” do the glossy statistics “flatten out” very early?
3) I would have been excited to see how the glossy statistics evolve during training of a model - it would be interesting to show that the statistics initially predict a large margin of improvement that reduces and slowly flattens out as training converges.
4) In the paragraph after Eq. 14: the argument hinges on the possibility of having an invertible (and continuously differentiable) g(z). To me it is not straightforward that a neural network would necessarily implement such a function (particularly the invertibility might be problematic). Is this just a technical condition required for the formal statement, or do you think that this issue could become problematic in practice as well such that the duality between improving prior and likelihood does not hold any longer?
Minor:
5) I think the Alemi et al. reference (first reference) has been published under a different name (Fixing a Broken ELBO) at this years’ ICML.
6) Consider calling the quantity l(x|z) below Eq. 1 “the likelihood of the latent variable given the data” (since the data is given, even though the data is not in the conditional, which is why it is a likelihood function).
7) Rather than using “the KL divergence between”, use “the KL divergence from … to” which nicely reflects its asymmetry. 
8) Page 4, last Equation: square brackets for E_X missing