Leveraging the Exact Likelihood of Deep Latent Variable Models
Deep latent variable models (DLVMs) combine the approximation abilities of deep neural networks and the statistical foundations of generative models. Variational methods are commonly used for inference; however, the exact likelihood of these models has been largely overlooked. The purpose of this work is to study the general properties of this quantity and to show how they can be leveraged in practice. We focus on important inferential problems that rely on the likelihood: estimation and missing data imputation. First, we investigate maximum likelihood estimation for DLVMs: in particular, we show that most unconstrained models used for continuous data have an unbounded likelihood function. This problematic behaviour is demonstrated to be a source of mode collapse. We also show how to ensure the existence of maximum likelihood estimates, and draw useful connections with nonparametric mixture models. Finally, we describe an algorithm for missing data imputation using the exact conditional likelihood of a DLVM. On several data sets, our algorithm consistently and significantly outperforms the usual imputation scheme used for DLVMs.
Updated Review after Rebuttal: After reading the authors response and re-evaluate the paper I do agree that most of my concerns that there was a fundamental issue with some of their statements were wrong, hence I'm changing my score from 3 to 6. From going into detail of the proof it resides on constructing a generative model where for half of the latent variables (w^t z < 0) the integral is bounded for all data points and for the other half for 1 data point the integral diverges while for the other goes to zero. This split allows them to say that the one integral diverges and the all of the others are finite hence the likelihood is infinite.  However, I'm still not convinced that this issue actually arises at all in practical settings.  First, in practice, we are optimizing an ELBO which is never tight, hence for this to be convincing argument the authors should investigate whether there are settings of the ELBO where it diverges except when it can perfectly reconstruct the posterior.  Furthermore, I still stand that I do not think that the results on the Frey Faces dataset are interpreted correctly and given that this is a fairly small dataset it is highly likely that the generative model overfits to the data (but not in the way for the divergence to happen).  The experimental section in this direction seems to be a bit weak, nevertheless, the paper is worth being accepted.   ================================================================ Summary of paper: The paper investigates various properties of the exact likelihood of Deep Latent Variable Models (DLVM). In the first part of the exposition, the authors argue that the likelihood of standard DLVMs is ill-posed in the sense that there exist parameters for which it diverges for continuous data and is well-posed for discrete. In the second part, they propose to use Metropolis-within-Gibbs instead of the standardly used pseudo-Gibbs sampler for missing data imputation. The experimental section on demonstrating unboundedness of the likelihood is very brief and needs more evidence, while the ones comparing different samplers is much more convincing.   Summary of review: The paper's arguments and conclusions about regarding the likelihood being unbounded for DLVMs seem to be fundamentally wrong. Although the proved theorems are mathmeatically correct their assumptions do not match the standard way these models are defined in the literature. Additionally, the experiments in this direction are minimal and I suspect that the reason for the generalization gap is that by not treating the network parameters in a Bayesian way we are still performing MLE on those, hence we are not guaranteed that the network can not overfit.  The proposed method for doing missing data imputation - Metropolis-within-Gibbs - is not known to me to have been used previously in the literature, hence there is a valid contribution here. The ideas presented clearly and are very easy to implement. Results to support the theoretical intuition that this method should perform well versus the pseudo-Gibbs most widely used in practice.  My general opinion is that the authors should reconsider their stance on the unboundness of the likelihood, based on my details comments below, and focus a lot more on the missing data imputation. The idea seems novel and interesting to me. More results and better presentation in this direction would definitely make the paper worth for publishing. However, given the inaccuracies in the first idea, which takes about half of the paper, at the current stage, I propose this work to be rejected.     Comments by sections:  On the boundedness of the likelihood of deep latent variable models ====================================================== The authors argue that standard DVMs as the one presented in Kingma and Welling [25] have unbounded loglikelihood. Although Theorem 1 is mathematically correct as well as Proposition 1 there is a major flaw in the logic and conclusions extracted from this. The parameterization considered in these and written out in equation (9) has been presented to be reflecting the model described in equation (8) which is a 2 layer generator network of a standard VAE. However, there is a fundamental mistake here - equation (9) assumes that there is a different set of parameters $\theta_i$ **per datapoint**, specifically by their definition $\theta_i$ contains $x_i$. This is significantly flawed as the standard VAE model of Kingma and Welling as well as any other model derived from this framework always assumed that the parameters are tight together between all data points (e.g. as the authors originally define $\theta$ below equation (8)). Under this wrong assumption, the following Theorem 1 seems very natural as this non-parametric model would just output a delta function around each data point. However, in any VAE in the literature, where this is not the case, the arguments of this section would not apply and the likelihood is most-likely bounded (I'm not aware of any way of proving this rigorously either way). Hence and the conclusions drawn regarding models known in the community are wrong.    Towards data-dependent likelihood upper bounds ============================================== In this section, the authors propose an upper bound on the likelihood based on finite mixture models. Theorem 2 is a valid bound, however, the assumptions in Theorem 3 are very unrealistic: 1. It is still under the assumption that the generative model parameters are per-data point. 2. It cites the universal approximation abilities of Neural Networks as a reason why the bound is tight. Although this is technically possible, if we assume 1. is not a flaw, however even we never actually train networks with infinite capacity, hence the bound is in fact arbitrarily (and most likely) very loose.   Additionally, the authors fail to cite previous works on upper bounds of the likelihood of DLVMs such as "Sandwiching the marginal likelihood using bidirectional Monte Carlo", "Approximate Bayesian inference with the weighted likelihood bootstrap". Specifically, the Bi-directional Monte Carlo Method can provide increasingly more accurate and tighter upper bounds based on the amount of computing resources provided.   Missing data imputation using the exact conditional likelihood ============================================================== The de-facto "standard" way for doing data-imputation with DLVMs is to use what the authors call pseudo-gibbs sampling. This invloves sequentially sampling p(z|x_o,x_m) and p(x_m|z,x_o) where one replaces the true posterior with the learned approximation. The authors propose to "fix" the method by rather than just sampling from the approximate posterior to use Metropolis-Hastings to account for the approximation gap. This is as correctly pointed out by the authors an instantion of Metropolis-within-Gibbs and indeed is a very valid observation which I have not seen so far in the literature. Although simple, such connections are indeed important.   Witnessing likelihood blow-up ============================================================== Here I think that the experiments are not enough - the authors only demonstrate work on the Frey faces. This is both relatively small as well as not too popular in order to compare with other results in the literature. Additionally, the authors conclude that the big generalization gap is a confirmation of the unboundedness of the likelihood. As discussed earlier, the likelihood is most likely bounded. Secondly, the most likely reason for the generalization gap is the fact that the training on the ELBO still performs an MLE estimation of the parameters $\theta$ of the generative network. This implies that if the network has enough capacity it **will** overfit the data and has been observed before in the literature. This is indeed an issue with the naive formulation of Variational Inference where one does not also provide prior for the network parameters. However, for more complicated datasets, it seems that in practice generating good images is difficult enough that the models used tend to not have enough capacity to do so. Thus my suspicion is that the Frey faces is too simple.    Comparing the pseudo-Gibbs and Metropolis-within-Gibbs samplers ============================================================== The results in this section provide good evidence that the Metropolis-within-Gibbs sampler performs bettern tha the pseudo-Gibbs. It would be interesting to show more statistics than just the F1-score (e.g. the raw accuracy etc...). Further, showing actual reconstructed images would make the presentation even better.