Variational Bayes on Monte Carlo Steroids
Variational approaches are often used to approximate intractable posteriors or normalization constants in hierarchical latent variable models. While often effective in practice, it is known that the approximation error can be arbitrarily large. We propose a new class of bounds on the marginal log-likelihood of directed latent variable models. Our approach relies on random projections to simplify the posterior. In contrast to standard variational methods, our bounds are guaranteed to be tight with high probability. We provide a new approach for learning latent variable models based on optimizing our new bounds on the log-likelihood. We demonstrate empirical improvements on benchmark datasets in vision and language for sigmoid belief networks, where a neural network is used to approximate the posterior.
This work extends hash-based inferences techniques to the learning of latent models. It bridges the gap between existent the theory for hash-based inference for discrete latent variable models and the existent practical inference techniques (sampling and variational methods). The authors provide a theoretical bound on the marginal likelihood of the latent variable models when using hash-based inference then apply it to learning sigmoid belief networks where they claim an improvement as compared to existing benchmarks.
This paper provides theoretical bounds that are tighter than existing variational bounds for the problem of learning latent variable models. The authors extend applied existing theory of hash-based learning and amortized inference to design a black-box learning algorithm. They later applied it to learning a Sigmoid Belief Network. The main advantage to this approach seems to be the partitioning of the search space for posterior distributions into buckets/subsets that are faster to search than with a typical sampling method. The proposed inference scheme then leverages mean-field inference (used heavily in the context of variational inference) within each subset. One of the main technical contributions is the tighter bound on the likelihood using two aggregate estimators which was an extension of an existing work (specific to undirected graphical models) to the directed models setting. However, it was not intuitive how this was more advantageous to stochastic (called mini-batch in the paper) or batch variational inference schemes. While the general variational bounds on the marginal likelihood are provenly tighter, we would have appreciated a less terse explanation (in comparison to mini-batch + mean-field). Additionally, the empirical analysis seemed lacking as there weren't enough details regarding  recent developments and heuristics in the domain were discarded as complementary rather than competing. However, it is clear that the goal of the paper was to prove the superiority of this novel hash-based inference scheme to the common sampling or variational (they used stochastic variational inference for learning the ELBO of the basic SBN unless I misunderstood?) and for that their analysis might be adequate. Overall, despite a few typographical errors, this paper was written eloquently and the methodology was presented adequately and the analysis of the results was quite appropriate.