Boosting Black Box Variational Inference
Approximating a probability density in a tractable manner is a central task in Bayesian statistics. Variational Inference (VI) is a popular technique that achieves tractability by choosing a relatively simple variational approximation. Borrowing ideas from the classic boosting framework, recent approaches attempt to boost VI by replacing the selection of a single density with an iteratively constructed mixture of densities. In order to guarantee convergence, previous works impose stringent assumptions that require significant effort for practitioners. Specifically, they require a custom implementation of the greedy step (called the LMO) for every probabilistic model with respect to an unnatural variational family of truncated distributions. Our work fixes these issues with novel theoretical and algorithmic insights. On the theoretical side, we show that boosting VI satisfies a relaxed smoothness assumption which is sufficient for the convergence of the functional Frank-Wolfe (FW) algorithm. Furthermore, we rephrase the LMO problem and propose to maximize the Residual ELBO (RELBO) which replaces the standard ELBO optimization in VI. These theoretical enhancements allow for black box implementation of the boosting subroutine. Finally, we present a stopping criterion drawn from the duality gap in the classic FW analyses and exhaustive experiments to illustrate the usefulness of our theoretical and algorithmic contributions.
In the submission, the authors aim at developing a black-box boosting method for variational inference, which takes a family of variational distributions and finds a mixture of distribution in a given family that approximates a given posterior distribution well. The main keyword here is black-box; white-box, restricted approaches exist. In order to achieve their aim, the authors formulate a version of the Frank-Wolfe algorithm, and instantiate it with the usual KL objective of variational inference. They then derive a condition on the convergence of this instantiation that is more permissive than the usual smoothness and is based on the reformulation of the bounded curvature condition (Theorem 2). They also show how the constrained optimization problem included in the instantiation of Frank-Wolfe can be expressed in terms of a more intuitive objective, called RELBO in the submission. Finally, the authors propose a criterion of stopping the algorithm (Lemma 4). The resulting black-box algorithm is tested with synthetic data, Bayesian logistic regression, and Bayesian matrix factorization, and these experiments show the promise of the algorithm.  I am not an expert on boosting, but I enjoyed reading the submission and found the results interesting. I particularly liked the fact that the RELBO can be understood intuitively. I have a few complaints though, mostly about presentation. Someone like me (who is familiar with variational inference but not boosting nor Frank-Wolfe) might have appreciated the submission more if the authors had explained in more detail what the LMO objective attempts to do intuitively and how the formula at the bottom of page 5 is derived from the definition of LMO in equation (6), and the authors had described how one should implement the stopping criterion. I also struggled to follow the discussion right after Theorem 2, and I couldn't quite see where the delta relaxation in (7) is used in the authors' algorithm. Finally, the held-out test likelihood for Bayesian matrix factorization does not seem to show that Boosted BBVI beats BBVI, but the text (329-330) says it differently. This made me very confused.  * equation (3), p3: p(x,z) ===> p(z|x) * What do you mean by L-smooth? * boldface q0 in Algorithm 1 ===> q^0 * second display, p6: D^KL(s || ...Z) ===> D^KL(s || .../Z)