Discretely Relaxing Continuous Variables for tractable Variational Inference
We explore a new research direction in Bayesian variational inference with discrete latent variable priors where we exploit Kronecker matrix algebra for efficient and exact computations of the evidence lower bound (ELBO). The proposed "DIRECT" approach has several advantages over its predecessors; (i) it can exactly compute ELBO gradients (i.e. unbiased, zero-variance gradient estimates), eliminating the need for high-variance stochastic gradient estimators and enabling the use of quasi-Newton optimization methods; (ii) its training complexity is independent of the number of training points, permitting inference on large datasets; and (iii) its posterior samples consist of sparse and low-precision quantized integers which permit fast inference on hardware limited devices. In addition, our DIRECT models can exactly compute statistical moments of the parameterized predictive posterior without relying on Monte Carlo sampling. The DIRECT approach is not practical for all likelihoods, however, we identify a popular model structure which is practical, and demonstrate accurate inference using latent variables discretized as extremely low-precision 4-bit quantized integers. While the ELBO computations considered in the numerical studies require over 10 2352 log-likelihood evaluations, we train on datasets with over two-million points in just seconds.
Update: read the author feedback and all reviews and still agree the paper should be accepted.  This paper addresses the problem of performing Bayesian inference on mobile hardware (e.g., self-driving car, phone) efficiently. As one would imagine, approaches that operate with discrete values have an advantage in hardware. Variational inference, a method for approximate Bayesian inference, often involves continuous latent variables and continuous variational parameters. This paper’s contribution is to cast everything in the discrete space with an approximating discrete prior. The authors explore and develop variational inference in such a setting and this appears to be a new idea in the field.   I like the approach (called DIRECT) but a few things remain in my mind that were not addressed in the paper. First, let me state that I did not understand all the details of the technique. It was not clear to me how to choose the discretization granularity: with discrete approximations of continuous variables it usually matters how granular the points are. I also don’t understand what exactly DIRECT gives up with the Kronecker matrix representation. It’s clear that it is impossible to operate in high dimensional discrete space so there is probably a smoothness assumption embedded in the representation but I do not have a good intuition about that.   The paper is well written and I like the worked examples with GLMs and neural networks. However, I found the beginning of Section 3 was hard to follow because of notation and the way the matrix W is introduced. Organizationally, the section on “predictive posterior computations” seems like it’s in the middle of the two worked examples. The experimental section was very thorough. With regards to related work, it would also be nice to compare to a more recent method than REINFORCE, which is known to have high variance. Also, whenever it is claimed that you compute the exact ELBO it needs to be clarified that it is only exact *after* the discrete approximation is applied. One minor addendum: a small niggle throughout the paper was claiming that you calculate “unbiased, zero-variance gradient estimators”. It’s not wrong but it just sounds like a roundabout way of saying “exact gradients”.