Fast Second-Order Stochastic Backpropagation for Variational Inference
We propose a second-order (Hessian or Hessian-free) based optimization method for variational inference inspired by Gaussian backpropagation, and argue that quasi-Newton optimization can be developed as well. This is accomplished by generalizing the gradient computation in stochastic backpropagation via a reparametrization trick with lower complexity. As an illustrative example, we apply this approach to the problems of Bayesian logistic regression and variational auto-encoder (VAE). Additionally, we compute bounds on the estimator variance of intractable expectations for the family of Lipschitz continuous function. Our method is practical, scalable and model free. We demonstrate our method on several real-world datasets and provide comparisons with other stochastic gradient methods to show substantial enhancement in convergence rates.
This paper presents a scalable second-order stochastic variational inference method for models with normally distributed latent variables. In order to efficiently compute the gradient and Hessian, a representrization trick for general location-scale family is adopted with computation scales quadratically w.r.t the number of latent variables for both gradient and Hessian (in practice with diagonal Gaussian). Furthermore, Hessian-free optimization is used to account for the high dimensionality of the underlying embedded parameters. R-operator technique is used and it enables exact Hessian-vector product computation in Hessian-free optimization. L-BFGS can also be used in place of Hessian-free within the proposed framework. On the theoretical side, the paper provides a variance bound which explains why in some earlier work even only one sample from the variational distribution is sufficient to construct a noisy gradient with low variance. Experimental results are reported on both Bayesian logistic regression and variational autoencoder. The proposed Hessian-free or L-BFGS based methods understandably converge faster and to better local optima. **Quality**Overall the paper is of good quality with both practical and theoretical contributions. Deriving bounds is not my expertise, so I couldn't follow all of the proof for Theorem 2 in the supplementary material. But as far as I can understand, the derivation and proof seem sound. The evaluation clearly demonstrates the advantage of the proposed methods. **Clarity**This paper seems to be written in a hurry with some parts not clear enough. Some of the descriptions are vague (mostly in the Section 3 and 4) and probably difficult to understand without enough context - I have to interpolate the authors' intention sometimes when reading the paper. I would suggest the authors carefully proof-reading the paper and filling in more context if possible. Detailed comments:1. Line 113: Presumably g and H denote the gradient and Hessian of f(z) respectively, but it was never formally defined in the main paper (it is defined in supplementary material).2. Line 128: leading to an algorithmic -> (based on supplementary material) leading to an algorithmic complexity3. Line 199: none-the-less -> nonetheless4. Supplementary material Line 160: lawyer-wise -> layer-wise5. Supplementary material Line 250: Should be Eq. (10) not Eq. (11).6. For the second part of the experiment, it is not clear to me why there is running time/objective reported for both train and test. From what I understand, the variational auto-encoder is trained on the training data to learn the parameters for both the encoder and decoder networks with the proposed and competing methods. For test, you feed the test images as the input to the encoder network and reconstruct them from the output of the decoder network (as I understand this is how the images in Figure 2(b) and 3(b) are generated, the authors didn't mention anything about how they are generated). Does the fact that there is also a running time for test mean that the test data is also used to train the model? I hope not, otherwise the reconstruction results would not be reliable. We just need more context to understand the evaluation. If the space is a concern, I would suggest removing some of the introduction to BFGS and some background stuff in Section 3.**Originality**The proposed method is built on top of the work on stochastic gradient variational Bayesian (SGVB), which mostly only uses first-order method with adaptive learning rate tuning mechanism (e.g. AdaGrad/RMSProp). Incorporating second-order information isn't something groundbreaking but the paper presents a way to efficiently compute the gradient and Hessian, which is still fairly valuable. Furthermore, Hessian-free and L-BFGS can be applied and benefit from the efficient gradient computation. The theoretical analysis on the variance bound is very exciting work because it provides the theoretical justification for something that has been observed empirically. **Significance**The evaluation is reasonably thorough. Given that there isn't new algorithm/modeling innovation in this paper, reporting optimization objective v.s. time is the most convincing way to demonstrate a better optimization strategy. I do have one comment on the Bayesian logistic regression experiment and would like the authors to investigate and maybe address it in the supplementary material (given that the space seems already a problem for the main paper): Looking at the coefficients in Figure 3 of the supplementary material, the magnitude of DSVI is orders of magnitude larger than the proposed methods for the first two datasets. Is there a particular reason behind this? On the much bigger A9a dataset, all three methods learn coefficients which are in the same scale.