DETERMINISTIC VARIATIONAL INFERENCE FOR ROBUST BAYESIAN NEURAL NETWORKS
Bayesian neural networks (BNNs) hold great promise as a flexible and principled solution to deal with uncertainty when learning from finite data. Among approaches to realize probabilistic inference in deep neural networks, variational Bayes (VB) is theoretically grounded, generally applicable, and computationally efficient. With wide recognition of potential advantages, why is it that variational Bayes has seen very limited practical use for BNNs in real applications? We argue that variational inference in neural networks is fragile: successful implementations require careful initialization and tuning of prior variances, as well as controlling the variance of Monte Carlo gradient estimates. We provide two innovations that aim to turn VB into a robust inference tool for Bayesian neural networks: first, we introduce a novel deterministic method to approximate moments in neural networks, eliminating gradient variance; second, we introduce a hierarchical prior for parameters and a novel Empirical Bayes procedure for automatically selecting prior variances. Combining these two innovations, the resulting method is highly efficient and robust. On the application of heteroscedastic regression we demonstrate good predictive performance over alternative approaches.
Summary:
This work is tackling two difficulties in current VB applied to DNNs ("Bayes by backprop"). First, MC approximations of intractable expectations are replaced by deterministic approximations. While this has been done before, the solution here is new and very interesting. Second, a Gaussian prior with length scales is learned by VB empirical Bayes alongside the normal training, which is also very useful.
The term "fixing VB" and some of the intro is not really supported by the rather weak experiments, done on small datasets and networks, where much older work like Barber&Bishop would apply without any problems. While interesting and potentially very useful novelties are presented, and the writing is excellent, both experiments and motivation can be improved.
- Quality: Extremely well written paper, I learned a lot from it. Approximations are
   tested, great figures to explain things. And the major technical novelty, the
   expression for <h_j h_l>, is really interesting and useful.
- Clarity: Excellent writing until it comes to the experiments. Here, important
   details are just missing, for example what q(w) is (fully factorized Gaussian?).
   Very nice literature review, also historical.
- Originality: The idea of matching Gaussian moments along the network graph is
   previously done in PBP (Lobato, Adams), as acknowledged here. Porting this from
   ADF to VB gives dDVI. PBP also has the property that a DL system gives you the
   gradients. Having said that, I think dDVI may be more useful than PBP.
   While Barber&BIshop 98 is cited, they miss the expression for <h_j h_l> in
   there. Now, what is done here, is more elegant, does not need 1D quadrature.
- Significance: Judging from the existing experiments, the significance may be
   rather small, *if one only looks at test log likelihood*. I'd still give this the
   benefit of the doubt, as in particular dDVI could be really interesting at large
   scale as well. But the authors may tone down their language a bit.
   To increase significance, I recommend to comment beyond just test log
   likelihood scores. For example:
   - Does the optimization become simpler, less tuning required, more automatic?
      Would one not expect so, given you make a big point out of reducing variance?
      Does it converge faster?
   - Can you do something with your posterior that normal DNN methods cannot
      do? Better decisions (bandits, active learning, HPO)? Continual learning?
      In the end, who really cares about test log likelihood?
Experiments:
- What is the q(w) family being used here? Fully factorized Gaussian? I
   suppose so for dDVI. But for DVI? Not said anywhere, in main paper or
   Appendix
- A bit disappointing. Why not evaluate at least dDVI with diagonal q(w) on
   some much larger models and datasets? Why not quote numbers on speed
   and robustness of learning, etc? Show what you really gain by reducing the
   variance.
- Experiments are OK, but on pretty small datasets, and for single hidden
   layer NNs. On such data and models, the Barber&Bishop 98 method could
   be run as well
- Was MCVI run with re-parameterization? This is really important. If not,
   this would be an important missing comparison. Please be clear in the main
   text
- Advantages over MCVI are not very large. At least, dDVI should be faster to
   converge than MCVI.
   Can you say something about robustness of training? Is it easier to train
   dDVI than MCVI?
- Why not show the PBP-1 results, comparing to dDVI, in the main text? Are they
   obtained with the same model? dDVI is doing better.
Other points:
- Please acknowledge the <h_j h_l> expression in Barber&Bishop 98. Yours is
   more elegant and faster (does not need 1D quadrature)
- Relation to PBP: Note that dDVI has an advantage in practice. With PBP, I need
   to compute gradients for every datapoint. In dDVI, I can do mini-batch
   updates.
- I just *love* the header "Wild approximations". I tend to refer to this kind of work
   as "weak analogies". Why do you not also compare against this, and show it really
   does not work?