FUNCTIONAL VARIATIONAL BAYESIAN NEURAL NETWORKS
Variational Bayesian neural networks (BNNs) perform variational inference over weights, but it is difficult to specify meaningful priors and approximate posteriors in a high-dimensional weight space. We introduce functional variational Bayesian neural networks (fBNNs), which maximize an Evidence Lower BOund (ELBO) defined directly on stochastic processes, i.e. distributions over functions. We prove that the KL divergence between stochastic processes equals the supremum of marginal KL divergences over all finite sets of inputs. Based on this, we introduce a practical training objective which approximates the functional ELBO using finite measurement sets and the spectral Stein gradient estimator. With fBNNs, we can specify priors entailing rich structures, including Gaussian processes and implicit stochastic processes. Empirically, we find fBNNs extrapolate well using various structured priors, provide reliable uncertainty estimates, and scale to large datasets.
-- Paper Summary --
The primary contribution of this paper is the presentation of a novel ELBO objective for training BNNs which allows for more meaningful priors to be encoded in the model rather than the less informative weight priors featured in the literature. This is achieved by way of introducing a KL measure over stochastic processes which allows for priors to take the form of GP priors and other custom variations. Two approaches are given for training the model, one inspired by GANs, and a more practical sampling-based scheme. The performance of this training scheme is validated on a variety of synthetic and real examples, choosing Bayes by Backprop as the primary competitor. An experiment on contextual bandit exploration, and an illustrative Bayesian optimisation example  provided in the supplementary material showcase the effectiveness of this method in applications where well-calibrated uncertainty is particularly pertinent.
-- Critique --
This paper makes important strides towards giving more meaningful interpretations to priors in BNNs. To the best of my knowledge, the KL divergence between stochastic processes that gives rise to an alternate ELBO has not been featured elsewhere, making this a rather interesting contribution that is supplemented by suitable theorems both in the main text and supplementary material. The introductory commentary regarding issues faced with increasing the model capacity of BNNs is particularly interesting, and the associated motivating example showing how degeneracy is countered by fBNN is clear and effective.
The GAN-inspired optimisation scheme is also well-motivated. Although the authors understandably do not pursue that scheme due to the longer computation time incurred (rendering its use impractical), it would have been interesting to see whether the optimum found using this technique is superior to the sampling based scheme used throughout the remainder of the paper. The experimental evaluation is also very solid, striking an adequate balance between synthetic and real-world examples, while also showcasing fBNNs’ effectiveness in scenarios relying on good uncertainty quantification.
In spite of the paper’s indisputable selling points, I have several issues with some aspects of this submission. For clarity, I shall distinguish my concerns between points that I believe to be particularly important, and others which are less significant:
- Monte Carlo dropout (Gal & Ghahramani, 2016), and its extensions (such as concrete dropout), are widely-regarded as being one of the most effective approaches for interpreting BNNs. Consequently, I would have expected this method to feature as a competitor in your evaluation, yet this method does not even get a cursory mention in the text.
 - The commentary on GPs in the related work paints a dour picture of their scalability by mostly listing older papers. However, flexible models such as AutoGP (Krauth et al, 2017) have been shown to obtain very good results on large datasets without imposing restrictions on the choice of kernels.
 - The regression experiments all deal with a one-layer architecture, for which the proposed method is shown to consistently obtain better results. In order to properly assess the effectiveness of the method, I would also be interested in seeing how it compares against BBB for deeper architectures on this problem. Although the authors cite the results in Figure 1 as an indicator that BBB with more layers isn’t particularly effective, it would be nice to also see this illustrated in the cross-dataset comparison presented in Section 5.2.
 - Furthermore, given that all methods are run for a fixed number of iterations, it might be sensible  to additionally report training time along with the results in the table. This should reflect the pre-processing time required to optimise GP hyperparameters when a GP prior is used. Carrying out Cholesky decompositions for 1000x1000 matrices 10k times (as described in Section 5.2.2) does not sound insignificant.
- The observation regarding the potential instability of GP priors without introducing function noise should be moved to the main text; while those who have previously worked with GPs will be familiar with such issues, this paper is directed towards a wider audience and such clarifications would be helpful for those seeking to replicate the paper’s results. On a related note, I would be keen on learning more about other potential issues with the stability of the optimisation procedure, which does not seem to be discussed upfront in the paper but is key for encouraging the widespread use of such methods.
- The paper contains more than just a handful of silly typos and grammatical errors - too many to list here. This single-handedly detracts from the overall quality of the work, and I highly advise the authors to diligently read through the paper in order to identify all such issues.
 - The references are in an absolute shambles, having inconsistent citation styles, arXiv papers cited instead of conference proceedings, etc. While this is obviously straightforward to set right, I’m nonetheless disappointed that this exercise was not carried out prior to the paper’s submission.
 - The theory presented in Appendix A of the supplementary material appears to be somewhat ‘dumped’ there. Given that this content is crucial for establishing the correctness of the proposed method, linking them more clearly to the main text would improve its readability and give it a greater sense of purpose. I found it hard to follow in its current state.
** Minor **
 - In the introduction there should some mention of deep Gaussian processes which are implicitly a direct competitor to BNNs, and can now also be scaled to millions and billions of observations (Cutajar et al. 2017; Salimbeni et al. 2017). The former is particularly relevant to this work since the architecture can be assimilated to a BNN with special structure for emulating certain kernels.
 - Experiment 5.1.1 is interesting, and the results in Figure 2 are convincing. I would also be interested in seeing how fBNN performs when the prior is misspecified however, which may be induced by using a less appropriate GP kernel. This would complement the already provided insight on using tanh vs ReLU activations.
 - The performance improvement for the experiment on large regression datasets is quite subdued, so it might be interesting to see how both methods compare against each other when deeper BNN architectures are considered. 
- With regards to Appendix C.2, which order arccosine kernel is being used here? One can easily draw similarities between the first order arccosine kernel and NN layers with ReLUs, so perhaps it would be useful to specify which order is being used in the experiment.  
- Given that the data used for experiments in Appendix C.3 effectively has grid structure, I would be interested in seeing how KISS-GP performs on this task. There should be easily accessible implementations in GPyTorch for testing this out. Given how GPs tend to not work very well on image completion tasks due to smoothness in the kernel, this comparison may also be in fBNNs favour.
- Restating the basic architecture of the BNN being used for the contextual bandits experiment in the paper itself would be helpful in order to avoid having to separately check out Riquieme et al (2018) to find such details.
- I wonder if the authors have already thought about the extendability of their proposal to more complex BNN architectures such as Bayesian ConvNets?
-- Recommendation --
Whereas several ICLR submissions tend heavily towards validation by way of empirical evaluation, I find that the theoretic contributions presented in this paper are by themselves interesting and well-developed, which is very commendable. However, there are multiple telling signs of this being a rushed submission, and I am less inclined to argue ardently for such a paper’s acceptance. Although the paper indeed has its strong points, both in terms of novelty and varied experimental evaluation, in view of this overall lack of finesse and other concerns listed above, I think that the paper is in dire need of a thorough clean-up before being published.
Pros/Cons summary:
+   Interesting concepts that extend beyond empirical fixes.
+   Defining more interpretable priors is a very pertinent topic in the study of BNNs.
+   The presented ideas could potentially have notable impact.
+   Illustrative experiments and benchmark tests are convincing.
-   Not enough connection to MC dropout.
-   Choice of experiments and description of stochastic processes overly similar to other recent widely-publicised papers. It feels on trend, but consequently also somewhat reductive.
-   More than a few typos and grammatical errors.
-   Presentation is quite rough around the edges. The references are in a particularly dire state.