summaries: The authors present a new method for Bayesian deep learning motivated by the difficulty of posterior inference in the "overparameterized" regime of deep neural network models. The proposed method provides a principled strategy for selecting a subset of the neural network's parameters (forming a so-called "subnetwork") for which a full-covariance approximate posterior can be computed. The authors use the well-studied Laplace approximation with the generalized Gauss-Newton Hessian approximation for the covariance. An empirical analysis is presented which attempts to assess the efficacy of the proposed method in prediction accuracy and uncertainty quantification.

summaries: The paper proposes to approximate the posterior distribution of a Bayesian neural network by an approximation that consists of a deterministic component. The authors select a sub network and infer approximate posterior distributions over the weights in the sub network. All other weights are estimated via MAP point estimation.  A sufficiently small sub-network allows high fidelity posterior approximations that do not make restrictive mean field assumptions to be tractable.

summaries: The authors focus on the important problem of scalable approximate inference in Bayesian NNs. More specifically, they propose a method for scalable BNNs via a (full-covariance Gaussian) Laplace approximation on a (Wasserstein-based) pruned subnetwork within a deterministically-trained model. They include a theoretical analysis for a simple generalized linear model, and experiments on 1D regression, tabular regression, and larger-scale image classification with CIFAR-10 (using the dataset shift setup from Ovadia et al., (2019)). From the experiments, they show that their method generally outperforms comparable methods (including deep ensembles) on metric performance and on the ability to capture in-between uncertainty.

abstractText: The Bayesian paradigm has the potential to solve some of the core issues in modern deep learning, such as poor calibration, data inefficiency, and catastrophic forgetting. However, scaling Bayesian inference to the high-dimensional parameter spaces of deep neural networks requires restrictive approximations. In this paper, we propose performing inference over only a small subset of the model parameters while keeping all others as point estimates. This enables us to use expressive posterior approximations that would otherwise be intractable for the full model. In particular, we develop a practical and scalable Bayesian deep learning method that first trains a point estimate, and then infers a full covariance Gaussian posterior approximation over a subnetwork. We propose a subnetwork selection procedure which aims to maximally preserve posterior uncertainty. We empirically demonstrate the effectiveness of our approach compared to point-estimated networks and methods that use less expressive posterior approximations over the full network.

authors:
Dario Amodei
Chris Olah
Jacob Steinhardt
Paul Christiano
John Schulman
Dan Mané

title: Concrete problems in ai safety

venue: arXiv preprint arXiv:1606.06565,

authors:
Arsenii Ashukha
Alexander Lyzhov
Dmitry Molchanov
Dmitry Vetrov

title: Pitfalls of in-domain uncertainty estimation and ensembling in deep learning

venue: In ICLR,

authors:
Arsenii Ashukha
Alexander Lyzhov
Dmitry Molchanov
Dmitry P. Vetrov

venue: In 8th International Conference on Learning Representations,

title: The fundamental incompatibility of scalable hamiltonian monte carlo and naive data subsampling

venue: Proceedings of Machine Learning Research, pp. 533–540,

title: Pattern recognition and machine learning

venue: springer,

authors:
Charles Blundell
Julien Cornebise
Koray Kavukcuoglu
Daan Wierstra

title: Weight Uncertainty in Neural Networks

venue: In Proceedings of The 32nd International Conference on Machine Learning (ICML),

authors:
Yu Cheng
Duo Wang
Pan Zhou
Tao Zhang

title: A survey of model compression and acceleration for deep neural networks

venue: arXiv preprint arXiv:1710.09282,

authors:
Tianyu Cui
Aki Havulinna
Pekka Marttinen
Samuel Kaski

title: Informative gaussian scale mixture priors for bayesian neural networks

venue: arXiv preprint arXiv:2002.10243,

authors:
Dheeru Dua
Casey Graff

title: UCI machine learning repository, 2017

venue: URL http://archive. ics.uci.edu/ml

authors:
Michael W Dusenberry
Ghassen Jerfel
Yeming Wen
Yi-an Ma
Jasper Snoek
Katherine Heller
Balaji Lakshminarayanan
Dustin Tran

title: Efficient and scalable bayesian neural nets with rank-1 factors

authors:
Angelos Filos
Sebastian Farquhar
Aidan N. Gomez
Tim G.J. Rudner
Zachary Kenton
Lewis Smith
Milad Alizadeh
Arnoud de Kroon
Yarin Gal

title: Benchmarking bayesian deep learning with diabetic retinopathy diagnosis

authors:
Angelos Filos
Sebastian Farquhar
Aidan N Gomez
Tim GJ Rudner
Zachary Kenton
Lewis Smith
Milad Alizadeh
Arnoud de Kroon
Yarin Gal

authors:
Andrew YK Foong
David R Burt
Yingzhen Li
Richard E Turner

title: On the expressiveness of approximate inference in bayesian neural networks

venue: arXiv, pp. arXiv–1909,

authors:
Andrew YK Foong
Yingzhen Li
José Miguel Hernández-Lobato
Richard E Turner

title: In-between uncertainty in bayesian neural networks

venue: ICML Workshop on Uncertainty and Robustness in Deep Learning,

authors:
Stanislav Fort
Huiyi Hu
Balaji Lakshminarayanan

title: Deep ensembles: A loss landscape perspective

venue: arXiv preprint arXiv:1912.02757,

authors:
Jonathan Frankle
Michael Carbin

title: The lottery ticket hypothesis: Finding sparse, trainable neural networks

venue: In International Conference on Learning Representations,

authors:
Lex Fridman
Li Ding
Benedikt Jenik
Bryan Reimer

title: Arguing machines: Human supervision of black box ai systems that make life-critical decisions

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops,

title: Uncertainty in deep learning

venue: University of Cambridge,

authors:
Yarin Gal
Zoubin Ghahramani

title: Dropout as a bayesian approximation: Representing model uncertainty in deep learning

venue: In international conference on machine learning,

authors:
Adrià Garriga-Alonso
Carl Edward Rasmussen
Laurence Aitchison

title: Deep convolutional networks as shallow gaussian processes

title: Probabilistic machine learning and artificial intelligence

authors:
Soumya Ghosh
Finale Doshi-Velez

title: Model selection in bayesian neural networks via horseshoe priors

venue: arXiv preprint arXiv:1705.10388,

title: Bayesian Gaussian processes for regression and classification

venue: PhD thesis, Citeseer,

authors:
Clark R Givens
Rae Michael Shortt

title: A class of wasserstein metrics for probability distributions

venue: The Michigan Mathematical Journal,

authors:
Aidan N Gomez
Ivan Zhang
Kevin Swersky
Yarin Gal
Geoffrey E Hinton

title: Learning sparse networks using targeted dropout

authors:
Priya Goyal
Piotr Dollár
Ross B. Girshick
Pieter Noordhuis
Lukasz Wesolowski
Aapo Kyrola
Andrew Tulloch
Yangqing Jia
Kaiming He

title: Accurate, large minibatch SGD: Training ImageNet in 1 hour

authors:
Chuan Guo
Geoff Pleiss
Yu Sun
Kilian Q Weinberger

title: On calibration of modern neural networks

venue: In Proceedings of the 34th International Conference on Machine Learning-Volume

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification

venue: In 2015 IEEE International Conference on Computer

title: Deep residual learning for image recognition

authors:
Dan Hendrycks
Thomas Dietterich

title: Benchmarking neural network robustness to common corruptions and perturbations

venue: arXiv preprint arXiv:1903.12261,

authors:
José Miguel Hernández-Lobato
Ryan Adams

title: Probabilistic backpropagation for scalable learning of bayesian neural networks

venue: In International Conference on Machine Learning,

authors:
Alexander Immer
Maciej Korzepa
Matthias Bauer

title: Improving predictions of bayesian neural networks via local linearization

authors:
Pavel Izmailov
Wesley J Maddox
Polina Kirichenko
Timur Garipov
Dmitry Vetrov
Andrew Gordon Wilson

title: Subspace inference for bayesian deep learning

venue: In 35th Conference on Uncertainty in Artificial Intelligence,

authors:
Mohammad Emtiyaz Khan
Didrik Nielsen
Voot Tangkaratt
Wu Lin
Yarin Gal
Akash Srivastava

title: Fast and scalable bayesian deep learning by weight-perturbation in adam

venue: arXiv preprint arXiv:1806.04854,

authors:
Mohammad Emtiyaz E Khan
Alexander Immer
Ehsan Abedi
Maciej Korzepa

title: Approximate inference turns deep networks into gaussian processes

venue: In Advances in neural information processing systems,

authors:
Durk P Kingma
Tim Salimans
Max Welling

title: Variational dropout and the local reparameterization trick

authors:
Agustinus Kristiadi
Matthias Hein
Philipp Hennig

title: Being bayesian, even just a bit, fixes overconfidence in relu networks

venue: arXiv preprint arXiv:2002.10118,

authors:
Alex Krizhevsky
Geoffrey Hinton

title: Learning Multiple Layers of Features from Tiny Images

venue: Technical report, University of Toronto,

authors:
Balaji Lakshminarayanan
Alexander Pritzel
Charles Blundell

title: Simple and scalable predictive uncertainty estimation using deep ensembles

venue: In Advances in Neural Information Processing Systems,

title: Variational inference in probabilistic models

venue: PhD thesis, University of Cambridge,

authors:
Yann LeCun
Léon Bottou
Yoshua Bengio
Patrick Haffner

title: Gradient-based learning applied to document recognition

venue: Proceedings of the IEEE,

authors:
Juho Lee
Saehoon Kim
Jaehong Yoon
Hae Beom Lee
Eunho Yang
Sung Ju Hwang

title: Adaptive network sparsification via dependent variational beta-bernoulli dropout

authors:
Ekaterina Lobacheva
Nadezhda Chirkova
Maxim Kodryan
Dmitry P Vetrov

title: On power laws in deep ensembles

authors:
Christos Louizos
Max Welling

title: Structured and efficient variational deep learning with matrix gaussian posteriors

authors:
Christos Louizos
Max Welling
Diederik P Kingma

title: Learning sparse neural networks through l 0 regularization

venue: arXiv preprint arXiv:1712.01312,

title: A practical bayesian framework for backpropagation networks

venue: Neural computation,

authors:
Wesley J Maddox
Pavel Izmailov
Timur Garipov
Dmitry P Vetrov
Andrew Gordon Wilson

title: A simple baseline for bayesian uncertainty in deep learning

authors:
Wesley J Maddox
Gregory Benton
Andrew Gordon Wilson

title: Rethinking parameter counting in deep models: Effective dimensionality revisited

venue: arXiv preprint arXiv:2003.02139,

title: Second-order optimization for neural networks

venue: University of Toronto (Canada),

authors:
James Martens
Ilya Sutskever

title: Learning recurrent neural networks with hessian-free optimization

venue: In Proceedings of the 28th international conference on machine learning

title: Scalable Gaussian process inference using variational methods

authors:
Aaron Mishkin
Frederik Kunstner
Didrik Nielsen
Mark Schmidt
Mohammad Emtiyaz Khan

title: Slang: Fast structured covariance approximations for bayesian deep learning with natural gradient

authors:
Dmitry Molchanov
Arsenii Ashukha
Dmitry Vetrov

title: Variational dropout sparsifies deep neural networks

authors:
Eric Nalisnick
Akihiro Matsukawa
Yee Whye Teh
Dilan Gorur
Balaji Lakshminarayanan

title: Do Deep Generative Models Know What They Don’t Know

venue: In International Conference on Learning Representations (ICLR),

title: Bayesian Learning for Neural Networks

venue: PhD thesis, CAN,

authors:
John Ashworth Nelder
R Jacob Baker

title: Generalized Linear Models

venue: Wiley Online Library,

authors:
Yuval Netzer
Tao Wang
Adam Coates
Alessandro Bissacco
Bo Wu
Andrew Y Ng

title: Reading Digits in Natural Images with Unsupervised Feature Learning

venue: In NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning,

authors:
Anh Nguyen
Jason Yosinski
Jeff Clune

title: Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

authors:
Sebastian W Ober
Carl Edward Rasmussen

title: Benchmarking the neural linear model for regression

venue: arXiv preprint arXiv:1912.08416,

authors:
Kazuki Osawa
Siddharth Swaroop
Anirudh Jain
Runa Eschenhagen
Richard E Turner
Rio Yokota
Mohammad Emtiyaz Khan

title: Practical deep learning with Bayesian principles

authors:
Yaniv Ovadia
Emily Fertig
Balaji Lakshminarayanan
Sebastian Nowozin
D Sculley
Joshua Dillon
Jie Ren
Zachary Nado
Jasper Snoek

title: Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift

authors:
Robert Pinsler
Jonathan Gordon
Eric Nalisnick
José Miguel Hernández-Lobato

title: Bayesian batch active learning as sparse subset approximation

authors:
Nicholas G Polson
Veronika

title: Ročková. Posterior concentration for sparse deep learning

authors:
Carlos Riquelme
George Tucker
Jasper Snoek

title: Deep bayesian bandits showdown: An empirical comparison of bayesian deep networks for thompson sampling

authors:
Hippolyt Ritter
Aleksandar Botev
David Barber

title: A scalable laplace approximation for neural networks

authors:
Simone Rossi
Sebastien Marmin
Maurizio Filippone

title: Walsh-hadamard variational inference for bayesian deep learning

venue: arXiv preprint arXiv:1905.11248,

title: Fast curvature matrix-vector products for second-order gradient descent

authors:
Jasper Snoek
Oren Rippel
Kevin Swersky
Ryan Kiros
Nadathur Satish
Narayanan Sundaram
Mostofa Patwary
Mr Prabhat
Ryan Adams

title: Scalable bayesian optimization using deep neural networks

venue: In International conference on machine learning,

authors:
Jakub Swiatkowski
Kevin Roth
Bastiaan S Veeling
Linh Tran
Joshua V Dillon
Stephan Mandt
Jasper Snoek
Tim Salimans
Rodolphe Jenatton
Sebastian Nowozin

title: The k-tied normal distribution: A compact parameterization of gaussian mean field posteriors in bayesian neural networks

venue: arXiv preprint arXiv:2002.02655,

authors:
Chaoqi Wang
Guodong Zhang
Roger Grosse

title: Picking winning tickets before training by preserving gradient flow

authors:
Han Xiao
Kashif Rasul
Roland Vollgraf

title: Fashion-MNIST: a novel image dataset for benchmarking machine learning

authors:
Sergey Zagoruyko
Nikos Komodakis

title: Wide residual networks

venue: Proceedings of the British Machine Vision Conference 2016,

venue: (He et al.,

venue: Kristiadi et al

title: The 1d toy dataset

venue: Dua & Graff

heading: 1 INTRODUCTION

text:
Deep neural networks (DNNs) still suffer from critical shortcomings that make them unfit for important applications. For instance, DNNs tend to be poorly calibrated and overconfident in their predictions, especially when there is a shift in the train and test distributions (Nguyen et al., 2015; Guo et al., 2017). To reliably inform decision making, DNNs must be able to robustly quantify the uncertainty in their predictions, which is particularly important in safety-critical areas such as healthcare or autonomous driving (Amodei et al., 2016; Filos et al., 2019a; Fridman et al., 2019).
Bayesian modeling (Ghahramani, 2015; Gal, 2016) presents a principled way to capture predictive uncertainty via the posterior distribution over model parameters. Unfortunately, due to their nonlinearities, exact posterior inference is intractable in DNNs. Despite recent successes in the field of Bayesian deep learning (Blundell et al., 2015; Gal & Ghahramani, 2016; Osawa et al., 2019; Maddox et al., 2019; Dusenberry et al., 2020), existing methods are only made scalable to modern DNNs with large numbers of parameters by invoking unrealistic assumptions. This severely limits the expressiveness of the inferred posterior and thus deteriorates the quality of the induced uncertainty estimates (Ovadia et al., 2019; Fort et al., 2019; Foong et al., 2019a; Ashukha et al., 2020a).
Due to the heavy overparameterization of DNNs, their accuracy is well-preserved by a small subnetwork (Cheng et al., 2017). Additionally, recent work by Izmailov et al. (2019) has shown how performing inference over a low dimensional subspace of the weights can result in accurate uncertainty quantification. These observations prompt the following question for a DNN’s uncertainty: Can a full DNN’s model uncertainty be well-preserved by a small subnetwork’s model uncertainty? We answer this question in the affirmative. We show both theoretically and empirically that the full network posterior can be well represented by a subnetwork’s posterior. As a result, we can use more expensive but faithful posterior approximations over just that subnetwork. We show that this achieves better uncertainty quantification than if we use cheaper, but more crude, posterior approximations over the full network.
The contributions of this paper are as follows:
1. We propose a new Bayesian deep learning approach that performs Bayesian inference over only a small subset of the model weights and keeps all other weights deterministic. This allows us to use expressive posterior approximations that are typically intractable in DNNs.
2. As a concrete instantiation of this framework, we develop a practical and scalable Bayesian deep learning method that uses the linearized Laplace approximation to infer a fullcovariance Gaussian posterior over a subnetwork within a point-estimated neural network.
3. We formally characterize the discrepancy between the posterior distributions over a subnetwork and the full network (in terms of their Wasserstein distance) in the linearized model, and derive a theoretically motivated strategy to select a subnetwork that minimizes this discrepancy under certain assumptions.
4. We empirically show, on various benchmarks, that our method compares favourably against point-estimated networks and other Bayesian deep learning methods, experimentally confirming that expressive subnetwork inference is superior to crude inference over full networks.

heading: 2 SUBNETWORK POSTERIOR APPROXIMATION

text:
Bayesian neural networks (BNNs) aim to capture model uncertainty, i.e., uncertainty about the choice of weightsW which arises due to multiple plausible explanations of the training data {y,X}. Here, y is the dependent variable (e.g. classification label) andX is the feature matrix. A prior distribution p(W) is specified over the BNN’s weights. We wish to infer their full posterior distribution
p(W|y,X) ∝ p(y|X,W) p(W) . (1) To make predictions, we then estimate the posterior predictive distribution that averages the network’s predictions across all possible settings of the weights, weighted by their posterior probability, i.e.
p(y∗|X∗,y,X) = ∫ W p(y∗|X∗,W)p(W|y,X)dW . (2)
Unfortunately, due to the size of modern deep neural networks, it is not only intractable to infer the exact posterior distribution p(W|y,X) in Eq. (1), but it is even computationally challenging to properly approximate it. As a consequence, crude posterior approximations such as complete factorization are commonly employed (Blundell et al., 2015; Hernández-Lobato & Adams, 2015; Kingma et al., 2015; Khan et al., 2018; Osawa et al., 2019), i.e. p(W|y,X) ≈∏Dd=1 q(wd) where wd denotes the d-th weight in the D-dimensional neural network weight vector W ∈ RD (the concatenation and flattening of all layers’ weight matrices). Clearly, this is a very wishful assumption; In practise, it suffers from severe pathologies (Foong et al., 2019a;b).
In this work, we question the implicit assumption that a good posterior approximation needs to include all BNN parameters. Instead, we aim to perform inference only over a small subset of the weights. This approach is well-motivated for two reasons:
1. Overparameterization: Maddox et al. (2020) have shown that, in the neighborhood of local optima, there are many directions that leave the NN’s predictions unchanged. Moreover, NNs can be heavily pruned without sacrificing test-set accuracy (Frankle & Carbin, 2019). Thus, the majority of a NN’s predictive power might be isolated to a small subnetwork.
2. Inference over submodels: Previous work1 has provided evidence that inference can be effective even when not done on the full parameter space. Izmailov et al. (2019) performed inference over a low-dimensional projection of the weights. Neural-linear models, which give a Bayesian treatment to only the last layer of a DNN, have shown to be competitive with full-network approaches (Riquelme et al., 2018; Ober & Rasmussen, 2019).
We thus combine these ideas, making the following two-step approximation of the posterior in Eq. (1): p(W|y,X) ≈ p(WS |y,X) ∏ r δ(wr − w∗r) ≈ q(WS) ∏ r δ(wr − w∗r) . (3)
The first approximation decomposes the full neural network posterior p(W|y,X) into a posterior p(WS |y,X) over the subnetwork WS and delta functions δ(wr − w∗r) over all remaining weights {wr}r, keeping them at fixed values w∗r ∈ R. This can be viewed as pruning the variances of the weights {wr}r to zero, which is in contrast to ordinary weight pruning methods that set the weights {wr}r themselves to zero. The second approximation is a result of posterior inference over the subnetwork still being intractable. In turn, we introduce the approximate distribution q(WS). Yet, as the subnetwork is much smaller than the full network, we can afford to make q(WS) expressive and able to capture rich dependencies across the weights within the subnetwork.

heading: 3 SUBNETWORK INFERENCE VIA LAPLACE APPROXIMATION

text:
To obtain a method that is as practical as possible, we propose to use inference techniques that can estimate a posterior distribution post-hoc from a point-estimated network. The Laplace approximation (MacKay, 1992) is well-suited to this task as it derives the approximate posterior from the local optimization landscape. Other inference procedures, such as SWAG (Maddox et al., 2019), could also be used. Nevertheless, we focus on Laplace due to it being a well-studied, fundamental technique.
Step #1: Point Estimation. The first step of the procedure is to train a neural network to obtain a point estimate of the weights, denoted WMAP . This estimate should respect the Bayesian model given in Eq. (1), and therefore we optimize the maximum a-posteriori (MAP) objective:
WMAP = argmaxW [log p(y|X,W ) + log p(W )] . (4) This can be done using standard stochastic gradient-based optimization methods commonly-used in modern deep learning (Goodfellow et al., 2016). This step is illustrated in Fig. 1 (a).
Step #2: Subnetwork Selection. The second step is to identify a small subnetworkWS . Ideally, we would like to identify the subnetwork whose posterior is ‘closest’ to the full-network posterior. We formalize this argument in Section 4 and describe a principled strategy that, under certain conditions, minimizes the 2-Wasserstein distance between the sub- and full-network posteriors. All other weights not belonging to that subnetwork are then assigned fixed values: the MAP estimates obtained in Step #1. See Fig. 1 (b) for an illustration of this step.
Step #3: Bayesian Inference. Given the subnetwork point estimateW SMAP , we use the Laplace approximation to infer a full-covariance Gaussian posterior distribution over the subnetwork WS :
p(WS |y,X) ≈ q(WS) = N ( WS ;W S MAP , H −1) (5) where the posterior covariance matrix H−1 ∈ RD×D corresponds to the inverse of the average Hessian of the negative log posterior, i.e. H = NE [ −∂2 log p(y|X,W )/∂W 2 ] + λI . Here, the expectation is w.r.t. the data generating distribution and λ is the precision of a zero-mean factorized Gaussian prior p(W) = N (W;0, λ−1I). In practice, we approximate the Hessian H with the generalized Gauss-Newton (GGN) matrix H̃ (Schraudolph, 2002), i.e.
H̃ = N∑ n=1 J>n HnJn + λI, with Jn = ∂f(xn,W ) ∂W and Hn = ∂2L(yn,f(xn,W )) ∂2f(xn,W ) (6)
1See Section 6 for a more thorough discussion of related work.
where Jn ∈ RO×D is the Jacobian of the neural network features f(xn,W ) ∈ RO w.r.t. the weights W , andHn ∈ RO×O is the Hessian of the loss L(yn,f(xn,W )) w.r.t. the features f(xn,W ).The GGN H̃ has clear practical advantages over the Hessian H; see Martens & Sutskever (2011) and Martens (2016). Using the Laplace approximation with the GGN Hessian can be viewed as an implicit local linearization of the underlying neural network f(x,W ) at its MAP estimateWMAP ,
fMAPlin (x,W ) = f(x,WMAP ) + JWMAP (x)(W −WMAP ) (7) where JWMAP (x) = ∂f(x,WMAP )/∂WMAP ∈ RO×D (Immer et al., 2020). Note that the model in Eq. (7) is linear in W , as only the term JWMAP (x)W depends linearly onW , while the other terms are constant w.r.t.W and can thus be subsumed into an additive bias term (Khan et al., 2019). The GGN approximation thus locally turns the underlying probabilistic model from a Bayesian neural network into a (generalized) linear model, with basis function expansion JWMAP (x) of covariate x (Immer et al., 2020). Put differently, linearized Laplace in the neural network f(x,W ) is equivalent to ordinary Laplace in the linear model fMAPlin (x,W ) in Eq. (7), as the GGN H̃ corresponding to f(x,W ) in Eq. (6) is equivalent to the Hessian H corresponding to fMAPlin (x,W ) in Eq. (7) (Khan et al., 2019). This is a useful property that will allow us to derive a principled subnetwork selection strategy in Section 4. This step is illustrated in Fig. 1 (c). We emphasize that this whole procedure (i.e. Steps #1-#3) is a perfectly valid mixed inference strategy, performing full Laplace inference over the selected subnetwork and MAP inference over all remaining weights.
Step #4: Prediction. Given the linearized Laplace approximation over the subnetwork WS in Eqs. (5) and (6), i.e. q(WS) = N (WS ;W SMAP , H̃−1), we can then compute the posterior predictive distribution. While, traditionally, one would compute the predictive distribution using the original Bayesian neural network likelihood, i.e. p(y|X,W) = p(y|f(x,W )), Immer et al. (2020) recently suggested that, since inference was (implicitly) done in the GGN-linearized model, it is more principled to instead predict using the linearized likelihood Eq. (7), i.e. p(y|X,W) = p(y|fMAPlin (x,W )). This provides a formal justification for the empirical superiority of this approach observed previously (Lawrence, 2001; Foong et al., 2019b). We thus obtain the linearized predictive distribution
p(y∗|X∗,y,X) ≈ ∫ W p(y∗|fMAPlin (X∗,W ))N (WS ;W SMAP , H̃−1) ∏ r δ(wr−w∗r) dW . (8)
There are two ways to compute Eq. (8): Firstly, via a Monte Carlo approximation p(y∗|X∗,y,X) ' 1 M ∑M m=1 p(y ∗|fMAPlin (X∗, W̃m)) by sampling W̃m from N (W SMAP , H̃−1) and ∏
r δ(wr − w∗r), the latter of which is trivial. Secondly, due to linearity of p(y∗|fMAPlin (X∗,W )), there are closedform expressions which are exact for Gaussian likelihoods (i.e. regression) and approximate for categorical ones (i.e. classification) (Bishop, 2006; Gibbs, 1998). This step is illustrated in Fig. 1 (d).

heading: 4 PRINCIPLED SUBNETWORK SELECTION FOR LINEAR(IZED) MODELS

text:
We next analyze the subnetwork inference procedure described in Section 3 for the case of a generalized linear model (GLM) (Nelder & Baker, 1972), which models the expected response yn given the basis function expansion of the covariates φn = φ(xn) as
E[yn|φn] = g−1(wTφn). (9) Here, w ∈ RD is the vector of model parameters (which subsumes a scalar bias β0 for notational convenience) and g−1(·) denotes a link function such that g−1 : R 7→ µy|φ. In particular, we consider a Bayesian GLM, by specifying a prior distribution p(w) over model parameters and aiming to infer the posterior distribution p(w|y,Φ) ∝ p(y|Φ,w)p(w), where Φ = [φ1, ...φN ]T .
1. Point Estimation. Obtain the MAP estimate, wMAP = argmaxw log p(y|Φ,w) + log p(w). For commonly-used link functions (e.g. the identity in case of a Gaussian likelihood for regression, or the sigmoid/softmax function in case of a categorical likelihood for classification) and commonly-used priors (e.g. a Gaussian), the log-posterior ∝ log p(y|Φ,w) + log p(w) is concave. This allows for simple gradient-based MAP optimisation. It also makes a full-covariance Gaussian, estimated via Laplace, a faithful approximation to the true, uni-modal posterior, i.e.
p(w|y,Φ) ≈ p̃(w|y,Φ) = N (w;wMAP , H−1) (10)
where H is the Hessian defined in Section 3. Note that for the GLM we consider, the Hessian H is equivalent to the GGN H̃ defined in Eq. (6), meaning that an ordinary Laplace approximation is equivalent to a linearized Laplace approximation (Martens, 2016). For the case of an identity link function (i.e. a Gaussian likelihood with noise variance σ20) and a Gaussian prior w ∼ N (0,Λ−10 ), the MAP estimate even has a closed-form expression, wMAP = (ΦTΦ + σ20Λ0)
−1ΦTy. Here, the Laplace approximation in Eq. (10) exactly corresponds to the true posterior, i.e. p̃(w|y,Φ) = p(w|y,Φ). We will thus refer to the posterior p̃(w|y,Φ) in Eq. (10) as the full posterior.
2. Subnetwork Selection. Select a subset of S model weights via a method of choice, yielding a binary vector m ∈ RD where md = 1 if the d-th weight is part of the subset, and md = 0 otherwise. For convenience, we define the binary mask matrix MS =mm> ∈ RD×D which contains 1s in the rows/columns corresponding to the S subnetwork weights2, and 0s otherwise.
3. Bayesian Inference. Compute the posterior over the subnetwork via a Laplace approximation: pS(w|y,Φ) = N (w;wMAP ,MS H−1) . (11)
Firstly, note that the mean of the subnetwork posterior in Eq. (11) is the MAP estimate wMAP and thus equal to the mean of the full posterior p̃(w|y,Φ) in Eq. (10). Secondly, note that the covariance matrix of the subnetwork posterior in Eq. (11) is the element-wise productMS H−1, which masks the (co-)variances of all weights not belonging to the subnetwork to zero, effectively making them deterministic. More precisely, the subnetwork covariance matrix,MS H−1, is a D ×D matrix that is equal to the full posterior covariance matrix H−1 in the rows/columns of the S weights in the subnetwork, and zero in the rows/columns of all other D − S weights.
We consider what we term the posterior gap—the Wasserstein distance3 (in particular the squared 2-Wasserstein distance) between the posterior distribution over the full network and the posterior distribution over the subnetwork. The proofs for all results below will be presented in Appendix A. Proposition 1 (Posterior Gap). For a subnetwork of size S < D, the Wasserstein gap between the full posterior p̃(w|y,Φ) in Eq. (10) and the subnetwork posterior pS(w|y,Φ) in Eq. (11) is: W [p̃(w|y,Φ) || pS(w|y,Φ)] = ∑D d=1 (1 +mdd)σ 2 d − trace(2(H−1(MS H−1))1/2) . (12)
The optimal subnetwork should then minimize the posterior gap in Eq. (12). However, for full covariance matrices H−1 and a large number of weights D, this will generally be infeasible as Eq. (12) depends on all entries of the D ×D-matrix H−1, which is intractable to compute/store. To derive a practical subnetwork selection strategy, we assume the covariance matrix to be diagonal. Corollary 1.1 (Optimality of Maximum Variance Subnetwork Selection under Decorrelation). For a generalized linear model with posterior covariance matrix H−1 = diag(σ21 , . . . , σ 2 D), the optimal subnetwork under the Wasserstein gap is comprised of the S weights with the largest variances σ2d.
Finally, since a GGN-linearized neural network, as in Eq. (7), corresponds to a GLM with basis expansion φn = JWMAP (xn) = ∂f(xn,WMAP )/∂WMAP (see Step #3 in Section 3), Corollary 1.1 implies that under decorrelation, the optimal subnetwork comprises of the weights with the largest variances. In practice, even just computing the diagonal of the covariance matrix is challenging, so we use a diagonal Laplace approximation which instead computes the inverse of the diagonal of the GGN (see e.g. Ritter et al. (2018)). Finally, note that we only have to make the decorrelation assumption for the purposes of subnetwork selection – when doing posterior inference over the selected subnetwork, we estimate a full covariance matrix for maximal expressiveness, as described in Step #3 in Section 3. In our experiments in Section 5, we empirically show that making the decorrelation assumption for subnetwork selection but then using a full-covariance Gaussian for inference performs significantly better than directly making the decorrelation assumption for inference (e.g. mean-field variational inference, diagonal Laplace).

heading: 5 EMPIRICAL ANALYSIS

text:
We empirically assess the effectiveness of subnetwork inference compared to point-estimated NNs and methods that do less expressive inference over the full network. We consider three tasks: 1) small-scale toy regression, 2) medium-scale tabular regression, and 3) large-scale image classification.
2For consistency, we will keep referring to the S selected linear model weights as a ”subnetwork”. 3We use the Wasserstein distance instead of the more common Kullback–Leibler divergence because the
Wasserstein is well-defined for degenerate distributions and is an actual distance metric (i.e. symmetric).

heading: 5.1 HOW DOES SUBNETWORK INFERENCE RETAIN POSTERIOR PREDICTIVE UNCERTAINTY?

text:
We first assess how the predictive distribution of a full-covariance Gaussian posterior over a selected subnetwork qualitatively compares to that obtained from 1) a full-covariance Gaussian over the full network (Full Cov), 2) a factorised Gaussian posterior over the full network (Diag), 3) a fullcovariance Gaussian over only the (Final layer) of the network (Kristiadi et al., 2020), and 4) a point estimate (MAP). For subnetwork inference, we consider both Wasserstein (Wass) (as described in Section 4) and uniform random subnetwork selection (Rand) to obtain subnetworks that comprise of only 50%, 3% and 1% of the model parameters. Note that while for this toy example, we could in principle use the full covariance matrix for the purpose of subnetwork selection, we still just use its diagonal (as described in Section 4) for consistency. Our NN consists of 2 ReLU hidden layers with 50 hidden units each. We employ a homoscedastic Gaussian likelihood function where the noise variance is optimised with maximum likelihood. We use GGN Laplace inference over network weights (not biases) in combination with the linearized predictive distribution in Eq. (8). Thus, all approaches considered share their predictive mean, allowing us to better compare their uncertainty estimates. All approaches share a single prior precision of λ = 3.
We use a synthetic 1D regression task with two separated clusters of inputs (Antorán et al., 2020), allowing us to probe for ‘in-between’ uncertainty (Foong et al., 2019b). Results are shown in Fig. 2. Subnetwork inference preserves more of the uncertainty of full network inference than diagonal Gaussian or final layer inference while doing inference over fewer weights. By capturing weight correlations, subnetwork inference retains uncertainty in between clusters of data. This is true for both random and Wasserstein subnetwork selection. However, the latter preserves more uncertainty with smaller subnetworks. Finally, the strong superiority to diagonal Laplace shows that making a diagonal assumption for subnetwork selection but then using a full-covariance Gaussian for inference (as we do) performs much signficantly better than making a diagonal assumption for the inferered posterior directly. These results suggest that expressive inference over a carefully selected subnetwork retains more predictive uncertainty than crude approximations over the full network.

heading: 5.2 SUBNETWORK INFERENCE IN LARGE MODELS VS FULL INFERENCE OVER SMALL MODELS

text:
Secondly, we study the following natural question: “Why should one use subnetwork inference in a large model when one can just perform full network inference over a smaller model?” We explore this by considering 4 fully connected models of increasing size. These have numbers of hidden layers hd={1, 2} and hidden layer widths wd={50, 100}. For a dataset with input dimension id, the number of weights is given by D=(id+1)wd+(hd−1)w2d. Our 2 hidden layer, 100 hidden unit models have a weight count of the order 104. Full covariance inference in these models borders the limit of computational tractability on commercial hardware. We first obtain a MAP estimate of each
model’s weights and our homoscedastic likelihood function’s noise variance. We then perform full network GGN Laplace inference for each model. We also use our proposed Wassertein rule to prune every network’s weight variances such that the number of variances that remain matches the size of every smaller network under consideration. In all cases, we employ the linearized predictive in Eq. (7). Consequently, networks with the same number of weights make the same mean predictions. Increasing the number of weight variances considered will thus only increase predictive uncertainty.
We employ 3 tabular datasets of increasing size (input dimensionality, n. points): wine (11, 1439), kin8nm (8, 7373) and protein (9, 41157). We consider their standard train-test splits (HernándezLobato & Adams, 2015) and their gap variants (Foong et al., 2019b), designed to test for out-ofdistribution uncertainty. Details are provided in Appendix C.4. For each split, we set aside 15% of the train data as a validation set. We use these for early stopping when finding MAP estimates and for selecting the weights’ prior precision. We keep other hyperparameters fixed across all models and datasets. Results are in Fig. 3.
We present mean test log-likelihood (LL) values, as these take into account both accuracy and uncertainty. Larger models tend to perform better when doing MAP inference, with wine-gap and protein-gap being exceptions. We also find larger models improve over their respective MAP LLs more than small ones when performing approximate inference over the same numbers of weights. We conjecture this is due to an abundance of degenerate directions (weights) in the weight posterior of all models (Maddox et al., 2020). Full network inference in small models captures information about both useful and non-useful weights. In larger models, our subnetwork selection strategy allows us to dedicate a larger proportion of our resources to modelling informative weight variances and covariances. In 3 out of 6 datasets, we find abrupt increases in LL as we increase the number of weights over which we perform inference, followed by a plateau. Such plateaus might be explained by all of the most informative weight variances having already been accounted for. These results suggest that, given the same amount of compute, larger models benefit more from subnetwork inference than small ones.

heading: 5.3 SCALING TO IMAGE CLASSIFICATION WITH DISTRIBUTION SHIFT

text:
We now assess the robustness of large convolutional neural networks with subnetwork inference to distribution shift on image classification tasks compared to the following baselines: point-estimated networks (MAP), Bayesian deep learning methods that do less expressive inference over the full network: MC Dropout (Gal & Ghahramani, 2016), diagonal Laplace, VOGN (Osawa et al., 2019) (all of which assume factorisation of the weight posterior), and SWAG (Maddox et al., 2019) (which assumes a diagonal plus low-rank posterior). We also benchmark deep ensembles (Lakshminarayanan et al., 2017). The latter is considered state-of-the-art for uncertainty quantification in deep learning (Ovadia et al., 2019; Ashukha et al., 2020a). We use ensembles of 5 DNNs, as suggested by (Ovadia et al., 2019), and 16 samples for MC Dropout, diagonal Laplace and SWAG. We use a Dropout probability of 0.1 and a prior precision of λ = 40, 000 for diagonal Laplace, found via grid search. We apply all approaches to ResNet-18 (He et al., 2016), which is composed of an input convolutional block, 8 residual blocks and a linear layer, for a total of 11,168,000 parameters. For subnetwork inference, we compute the linearized predictive distribution in Eq. (8) via the closed-form
approximation for integrals between Gaussians and multi-class cross-entropy likelihoods described in (Gibbs, 1998). We use Wasserstein subnetwork selection to retain only 0.38% of the weights, yielding a subnetwork with only 42,438 weights. This is the largest subnetwork for which we can tractably compute a full covariance matrix. Its size is 42, 4382 × 4 Bytes ≈ 7.2 GB. We use a prior precision of λ = 500, found via grid search. Finally, to assess to importance of principled subnetwork selection, we also consider the baseline where we select the subnetwork uniformly at random (called Ours (Rand)). We perform the following two experiments, with results in Fig. 4. See Appendix B for additional results.
Rotated MNIST: Following Ovadia et al. (2019); Antorán et al. (2020), we train all methods on MNIST and evaluate their predictive distributions on increasingly rotated digits. While all methods perform well on the original MNIST test set, their accuracy degrades quickly for rotations larger than 30 degrees. In terms of LL, ensembles perform best out of our baselines. Subnetwork inference obtains significantly larger LL values than almost all baselines, including ensembles. The only exception is VOGN, which achieves slightly better performance. It was also observed in Ovadia et al. (2019) that mean-field variational inference (which VOGN also is an instance of) is very strong on MNIST, but its performance deteriorates on larger datasets. Subnetwork inference makes accurate predictions in-distribution while assigning higher uncertainty than the baselines to out-of-distribution points. Corrupted CIFAR: Again following Ovadia et al. (2019); Antorán et al. (2020), we train on CIFAR10 and evaluate on data subject to 16 different corruptions with 5 levels of intensity each (Hendrycks & Dietterich, 2019). Our approach matches a MAP estimated network in terms of predictive error as local linearization makes their predictions the same. Ensembles and SWAG are the most accurate. Even so, subnetwork inference differentiates itself by being the least overconfident, outperforming all baselines in terms of log-likelihood at all corruption levels. Here, VOGN performs rather badly; while this might appear in stark contrast to its strong performance on the MNIST benchmark, the behaviour that mean-field VI performs well on MNIST but not on larger datasets was also observed in Ovadia et al. (2019).
On both benchmarks, we furthermore find that randomly selecting the subnetwork performs substantially worse than using our more principled subnetwork selection strategy. This highlights the importance of the way subnetworks are selected. These results suggest that subnetwork inference results in better uncertainty calibration and robustness to distribution shift than other popular uncertainty quantification approaches.

heading: 6 RELATED WORK

text:
Bayesian Deep Learning. There have significant efforts to characterise the posterior distribution over NN weights p(W |D). Hamiltonian Monte Carlo (Neal, 1995) remains the golden standard
for approximate inference in BNNs to this day. Although asymptotically unbiased, sampling based approaches are difficult to scale to the large datasets (Betancourt, 2015). As a result, approaches which find the best surrogate posterior among an approximating family (most often Gaussians) have gained popularity. The first of these was the Laplace approximation, introduced by MacKay (1992), who also proposed approximating the predictive posterior with that of the linearised model (Khan et al., 2019; Immer et al., 2020). The popularisation of larger NN models has made surrogate distributions that capture correlations between weights computationally intractable. Thus, most modern methods make use of the mean field assumption (Blundell et al., 2015; Hernández-Lobato & Adams, 2015; Gal & Ghahramani, 2016; Mishkin et al., 2018; Osawa et al., 2019). This comes at the cost of limited expressivity (Foong et al., 2019a) and empirical under-performance (Ovadia et al., 2019; Antorán et al., 2020) of uncertainty estimates. Our proposed approach recovers predictive posterior expressivity while maintaining tractability by lowering the dimensionality of the weight space considered. This allows us to scale up approximations that do consider weight correlations (MacKay, 1992; Louizos & Welling, 2016; Maddox et al., 2019; Ritter et al., 2018).
Neural Network Linearization. In the limit of infinite width, NNs converge to Gaussian process (GP) behaviour (Neal, 1995; Matthews, 2017; Garriga-Alonso et al., 2018). Recently, these results have been extended to finite width BNNs when the surrogate posterior is Gaussian (Khan et al., 2019). We draw upon these results to formulate a subnetwork selection strategy for BNNs. Neural linear methods perform inference over only the last layer of a NN, while keeping all other layers fixed (Snoek et al., 2015; Riquelme et al., 2018; Ovadia et al., 2019; Ober & Rasmussen, 2019; Pinsler et al., 2019; Kristiadi et al., 2020). These represent a different generalised linear model in which the basis functions are defined by the l−1 first layers of a NN. They can also be viewed as a special case of subnetwork inference, in which the subnetwork is simply defined to be the last NN layer.
Inference over Subspaces. The subfield of NN pruning aims to increase the computational efficiency of NNs by identifying the smallest subset of weights which are required to make accurate predictions. Approaches trade-off computational cost with compression efficiency, ranging from those that require multiple training runs (Frankle & Carbin, 2019) to those that prune before training (Wang et al., 2020). Our work differs in that it retains all NN weights but aims to find a small subset over which to perform probabilistic reasoning. More closely related work to ours is that of Izmailov et al. (2019), who propose to perform inference over a low-dimensional subspace of weights; e.g. one constructed from the principal components of the SGD trajectory. Moreover, several recent approaches use low-rank parameterizations of approximate posteriors in the context of variational inference (Rossi et al., 2019; Swiatkowski et al., 2020; Dusenberry et al., 2020). This could also be viewed as doing inference over an implicit subspace of weight space. In contrast, we propose a technique to find subsets of weights which are relevant to predictive uncertainty, i.e., we identify axis aligned subspaces. Finally, there have been recent works studying neural network sparsity / pruning from a Bayesian perspective (Ghosh & Doshi-Velez, 2017; Polson & Ročková, 2018; Cui et al., 2020; Louizos et al., 2017; Molchanov et al., 2017; Gomez et al., 2019; Lee et al., 2018). While these seem conceptually related at first glance, their goal is fundamentally different to ours: While those methods aim to perform model selection / sparsification by either explicitly or implicitly pruning unnecessary weights, our goal is to make inference more tractable. More precisely, while those sparse Bayesian deep learning methods prune individual weights, we instead just prune the variances of certain weights, which, importantly retains the full predictive power of the full network to retain high predictive accuracy.

heading: 7 CONCLUSION

text: In this paper, we develop a practical and scalable method for expressive yet tractable probabilistic inference in deep neural networks. We approximate the posterior over a subset of the weights while keeping all other weights deterministic. Computational cost is decoupled from network size, allowing us to scale expressive approximations, such as full-covariance Gaussian distributions, to real-world sized NNs. Our approach can be applied post-hoc to any pre-trained model, making it particularly attractive for practical use. Our empirical analysis suggests that subnetwork inference 1) is more expressive and retains more uncertainty than crude approximations over the full network, 2) allows us to employ larger NNs, which fit a broader range of functions, without sacrificing the quality of our uncertainty estimates, and 3) is competitive with state-of-the-art uncertainty quantification methods, like deep ensembles (Lakshminarayanan et al., 2017), on real-world scale problems.

heading: A PROOFS FOR THE THEORETICAL RESULTS

text: We now provide the proofs for the results in Section 4.

heading: A.1 PROOF OF PROPOSITION 1

text:
Proof. Note that the posterior distributions p̃(w|y,Φ) and pS(w|y,Φ) are both Gaussian. We thus consider the squared 2-Wasserstein distance between two Gaussian distributions N (µ1,Σ1) and N (µ2,Σ2), which has the following closed-form expression (Givens et al., 1984)4:
W [N (µ1,Σ1) || N (µ2,Σ2)] = ‖µ1 − µ2‖22 + trace ( Σ1 + Σ2 − 2 (Σ1Σ2)1/2 ) . (13)
Plugging in µ1 = µ2 = wMAP , Σ1 = H−1 and Σ2 =MS H−1, we obtain W [p̃(w|y,X) || pS(w|y,X)]
=W [ N (wMAP , H−1) || N (wMAP ,MS H−1) ] =((( (((( ((‖wMAP −wMAP ‖22 + trace ( H−1 + (MS H−1)− 2 ( H−1(MS H−1)
)1/2) = trace ( (1 +MS) H−1 ) − trace ( 2 ( H−1 ( MS H−1
))1/2) =
D∑ d=1 (1 +mdd)σ 2 d − trace ( 2 ( H−1 ( MS H−1 ))1/2)

heading: A.2 PROOF OF COROLLARY 1.1

text:
Proof. For H−1 = diag(σ21 , . . . , σ 2 D), the Wasserstein posterior gap in Eq. (12) simplifies to
W [p̃(w|y,Φ) || pS(w|y,Φ)] = D∑
d=1
( (1 +mdd)σ 2 d − 2mddσ2d ) . (14)
The optimal subnetwork selection strategy amounts to choosing the binary vectorm = [mdd]Dd=1 with∑D d=1md = S (i.e., we select S out of D parameters) s.t. the posterior gap in Eq. (14) is minimized. Observing that the contribution of the d-th parameter to the posterior gap is (1+ 1)σ2d − 1× 2σ2d = 0 if it is selected (i.e. if mdd = 1), and (1 + 0)σ2d − 0× 2σ2d = σ2d if it is not selected (i.e. if mdd = 0), we see that the optimal subnetwork comprises of the S weights with the largest variances σ2d.

heading: B ADDITIONAL IMAGE CLASSIFICATION RESULTS

heading: C EXPERIMENTAL SETUP

heading: C.1 TOY EXPERIMENTS

text:
We train a single, 2 hidden layer network, with 50 hidden ReLU units per layer using MAP inference until convergence. Specifically, we use SGD with a learning rate of 1× 10−3, momentum of 0.9 and weight decay of 1× 10−4. We use a batch size of 512. The objective we optimise is the Gaussian log-likelihood of our data, where the mean is outputted by the network and the the variance is a hyperparameter learnt jointly with NN parameters by SGD. This variance parameters is shared among all datapoints. Once the network is trained, we perform post-hoc inference on it using different approaches. Since all of these involve the linearized approximation, the mean prediction is the same for all methods. Only their uncertainty estimates vary.
Note that while for this toy example, we could in principle use the full covariance matrix for the purpose of subnetwork selection, we still just use its diagonal (as described in Section 4) for consistency. We use GGN Laplace inference over network weights (not biases) in combination with the linearized predictive distribution in Eq. (8). Thus, all approaches considered share their predictive mean, allowing us to better compare their uncertainty estimates.
All approaches share a single prior precision of λ = 3. We chose to select the prior precision such that the full covariance approach (optimistic baseline) presents reasonable results. We use the same value for all other methods. We first tried a precision of 1 and found the full covariance approach to produce excessively large errorbars (covering the whole plot). A value of 3 produces more reasonable results.
Final layer inference is performed by computing the full Laplace covariance matrix and discarding all entries except those corresponding to the final layer of the NN. Results for random sub-network selection are obtained with a single sample from a scaled uniform distribution over weight choice.

heading: C.2 UCI EXPERIMENTS

text:
In this experiment, our fully connected NNs have numbers of hidden layers hd={1, 2} and hidden layer widths wd={50, 100}. For a dataset with input dimension id, the number of weights is given by D=(id+1)wd+(hd−1)w2d. Our 2 hidden layer, 100 hidden unit models have a weight count of the order 104. The non-linearity used is ReLU.
We first obtain a MAP estimate of each model’s weights. Specifically, we use SGD with a learning rate of 1× 10−3, momentum of 0.9 and weight decay of 1× 10−4. We use a batch size of 512. The objective we optimise is the Gaussian log-likelihood of our data, where the mean is outputted by the network and the the variance is a hyperparameter learnt jointly with NN parameters by SGD.
For each dataset split, we set aside 15% of the train data as a validation set. We use these for early stopping training. Training runs for a maximum of 2000 epochs but early stops with a patience of 500 if validation performance does not increase. For the larger Protein dataset, these values are 500 and 125. The weight settings which provide best validation performance are kept.
We then perform full network GGN Laplace inference for each model. We also use our proposed Wassertein rule together with the diagonal Hessian assumption to prune every network’s weight variances such that the number of variances that remain matches the size of every smaller network under consideration. The prior precision used for these steps is chosen such that the resulting predictor’s logliklihood performance on the validation set is maximised. Specifically, we employ a grid search over the values: λ : [0.0001, 0.001, 0.1, 0.5, 1, 2, 5, 10, 100, 1000]. In all cases, we employ the linearized predictive in Eq. (7). Consequently, networks with the same number of weights make the same mean predictions. Increasing the number of weight variances considered will thus only increase predictive uncertainty.
C.3 IMAGE EXPERIMENTS
The results shown in Section 5.3 and Appendix B are obtained by training ResNet-18 (and ResNet-50) models using SGD with momentum. For each experiment repetition, we train 7 different models: The first is for: ‘MAP’, ‘Ours’, ‘Ours (Rand)’, ‘SWAG’, ‘Diag-Laplace’ and as the first element of
‘Ensemble’. We train 4 additional ‘Ensemble’ elements, 1 network with ‘Dropout’, and, finally 1 network for ‘VOGN’. The methods ‘Ours’, ‘Ours (Rand)’, ‘SWAG’, and ‘Diag-Laplace’ are applied post training.
For all methods except ‘VOGN’ we use the following training procedure. The (initial) learning rate, momentum, and weight decay are 0.1, 0.9, and 1× 10−4, respectively. For ‘MAP’ we use 4 Nvidia P100 GPUs with a total batch size of 2048. For the calculation of the Jacobian in the subnetwork selection phase we use a single P100 GPU with a batch size of 4. For the calculation of the hessian we use a single P100 GPU with a batch size of 2. We train on 1 Nvidia P100 GPU with a batch size of 256 for all other methods. Each dataset is trained for a different number of epochs, shown in Table 21. We decay the learning rate by a factor of 10 at scheduled epochs, also shown in Table 21. Otherwise, all methods and datasets share hyperparameters. These hyperparameter settings are the defaults provided by PyTorch for training on ImageNet. We found them to perform well across the board. We report results obtained at the final training epoch. We do not use a separate validation set to determine the best epoch as we found ResNet-18 and ResNet-50 to not overfit with the chosen schedules.
For ‘Dropout’, we add dropout to the standard ResNet-50 model (He et al., 2016) in between the 2 and 3 convolutions in the bottleneck blocks. This approach follows Zagoruyko & Komodakis (2016) and Ashukha et al. (2020b) who add dropout in-between the two convolutions of a WideResNet-50’s basic block. Following Antorán et al. (2020), we choose a dropout probability of 0.1, as they found it to perform better than the value of 0.3 suggested by Ashukha et al. (2020b). We use 16 MC samples for predictions. ‘Ensemble’ uses 5 elements for prediction. Ensemble elements differ from each other in their initialisation, which is sampled from the He initialisation distribution (He et al., 2015). We do not use adversarial training as, inline with Ashukha et al. (2020b), we do not find it to improve results. For ‘VOGN’ we use the same procedure and hyper-parameters as used by Osawa et al. (2019) in their CIFAR10 experiments, with the exception that we use a learning rate of 1 × 10−3 as we we found a value of 1 × 10−4 not to result in convergence. We train on a single Nvidia P100 GPU with a batch size of 256. See the authors’ GitHub for more details: github.com/team-approx-bayes/dl-with-bayes/blob/master/ distributed/classification/configs/cifar10/resnet18_vogn_bs256_ 8gpu.json.
We modify the standard ResNet-50 and ResNet-18 architectures such that the first 7× 7 convolution is replaced with a 3× 3 convolution. Additionally, we remove the first max-pooling layer. Following Goyal et al. (2017), we zero-initialise the last batch normalisation layer in residual blocks so that they act as identity functions at the start of training.
At test time, we tune the prior precision used for ‘Ours’, ‘Diag-Laplace’ and ‘SWAG’ approximation on a validation set for each approach individually, as in Ritter et al. (2018); Kristiadi et al. (2020). We use a grid search from 1× 10−4 to 1× 104 in logarithmic steps, and then a second, finer-grained grid search between the two best performing values (again with logarithmic steps).

heading: C.4 DATASETS

text:
The 1d toy dataset used in Section 5.1 was taken from Antorán et al. (2020). We obtained it from the authors’ github repo: https://github.com/cambridge-mlg/DUN. Table 22 summarises the datasets used in Section 5.2. Wine and Protein are available from the UCI dataset repository Dua & Graff (2017). Kin8nm is available from https://www.openml.org/d/189 Foong et al. (2019b). For the standard splits (Hernández-Lobato & Adams, 2015) 90% of the data is used for training and 10% for validation. For the gap splits (Foong et al., 2019b) a split is obtained per input dimension by ordering points by their values across that dimension and removing the middle 33% of
the points. These are used for validation. The datasets used for our image experiments are outlined in Table 23.

heading: NAME SIZE INPUT DIM. NO. CLASSES NO. SPLITS