summaries: The authors present a new method for Bayesian deep learning motivated by the difficulty of posterior inference in the "overparameterized" regime of deep neural network models. The proposed method provides a principled strategy for selecting a subset of the neural network's parameters (forming a so-called "subnetwork") for which a full-covariance approximate posterior can be computed. The authors use the well-studied Laplace approximation with the generalized Gauss-Newton Hessian approximation for the covariance. An empirical analysis is presented which attempts to assess the efficacy of the proposed method in prediction accuracy and uncertainty quantification. The paper proposes to approximate the posterior distribution of a Bayesian neural network by an approximation that consists of a deterministic component. The authors select a sub network and infer approximate posterior distributions over the weights in the sub network. All other weights are estimated via MAP point estimation.  A sufficiently small sub-network allows high fidelity posterior approximations that do not make restrictive mean field assumptions to be tractable. The authors focus on the important problem of scalable approximate inference in Bayesian NNs. More specifically, they propose a method for scalable BNNs via a (full-covariance Gaussian) Laplace approximation on a (Wasserstein-based) pruned subnetwork within a deterministically-trained model. They include a theoretical analysis for a simple generalized linear model, and experiments on 1D regression, tabular regression, and larger-scale image classification with CIFAR-10 (using the dataset shift setup from Ovadia et al., (2019)). From the experiments, they show that their method generally outperforms comparable methods (including deep ensembles) on metric performance and on the ability to capture in-between uncertainty.
abstractText: The Bayesian paradigm has the potential to solve some of the core issues in modern deep learning, such as poor calibration, data inefficiency, and catastrophic forgetting. However, scaling Bayesian inference to the high-dimensional parameter spaces of deep neural networks requires restrictive approximations. In this paper, we propose performing inference over only a small subset of the model parameters while keeping all others as point estimates. This enables us to use expressive posterior approximations that would otherwise be intractable for the full model. In particular, we develop a practical and scalable Bayesian deep learning method that first trains a point estimate, and then infers a full covariance Gaussian posterior approximation over a subnetwork. We propose a subnetwork selection procedure which aims to maximally preserve posterior uncertainty. We empirically demonstrate the effectiveness of our approach compared to point-estimated networks and methods that use less expressive posterior approximations over the full network.
authors.name: SUBNETWORK INFERENCE
references.authors: Dario Amodei Chris Olah Jacob Steinhardt Paul Christiano John Schulman Dan Mané
references.title: Concrete problems in ai safety
references.venue: arXiv preprint arXiv:1606.06565,
references.authors: Arsenii Ashukha Alexander Lyzhov Dmitry Molchanov Dmitry Vetrov
references.title: Pitfalls of in-domain uncertainty estimation and ensembling in deep learning
references.venue: In ICLR,
references.authors: Arsenii Ashukha Alexander Lyzhov Dmitry Molchanov Dmitry P. Vetrov
references.title: Pitfalls of in-domain uncertainty estimation and ensembling in deep learning
references.venue: In 8th International Conference on Learning Representations,
references.authors: Michael Betancourt
references.title: The fundamental incompatibility of scalable hamiltonian monte carlo and naive data subsampling
references.venue: Proceedings of Machine Learning Research, pp. 533–540,
references.authors: Christopher M Bishop
references.title: Pattern recognition and machine learning
references.venue: springer,
references.authors: Charles Blundell Julien Cornebise Koray Kavukcuoglu Daan Wierstra
references.title: Weight Uncertainty in Neural Networks
references.venue: In Proceedings of The 32nd International Conference on Machine Learning (ICML),
references.authors: Yu Cheng Duo Wang Pan Zhou Tao Zhang
references.title: A survey of model compression and acceleration for deep neural networks
references.venue: arXiv preprint arXiv:1710.09282,
references.authors: Tianyu Cui Aki Havulinna Pekka Marttinen Samuel Kaski
references.title: Informative gaussian scale mixture priors for bayesian neural networks
references.venue: arXiv preprint arXiv:2002.10243,
references.authors: Dheeru Dua Casey Graff
references.title: UCI machine learning repository, 2017
references.venue: URL http://archive. ics.uci.edu/ml
references.authors: Michael W Dusenberry Ghassen Jerfel Yeming Wen Yi-an Ma Jasper Snoek Katherine Heller Balaji Lakshminarayanan Dustin Tran
references.title: Efficient and scalable bayesian neural nets with rank-1 factors
references.authors: Angelos Filos Sebastian Farquhar Aidan N. Gomez Tim G.J. Rudner Zachary Kenton Lewis Smith Milad Alizadeh Arnoud de Kroon Yarin Gal
references.title: Benchmarking bayesian deep learning with diabetic retinopathy diagnosis
references.venue: https://github.com/OATML/bdl-benchmarks,
references.authors: Angelos Filos Sebastian Farquhar Aidan N Gomez Tim GJ Rudner Zachary Kenton Lewis Smith Milad Alizadeh Arnoud de Kroon Yarin Gal
references.title: Benchmarking bayesian deep learning with diabetic retinopathy diagnosis
references.authors: Andrew YK Foong David R Burt Yingzhen Li Richard E Turner
references.title: On the expressiveness of approximate inference in bayesian neural networks
references.venue: arXiv, pp. arXiv–1909,
references.authors: Andrew YK Foong Yingzhen Li José Miguel Hernández-Lobato Richard E Turner
references.title: In-between uncertainty in bayesian neural networks
references.venue: ICML Workshop on Uncertainty and Robustness in Deep Learning,
references.authors: Stanislav Fort Huiyi Hu Balaji Lakshminarayanan
references.title: Deep ensembles: A loss landscape perspective
references.venue: arXiv preprint arXiv:1912.02757,
references.authors: Jonathan Frankle Michael Carbin
references.title: The lottery ticket hypothesis: Finding sparse, trainable neural networks
references.venue: In International Conference on Learning Representations,
references.authors: Lex Fridman Li Ding Benedikt Jenik Bryan Reimer
references.title: Arguing machines: Human supervision of black box ai systems that make life-critical decisions
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops,
references.authors: Yarin Gal
references.title: Uncertainty in deep learning
references.venue: University of Cambridge,
references.authors: Yarin Gal Zoubin Ghahramani
references.title: Dropout as a bayesian approximation: Representing model uncertainty in deep learning
references.venue: In international conference on machine learning,
references.authors: Adrià Garriga-Alonso Carl Edward Rasmussen Laurence Aitchison
references.title: Deep convolutional networks as shallow gaussian processes
references.venue: In International Conference on Learning Representations,
references.authors: Zoubin Ghahramani
references.title: Probabilistic machine learning and artificial intelligence
references.venue: Nature, 521(7553):
references.authors: Soumya Ghosh Finale Doshi-Velez
references.title: Model selection in bayesian neural networks via horseshoe priors
references.venue: arXiv preprint arXiv:1705.10388,
references.authors: Mark N Gibbs
references.title: Bayesian Gaussian processes for regression and classification
references.venue: PhD thesis, Citeseer,
references.authors: Clark R Givens Rae Michael Shortt
references.title: A class of wasserstein metrics for probability distributions
references.venue: The Michigan Mathematical Journal,
references.authors: Aidan N Gomez Ivan Zhang Kevin Swersky Yarin Gal Geoffrey E Hinton
references.title: Learning sparse networks using targeted dropout
references.authors: Priya Goyal Piotr Dollár Ross B. Girshick Pieter Noordhuis Lukasz Wesolowski Aapo Kyrola Andrew Tulloch Yangqing Jia Kaiming He
references.title: Accurate, large minibatch SGD: Training ImageNet in 1 hour
references.authors: Chuan Guo Geoff Pleiss Yu Sun Kilian Q Weinberger
references.title: On calibration of modern neural networks
references.venue: In Proceedings of the 34th International Conference on Machine Learning-Volume
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification
references.venue: In 2015 IEEE International Conference on Computer
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Dan Hendrycks Thomas Dietterich
references.title: Benchmarking neural network robustness to common corruptions and perturbations
references.venue: arXiv preprint arXiv:1903.12261,
references.authors: José Miguel Hernández-Lobato Ryan Adams
references.title: Probabilistic backpropagation for scalable learning of bayesian neural networks
references.venue: In International Conference on Machine Learning,
references.authors: Alexander Immer Maciej Korzepa Matthias Bauer
references.title: Improving predictions of bayesian neural networks via local linearization
references.venue: ICML Workshop on Uncertainty and Robustness in Deep Learning,
references.authors: Pavel Izmailov Wesley J Maddox Polina Kirichenko Timur Garipov Dmitry Vetrov Andrew Gordon Wilson
references.title: Subspace inference for bayesian deep learning
references.venue: In 35th Conference on Uncertainty in Artificial Intelligence,
references.authors: Mohammad Emtiyaz Khan Didrik Nielsen Voot Tangkaratt Wu Lin Yarin Gal Akash Srivastava
references.title: Fast and scalable bayesian deep learning by weight-perturbation in adam
references.venue: arXiv preprint arXiv:1806.04854,
references.authors: Mohammad Emtiyaz E Khan Alexander Immer Ehsan Abedi Maciej Korzepa
references.title: Approximate inference turns deep networks into gaussian processes
references.venue: In Advances in neural information processing systems,
references.authors: Durk P Kingma Tim Salimans Max Welling
references.title: Variational dropout and the local reparameterization trick
references.venue: In Advances in neural information processing systems,
references.authors: Agustinus Kristiadi Matthias Hein Philipp Hennig
references.title: Being bayesian, even just a bit, fixes overconfidence in relu networks
references.venue: arXiv preprint arXiv:2002.10118,
references.authors: Alex Krizhevsky Geoffrey Hinton
references.title: Learning Multiple Layers of Features from Tiny Images
references.venue: Technical report, University of Toronto,
references.authors: Balaji Lakshminarayanan Alexander Pritzel Charles Blundell
references.title: Simple and scalable predictive uncertainty estimation using deep ensembles
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Neil David Lawrence
references.title: Variational inference in probabilistic models
references.venue: PhD thesis, University of Cambridge,
references.authors: Yann LeCun Léon Bottou Yoshua Bengio Patrick Haffner
references.title: Gradient-based learning applied to document recognition
references.venue: Proceedings of the IEEE,
references.authors: Juho Lee Saehoon Kim Jaehong Yoon Hae Beom Lee Eunho Yang Sung Ju Hwang
references.title: Adaptive network sparsification via dependent variational beta-bernoulli dropout
references.authors: Ekaterina Lobacheva Nadezhda Chirkova Maxim Kodryan Dmitry P Vetrov
references.title: On power laws in deep ensembles
references.venue: Advances in Neural Information Processing Systems,
references.authors: Christos Louizos Max Welling
references.title: Structured and efficient variational deep learning with matrix gaussian posteriors
references.venue: In International Conference on Machine Learning,
references.authors: Christos Louizos Max Welling Diederik P Kingma
references.title: Learning sparse neural networks through l 0 regularization
references.venue: arXiv preprint arXiv:1712.01312,
references.authors: David JC MacKay
references.title: A practical bayesian framework for backpropagation networks
references.venue: Neural computation,
references.authors: Wesley J Maddox Pavel Izmailov Timur Garipov Dmitry P Vetrov Andrew Gordon Wilson
references.title: A simple baseline for bayesian uncertainty in deep learning
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Wesley J Maddox Gregory Benton Andrew Gordon Wilson
references.title: Rethinking parameter counting in deep models: Effective dimensionality revisited
references.venue: arXiv preprint arXiv:2003.02139,
references.authors: James Martens
references.title: Second-order optimization for neural networks
references.venue: University of Toronto (Canada),
references.authors: James Martens Ilya Sutskever
references.title: Learning recurrent neural networks with hessian-free optimization
references.venue: In Proceedings of the 28th international conference on machine learning
references.title: Scalable Gaussian process inference using variational methods
references.venue: PhD thesis, University of Cambridge,
references.authors: Aaron Mishkin Frederik Kunstner Didrik Nielsen Mark Schmidt Mohammad Emtiyaz Khan
references.title: Slang: Fast structured covariance approximations for bayesian deep learning with natural gradient
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Dmitry Molchanov Arsenii Ashukha Dmitry Vetrov
references.title: Variational dropout sparsifies deep neural networks
references.venue: In Proceedings of the 34th International Conference on Machine Learning-Volume
references.authors: Eric Nalisnick Akihiro Matsukawa Yee Whye Teh Dilan Gorur Balaji Lakshminarayanan
references.title: Do Deep Generative Models Know What They Don’t Know
references.venue: In International Conference on Learning Representations (ICLR),
references.authors: Radford M. Neal
references.title: Bayesian Learning for Neural Networks
references.venue: PhD thesis, CAN,
references.authors: John Ashworth Nelder R Jacob Baker
references.title: Generalized Linear Models
references.venue: Wiley Online Library,
references.authors: Yuval Netzer Tao Wang Adam Coates Alessandro Bissacco Bo Wu Andrew Y Ng
references.title: Reading Digits in Natural Images with Unsupervised Feature Learning
references.venue: In NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning,
references.authors: Anh Nguyen Jason Yosinski Jeff Clune
references.title: Deep neural networks are easily fooled: High confidence predictions for unrecognizable images
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Sebastian W Ober Carl Edward Rasmussen
references.title: Benchmarking the neural linear model for regression
references.venue: arXiv preprint arXiv:1912.08416,
references.authors: Kazuki Osawa Siddharth Swaroop Anirudh Jain Runa Eschenhagen Richard E Turner Rio Yokota Mohammad Emtiyaz Khan
references.title: Practical deep learning with Bayesian principles
references.authors: Yaniv Ovadia Emily Fertig Balaji Lakshminarayanan Sebastian Nowozin D Sculley Joshua Dillon Jie Ren Zachary Nado Jasper Snoek
references.title: Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Robert Pinsler Jonathan Gordon Eric Nalisnick José Miguel Hernández-Lobato
references.title: Bayesian batch active learning as sparse subset approximation
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Nicholas G Polson Veronika
references.title: Ročková. Posterior concentration for sparse deep learning
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Carlos Riquelme George Tucker Jasper Snoek
references.title: Deep bayesian bandits showdown: An empirical comparison of bayesian deep networks for thompson sampling
references.venue: In International Conference on Learning Representations,
references.authors: Hippolyt Ritter Aleksandar Botev David Barber
references.title: A scalable laplace approximation for neural networks
references.venue: In International Conference on Learning Representations,
references.authors: Simone Rossi Sebastien Marmin Maurizio Filippone
references.title: Walsh-hadamard variational inference for bayesian deep learning
references.venue: arXiv preprint arXiv:1905.11248,
references.authors: Nicol N Schraudolph
references.title: Fast curvature matrix-vector products for second-order gradient descent
references.venue: Neural computation,
references.authors: Jasper Snoek Oren Rippel Kevin Swersky Ryan Kiros Nadathur Satish Narayanan Sundaram Mostofa Patwary Mr Prabhat Ryan Adams
references.title: Scalable bayesian optimization using deep neural networks
references.venue: In International conference on machine learning,
references.authors: Jakub Swiatkowski Kevin Roth Bastiaan S Veeling Linh Tran Joshua V Dillon Stephan Mandt Jasper Snoek Tim Salimans Rodolphe Jenatton Sebastian Nowozin
references.title: The k-tied normal distribution: A compact parameterization of gaussian mean field posteriors in bayesian neural networks
references.venue: arXiv preprint arXiv:2002.02655,
references.authors: Chaoqi Wang Guodong Zhang Roger Grosse
references.title: Picking winning tickets before training by preserving gradient flow
references.venue: In International Conference on Learning Representations,
references.authors: Han Xiao Kashif Rasul Roland Vollgraf
references.title: Fashion-MNIST: a novel image dataset for benchmarking machine learning
references.authors: Sergey Zagoruyko Nikos Komodakis
references.title: Wide residual networks
references.venue: Proceedings of the British Machine Vision Conference 2016,
references.authors: Ashukha
references.title: 2020b). We use 16 MC samples for predictions. ‘Ensemble’ uses 5 elements for prediction. Ensemble elements differ from each other in their initialisation, which is sampled from the He initialisation distribution
references.venue: (He et al.,
references.authors: Osawa
references.title: CIFAR10 experiments, with the exception that we use a learning rate of 1 × 10−3 as we we found a value of 1 × 10−4 not to result in convergence. We train on a single Nvidia P100 GPU with a batch size of 256. See the authors’ GitHub for more details: github.com/team-approx-bayes/dl-with-bayes/blob/master/ distributed/classification/configs/cifar10/resnet18_vogn_bs256
references.authors: Ritter
references.title: At test time, we tune the prior precision used for ‘Ours’, ‘Diag-Laplace’ and ‘SWAG’ approximation on a validation set for each approach individually
references.venue: Kristiadi et al
references.authors: Antorán
references.title: The 1d toy dataset
references.venue: Dua & Graff
sections.heading: 1 INTRODUCTION
sections.text: Deep neural networks (DNNs) still suffer from critical shortcomings that make them unfit for important applications. For instance, DNNs tend to be poorly calibrated and overconfident in their predictions, especially when there is a shift in the train and test distributions (Nguyen et al., 2015; Guo et al., 2017). To reliably inform decision making, DNNs must be able to robustly quantify the uncertainty in their predictions, which is particularly important in safety-critical areas such as healthcare or autonomous driving (Amodei et al., 2016; Filos et al., 2019a; Fridman et al., 2019).
  Bayesian modeling (Ghahramani, 2015; Gal, 2016) presents a principled way to capture predictive uncertainty via the posterior distribution over model parameters. Unfortunately, due to their nonlinearities, exact posterior inference is intractable in DNNs. Despite recent successes in the field of Bayesian deep learning (Blundell et al., 2015; Gal & Ghahramani, 2016; Osawa et al., 2019; Maddox et al., 2019; Dusenberry et al., 2020), existing methods are only made scalable to modern DNNs with large numbers of parameters by invoking unrealistic assumptions. This severely limits the expressiveness of the inferred posterior and thus deteriorates the quality of the induced uncertainty estimates (Ovadia et al., 2019; Fort et al., 2019; Foong et al., 2019a; Ashukha et al., 2020a).
  Due to the heavy overparameterization of DNNs, their accuracy is well-preserved by a small subnetwork (Cheng et al., 2017). Additionally, recent work by Izmailov et al. (2019) has shown how performing inference over a low dimensional subspace of the weights can result in accurate uncertainty quantification. These observations prompt the following question for a DNN’s uncertainty: Can a full DNN’s model uncertainty be well-preserved by a small subnetwork’s model uncertainty? We answer this question in the affirmative. We show both theoretically and empirically that the full network posterior can be well represented by a subnetwork’s posterior. As a result, we can use more expensive but faithful posterior approximations over just that subnetwork. We show that this achieves better uncertainty quantification than if we use cheaper, but more crude, posterior approximations over the full network.
  The contributions of this paper are as follows:
  1. We propose a new Bayesian deep learning approach that performs Bayesian inference over only a small subset of the model weights and keeps all other weights deterministic. This allows us to use expressive posterior approximations that are typically intractable in DNNs.
  2. As a concrete instantiation of this framework, we develop a practical and scalable Bayesian deep learning method that uses the linearized Laplace approximation to infer a fullcovariance Gaussian posterior over a subnetwork within a point-estimated neural network.
  3. We formally characterize the discrepancy between the posterior distributions over a subnetwork and the full network (in terms of their Wasserstein distance) in the linearized model, and derive a theoretically motivated strategy to select a subnetwork that minimizes this discrepancy under certain assumptions.
  4. We empirically show, on various benchmarks, that our method compares favourably against point-estimated networks and other Bayesian deep learning methods, experimentally confirming that expressive subnetwork inference is superior to crude inference over full networks.
sections.heading: 2 SUBNETWORK POSTERIOR APPROXIMATION
sections.text: Bayesian neural networks (BNNs) aim to capture model uncertainty, i.e., uncertainty about the choice of weightsW which arises due to multiple plausible explanations of the training data {y,X}. Here, y is the dependent variable (e.g. classification label) andX is the feature matrix. A prior distribution p(W) is specified over the BNN’s weights. We wish to infer their full posterior distribution
  p(W|y,X) ∝ p(y|X,W) p(W) . (1) To make predictions, we then estimate the posterior predictive distribution that averages the network’s predictions across all possible settings of the weights, weighted by their posterior probability, i.e.
  p(y∗|X∗,y,X) = ∫ W p(y∗|X∗,W)p(W|y,X)dW . (2)
  Unfortunately, due to the size of modern deep neural networks, it is not only intractable to infer the exact posterior distribution p(W|y,X) in Eq. (1), but it is even computationally challenging to properly approximate it. As a consequence, crude posterior approximations such as complete factorization are commonly employed (Blundell et al., 2015; Hernández-Lobato & Adams, 2015; Kingma et al., 2015; Khan et al., 2018; Osawa et al., 2019), i.e. p(W|y,X) ≈∏Dd=1 q(wd) where wd denotes the d-th weight in the D-dimensional neural network weight vector W ∈ RD (the concatenation and flattening of all layers’ weight matrices). Clearly, this is a very wishful assumption; In practise, it suffers from severe pathologies (Foong et al., 2019a;b).
  In this work, we question the implicit assumption that a good posterior approximation needs to include all BNN parameters. Instead, we aim to perform inference only over a small subset of the weights. This approach is well-motivated for two reasons:
  1. Overparameterization: Maddox et al. (2020) have shown that, in the neighborhood of local optima, there are many directions that leave the NN’s predictions unchanged. Moreover, NNs can be heavily pruned without sacrificing test-set accuracy (Frankle & Carbin, 2019). Thus, the majority of a NN’s predictive power might be isolated to a small subnetwork.
  2. Inference over submodels: Previous work1 has provided evidence that inference can be effective even when not done on the full parameter space. Izmailov et al. (2019) performed inference over a low-dimensional projection of the weights. Neural-linear models, which give a Bayesian treatment to only the last layer of a DNN, have shown to be competitive with full-network approaches (Riquelme et al., 2018; Ober & Rasmussen, 2019).
  We thus combine these ideas, making the following two-step approximation of the posterior in Eq. (1): p(W|y,X) ≈ p(WS |y,X) ∏ r δ(wr − w∗r) ≈ q(WS) ∏ r δ(wr − w∗r) . (3)
  The first approximation decomposes the full neural network posterior p(W|y,X) into a posterior p(WS |y,X) over the subnetwork WS and delta functions δ(wr − w∗r) over all remaining weights {wr}r, keeping them at fixed values w∗r ∈ R. This can be viewed as pruning the variances of the weights {wr}r to zero, which is in contrast to ordinary weight pruning methods that set the weights {wr}r themselves to zero. The second approximation is a result of posterior inference over the subnetwork still being intractable. In turn, we introduce the approximate distribution q(WS). Yet, as the subnetwork is much smaller than the full network, we can afford to make q(WS) expressive and able to capture rich dependencies across the weights within the subnetwork.
sections.heading: 3 SUBNETWORK INFERENCE VIA LAPLACE APPROXIMATION
sections.text: To obtain a method that is as practical as possible, we propose to use inference techniques that can estimate a posterior distribution post-hoc from a point-estimated network. The Laplace approximation (MacKay, 1992) is well-suited to this task as it derives the approximate posterior from the local optimization landscape. Other inference procedures, such as SWAG (Maddox et al., 2019), could also be used. Nevertheless, we focus on Laplace due to it being a well-studied, fundamental technique.
  Step #1: Point Estimation. The first step of the procedure is to train a neural network to obtain a point estimate of the weights, denoted WMAP . This estimate should respect the Bayesian model given in Eq. (1), and therefore we optimize the maximum a-posteriori (MAP) objective:
  WMAP = argmaxW [log p(y|X,W ) + log p(W )] . (4) This can be done using standard stochastic gradient-based optimization methods commonly-used in modern deep learning (Goodfellow et al., 2016). This step is illustrated in Fig. 1 (a).
  Step #2: Subnetwork Selection. The second step is to identify a small subnetworkWS . Ideally, we would like to identify the subnetwork whose posterior is ‘closest’ to the full-network posterior. We formalize this argument in Section 4 and describe a principled strategy that, under certain conditions, minimizes the 2-Wasserstein distance between the sub- and full-network posteriors. All other weights not belonging to that subnetwork are then assigned fixed values: the MAP estimates obtained in Step #1. See Fig. 1 (b) for an illustration of this step.
  Step #3: Bayesian Inference. Given the subnetwork point estimateW SMAP , we use the Laplace approximation to infer a full-covariance Gaussian posterior distribution over the subnetwork WS :
  p(WS |y,X) ≈ q(WS) = N ( WS ;W S MAP , H −1) (5) where the posterior covariance matrix H−1 ∈ RD×D corresponds to the inverse of the average Hessian of the negative log posterior, i.e. H = NE [ −∂2 log p(y|X,W )/∂W 2 ] + λI . Here, the expectation is w.r.t. the data generating distribution and λ is the precision of a zero-mean factorized Gaussian prior p(W) = N (W;0, λ−1I). In practice, we approximate the Hessian H with the generalized Gauss-Newton (GGN) matrix H̃ (Schraudolph, 2002), i.e.
  H̃ = N∑ n=1 J>n HnJn + λI, with Jn = ∂f(xn,W ) ∂W and Hn = ∂2L(yn,f(xn,W )) ∂2f(xn,W ) (6)
  1See Section 6 for a more thorough discussion of related work.
  where Jn ∈ RO×D is the Jacobian of the neural network features f(xn,W ) ∈ RO w.r.t. the weights W , andHn ∈ RO×O is the Hessian of the loss L(yn,f(xn,W )) w.r.t. the features f(xn,W ).The GGN H̃ has clear practical advantages over the Hessian H; see Martens & Sutskever (2011) and Martens (2016). Using the Laplace approximation with the GGN Hessian can be viewed as an implicit local linearization of the underlying neural network f(x,W ) at its MAP estimateWMAP ,
  fMAPlin (x,W ) = f(x,WMAP ) + JWMAP (x)(W −WMAP ) (7) where JWMAP (x) = ∂f(x,WMAP )/∂WMAP ∈ RO×D (Immer et al., 2020). Note that the model in Eq. (7) is linear in W , as only the term JWMAP (x)W depends linearly onW , while the other terms are constant w.r.t.W and can thus be subsumed into an additive bias term (Khan et al., 2019). The GGN approximation thus locally turns the underlying probabilistic model from a Bayesian neural network into a (generalized) linear model, with basis function expansion JWMAP (x) of covariate x (Immer et al., 2020). Put differently, linearized Laplace in the neural network f(x,W ) is equivalent to ordinary Laplace in the linear model fMAPlin (x,W ) in Eq. (7), as the GGN H̃ corresponding to f(x,W ) in Eq. (6) is equivalent to the Hessian H corresponding to fMAPlin (x,W ) in Eq. (7) (Khan et al., 2019). This is a useful property that will allow us to derive a principled subnetwork selection strategy in Section 4. This step is illustrated in Fig. 1 (c). We emphasize that this whole procedure (i.e. Steps #1-#3) is a perfectly valid mixed inference strategy, performing full Laplace inference over the selected subnetwork and MAP inference over all remaining weights.
  Step #4: Prediction. Given the linearized Laplace approximation over the subnetwork WS in Eqs. (5) and (6), i.e. q(WS) = N (WS ;W SMAP , H̃−1), we can then compute the posterior predictive distribution. While, traditionally, one would compute the predictive distribution using the original Bayesian neural network likelihood, i.e. p(y|X,W) = p(y|f(x,W )), Immer et al. (2020) recently suggested that, since inference was (implicitly) done in the GGN-linearized model, it is more principled to instead predict using the linearized likelihood Eq. (7), i.e. p(y|X,W) = p(y|fMAPlin (x,W )). This provides a formal justification for the empirical superiority of this approach observed previously (Lawrence, 2001; Foong et al., 2019b). We thus obtain the linearized predictive distribution
  p(y∗|X∗,y,X) ≈ ∫ W p(y∗|fMAPlin (X∗,W ))N (WS ;W SMAP , H̃−1) ∏ r δ(wr−w∗r) dW . (8)
  There are two ways to compute Eq. (8): Firstly, via a Monte Carlo approximation p(y∗|X∗,y,X) ' 1 M ∑M m=1 p(y ∗|fMAPlin (X∗, W̃m)) by sampling W̃m from N (W SMAP , H̃−1) and ∏
  r δ(wr − w∗r), the latter of which is trivial. Secondly, due to linearity of p(y∗|fMAPlin (X∗,W )), there are closedform expressions which are exact for Gaussian likelihoods (i.e. regression) and approximate for categorical ones (i.e. classification) (Bishop, 2006; Gibbs, 1998). This step is illustrated in Fig. 1 (d).
sections.heading: 4 PRINCIPLED SUBNETWORK SELECTION FOR LINEAR(IZED) MODELS
sections.text: We next analyze the subnetwork inference procedure described in Section 3 for the case of a generalized linear model (GLM) (Nelder & Baker, 1972), which models the expected response yn given the basis function expansion of the covariates φn = φ(xn) as
  E[yn|φn] = g−1(wTφn). (9) Here, w ∈ RD is the vector of model parameters (which subsumes a scalar bias β0 for notational convenience) and g−1(·) denotes a link function such that g−1 : R 7→ µy|φ. In particular, we consider a Bayesian GLM, by specifying a prior distribution p(w) over model parameters and aiming to infer the posterior distribution p(w|y,Φ) ∝ p(y|Φ,w)p(w), where Φ = [φ1, ...φN ]T .
  1. Point Estimation. Obtain the MAP estimate, wMAP = argmaxw log p(y|Φ,w) + log p(w). For commonly-used link functions (e.g. the identity in case of a Gaussian likelihood for regression, or the sigmoid/softmax function in case of a categorical likelihood for classification) and commonly-used priors (e.g. a Gaussian), the log-posterior ∝ log p(y|Φ,w) + log p(w) is concave. This allows for simple gradient-based MAP optimisation. It also makes a full-covariance Gaussian, estimated via Laplace, a faithful approximation to the true, uni-modal posterior, i.e.
  p(w|y,Φ) ≈ p̃(w|y,Φ) = N (w;wMAP , H−1) (10)
  where H is the Hessian defined in Section 3. Note that for the GLM we consider, the Hessian H is equivalent to the GGN H̃ defined in Eq. (6), meaning that an ordinary Laplace approximation is equivalent to a linearized Laplace approximation (Martens, 2016). For the case of an identity link function (i.e. a Gaussian likelihood with noise variance σ20) and a Gaussian prior w ∼ N (0,Λ−10 ), the MAP estimate even has a closed-form expression, wMAP = (ΦTΦ + σ20Λ0)
  −1ΦTy. Here, the Laplace approximation in Eq. (10) exactly corresponds to the true posterior, i.e. p̃(w|y,Φ) = p(w|y,Φ). We will thus refer to the posterior p̃(w|y,Φ) in Eq. (10) as the full posterior.
  2. Subnetwork Selection. Select a subset of S model weights via a method of choice, yielding a binary vector m ∈ RD where md = 1 if the d-th weight is part of the subset, and md = 0 otherwise. For convenience, we define the binary mask matrix MS =mm> ∈ RD×D which contains 1s in the rows/columns corresponding to the S subnetwork weights2, and 0s otherwise.
  3. Bayesian Inference. Compute the posterior over the subnetwork via a Laplace approximation: pS(w|y,Φ) = N (w;wMAP ,MS H−1) . (11)
  Firstly, note that the mean of the subnetwork posterior in Eq. (11) is the MAP estimate wMAP and thus equal to the mean of the full posterior p̃(w|y,Φ) in Eq. (10). Secondly, note that the covariance matrix of the subnetwork posterior in Eq. (11) is the element-wise productMS H−1, which masks the (co-)variances of all weights not belonging to the subnetwork to zero, effectively making them deterministic. More precisely, the subnetwork covariance matrix,MS H−1, is a D ×D matrix that is equal to the full posterior covariance matrix H−1 in the rows/columns of the S weights in the subnetwork, and zero in the rows/columns of all other D − S weights.
  We consider what we term the posterior gap—the Wasserstein distance3 (in particular the squared 2-Wasserstein distance) between the posterior distribution over the full network and the posterior distribution over the subnetwork. The proofs for all results below will be presented in Appendix A. Proposition 1 (Posterior Gap). For a subnetwork of size S < D, the Wasserstein gap between the full posterior p̃(w|y,Φ) in Eq. (10) and the subnetwork posterior pS(w|y,Φ) in Eq. (11) is: W [p̃(w|y,Φ) || pS(w|y,Φ)] = ∑D d=1 (1 +mdd)σ 2 d − trace(2(H−1(MS H−1))1/2) . (12)
  The optimal subnetwork should then minimize the posterior gap in Eq. (12). However, for full covariance matrices H−1 and a large number of weights D, this will generally be infeasible as Eq. (12) depends on all entries of the D ×D-matrix H−1, which is intractable to compute/store. To derive a practical subnetwork selection strategy, we assume the covariance matrix to be diagonal. Corollary 1.1 (Optimality of Maximum Variance Subnetwork Selection under Decorrelation). For a generalized linear model with posterior covariance matrix H−1 = diag(σ21 , . . . , σ 2 D), the optimal subnetwork under the Wasserstein gap is comprised of the S weights with the largest variances σ2d.
  Finally, since a GGN-linearized neural network, as in Eq. (7), corresponds to a GLM with basis expansion φn = JWMAP (xn) = ∂f(xn,WMAP )/∂WMAP (see Step #3 in Section 3), Corollary 1.1 implies that under decorrelation, the optimal subnetwork comprises of the weights with the largest variances. In practice, even just computing the diagonal of the covariance matrix is challenging, so we use a diagonal Laplace approximation which instead computes the inverse of the diagonal of the GGN (see e.g. Ritter et al. (2018)). Finally, note that we only have to make the decorrelation assumption for the purposes of subnetwork selection – when doing posterior inference over the selected subnetwork, we estimate a full covariance matrix for maximal expressiveness, as described in Step #3 in Section 3. In our experiments in Section 5, we empirically show that making the decorrelation assumption for subnetwork selection but then using a full-covariance Gaussian for inference performs significantly better than directly making the decorrelation assumption for inference (e.g. mean-field variational inference, diagonal Laplace).
sections.heading: 5 EMPIRICAL ANALYSIS
sections.text: We empirically assess the effectiveness of subnetwork inference compared to point-estimated NNs and methods that do less expressive inference over the full network. We consider three tasks: 1) small-scale toy regression, 2) medium-scale tabular regression, and 3) large-scale image classification.
  2For consistency, we will keep referring to the S selected linear model weights as a ”subnetwork”. 3We use the Wasserstein distance instead of the more common Kullback–Leibler divergence because the
  Wasserstein is well-defined for degenerate distributions and is an actual distance metric (i.e. symmetric).
sections.heading: 5.1 HOW DOES SUBNETWORK INFERENCE RETAIN POSTERIOR PREDICTIVE UNCERTAINTY?
sections.text: We first assess how the predictive distribution of a full-covariance Gaussian posterior over a selected subnetwork qualitatively compares to that obtained from 1) a full-covariance Gaussian over the full network (Full Cov), 2) a factorised Gaussian posterior over the full network (Diag), 3) a fullcovariance Gaussian over only the (Final layer) of the network (Kristiadi et al., 2020), and 4) a point estimate (MAP). For subnetwork inference, we consider both Wasserstein (Wass) (as described in Section 4) and uniform random subnetwork selection (Rand) to obtain subnetworks that comprise of only 50%, 3% and 1% of the model parameters. Note that while for this toy example, we could in principle use the full covariance matrix for the purpose of subnetwork selection, we still just use its diagonal (as described in Section 4) for consistency. Our NN consists of 2 ReLU hidden layers with 50 hidden units each. We employ a homoscedastic Gaussian likelihood function where the noise variance is optimised with maximum likelihood. We use GGN Laplace inference over network weights (not biases) in combination with the linearized predictive distribution in Eq. (8). Thus, all approaches considered share their predictive mean, allowing us to better compare their uncertainty estimates. All approaches share a single prior precision of λ = 3.
  We use a synthetic 1D regression task with two separated clusters of inputs (Antorán et al., 2020), allowing us to probe for ‘in-between’ uncertainty (Foong et al., 2019b). Results are shown in Fig. 2. Subnetwork inference preserves more of the uncertainty of full network inference than diagonal Gaussian or final layer inference while doing inference over fewer weights. By capturing weight correlations, subnetwork inference retains uncertainty in between clusters of data. This is true for both random and Wasserstein subnetwork selection. However, the latter preserves more uncertainty with smaller subnetworks. Finally, the strong superiority to diagonal Laplace shows that making a diagonal assumption for subnetwork selection but then using a full-covariance Gaussian for inference (as we do) performs much signficantly better than making a diagonal assumption for the inferered posterior directly. These results suggest that expressive inference over a carefully selected subnetwork retains more predictive uncertainty than crude approximations over the full network.
sections.heading: 5.2 SUBNETWORK INFERENCE IN LARGE MODELS VS FULL INFERENCE OVER SMALL MODELS
sections.text: Secondly, we study the following natural question: “Why should one use subnetwork inference in a large model when one can just perform full network inference over a smaller model?” We explore this by considering 4 fully connected models of increasing size. These have numbers of hidden layers hd={1, 2} and hidden layer widths wd={50, 100}. For a dataset with input dimension id, the number of weights is given by D=(id+1)wd+(hd−1)w2d. Our 2 hidden layer, 100 hidden unit models have a weight count of the order 104. Full covariance inference in these models borders the limit of computational tractability on commercial hardware. We first obtain a MAP estimate of each
  model’s weights and our homoscedastic likelihood function’s noise variance. We then perform full network GGN Laplace inference for each model. We also use our proposed Wassertein rule to prune every network’s weight variances such that the number of variances that remain matches the size of every smaller network under consideration. In all cases, we employ the linearized predictive in Eq. (7). Consequently, networks with the same number of weights make the same mean predictions. Increasing the number of weight variances considered will thus only increase predictive uncertainty.
  We employ 3 tabular datasets of increasing size (input dimensionality, n. points): wine (11, 1439), kin8nm (8, 7373) and protein (9, 41157). We consider their standard train-test splits (HernándezLobato & Adams, 2015) and their gap variants (Foong et al., 2019b), designed to test for out-ofdistribution uncertainty. Details are provided in Appendix C.4. For each split, we set aside 15% of the train data as a validation set. We use these for early stopping when finding MAP estimates and for selecting the weights’ prior precision. We keep other hyperparameters fixed across all models and datasets. Results are in Fig. 3.
  We present mean test log-likelihood (LL) values, as these take into account both accuracy and uncertainty. Larger models tend to perform better when doing MAP inference, with wine-gap and protein-gap being exceptions. We also find larger models improve over their respective MAP LLs more than small ones when performing approximate inference over the same numbers of weights. We conjecture this is due to an abundance of degenerate directions (weights) in the weight posterior of all models (Maddox et al., 2020). Full network inference in small models captures information about both useful and non-useful weights. In larger models, our subnetwork selection strategy allows us to dedicate a larger proportion of our resources to modelling informative weight variances and covariances. In 3 out of 6 datasets, we find abrupt increases in LL as we increase the number of weights over which we perform inference, followed by a plateau. Such plateaus might be explained by all of the most informative weight variances having already been accounted for. These results suggest that, given the same amount of compute, larger models benefit more from subnetwork inference than small ones.
sections.heading: 5.3 SCALING TO IMAGE CLASSIFICATION WITH DISTRIBUTION SHIFT
sections.text: We now assess the robustness of large convolutional neural networks with subnetwork inference to distribution shift on image classification tasks compared to the following baselines: point-estimated networks (MAP), Bayesian deep learning methods that do less expressive inference over the full network: MC Dropout (Gal & Ghahramani, 2016), diagonal Laplace, VOGN (Osawa et al., 2019) (all of which assume factorisation of the weight posterior), and SWAG (Maddox et al., 2019) (which assumes a diagonal plus low-rank posterior). We also benchmark deep ensembles (Lakshminarayanan et al., 2017). The latter is considered state-of-the-art for uncertainty quantification in deep learning (Ovadia et al., 2019; Ashukha et al., 2020a). We use ensembles of 5 DNNs, as suggested by (Ovadia et al., 2019), and 16 samples for MC Dropout, diagonal Laplace and SWAG. We use a Dropout probability of 0.1 and a prior precision of λ = 40, 000 for diagonal Laplace, found via grid search. We apply all approaches to ResNet-18 (He et al., 2016), which is composed of an input convolutional block, 8 residual blocks and a linear layer, for a total of 11,168,000 parameters. For subnetwork inference, we compute the linearized predictive distribution in Eq. (8) via the closed-form
  approximation for integrals between Gaussians and multi-class cross-entropy likelihoods described in (Gibbs, 1998). We use Wasserstein subnetwork selection to retain only 0.38% of the weights, yielding a subnetwork with only 42,438 weights. This is the largest subnetwork for which we can tractably compute a full covariance matrix. Its size is 42, 4382 × 4 Bytes ≈ 7.2 GB. We use a prior precision of λ = 500, found via grid search. Finally, to assess to importance of principled subnetwork selection, we also consider the baseline where we select the subnetwork uniformly at random (called Ours (Rand)). We perform the following two experiments, with results in Fig. 4. See Appendix B for additional results.
  Rotated MNIST: Following Ovadia et al. (2019); Antorán et al. (2020), we train all methods on MNIST and evaluate their predictive distributions on increasingly rotated digits. While all methods perform well on the original MNIST test set, their accuracy degrades quickly for rotations larger than 30 degrees. In terms of LL, ensembles perform best out of our baselines. Subnetwork inference obtains significantly larger LL values than almost all baselines, including ensembles. The only exception is VOGN, which achieves slightly better performance. It was also observed in Ovadia et al. (2019) that mean-field variational inference (which VOGN also is an instance of) is very strong on MNIST, but its performance deteriorates on larger datasets. Subnetwork inference makes accurate predictions in-distribution while assigning higher uncertainty than the baselines to out-of-distribution points. Corrupted CIFAR: Again following Ovadia et al. (2019); Antorán et al. (2020), we train on CIFAR10 and evaluate on data subject to 16 different corruptions with 5 levels of intensity each (Hendrycks & Dietterich, 2019). Our approach matches a MAP estimated network in terms of predictive error as local linearization makes their predictions the same. Ensembles and SWAG are the most accurate. Even so, subnetwork inference differentiates itself by being the least overconfident, outperforming all baselines in terms of log-likelihood at all corruption levels. Here, VOGN performs rather badly; while this might appear in stark contrast to its strong performance on the MNIST benchmark, the behaviour that mean-field VI performs well on MNIST but not on larger datasets was also observed in Ovadia et al. (2019).
  On both benchmarks, we furthermore find that randomly selecting the subnetwork performs substantially worse than using our more principled subnetwork selection strategy. This highlights the importance of the way subnetworks are selected. These results suggest that subnetwork inference results in better uncertainty calibration and robustness to distribution shift than other popular uncertainty quantification approaches.
sections.heading: 6 RELATED WORK
sections.text: Bayesian Deep Learning. There have significant efforts to characterise the posterior distribution over NN weights p(W |D). Hamiltonian Monte Carlo (Neal, 1995) remains the golden standard
  for approximate inference in BNNs to this day. Although asymptotically unbiased, sampling based approaches are difficult to scale to the large datasets (Betancourt, 2015). As a result, approaches which find the best surrogate posterior among an approximating family (most often Gaussians) have gained popularity. The first of these was the Laplace approximation, introduced by MacKay (1992), who also proposed approximating the predictive posterior with that of the linearised model (Khan et al., 2019; Immer et al., 2020). The popularisation of larger NN models has made surrogate distributions that capture correlations between weights computationally intractable. Thus, most modern methods make use of the mean field assumption (Blundell et al., 2015; Hernández-Lobato & Adams, 2015; Gal & Ghahramani, 2016; Mishkin et al., 2018; Osawa et al., 2019). This comes at the cost of limited expressivity (Foong et al., 2019a) and empirical under-performance (Ovadia et al., 2019; Antorán et al., 2020) of uncertainty estimates. Our proposed approach recovers predictive posterior expressivity while maintaining tractability by lowering the dimensionality of the weight space considered. This allows us to scale up approximations that do consider weight correlations (MacKay, 1992; Louizos & Welling, 2016; Maddox et al., 2019; Ritter et al., 2018).
  Neural Network Linearization. In the limit of infinite width, NNs converge to Gaussian process (GP) behaviour (Neal, 1995; Matthews, 2017; Garriga-Alonso et al., 2018). Recently, these results have been extended to finite width BNNs when the surrogate posterior is Gaussian (Khan et al., 2019). We draw upon these results to formulate a subnetwork selection strategy for BNNs. Neural linear methods perform inference over only the last layer of a NN, while keeping all other layers fixed (Snoek et al., 2015; Riquelme et al., 2018; Ovadia et al., 2019; Ober & Rasmussen, 2019; Pinsler et al., 2019; Kristiadi et al., 2020). These represent a different generalised linear model in which the basis functions are defined by the l−1 first layers of a NN. They can also be viewed as a special case of subnetwork inference, in which the subnetwork is simply defined to be the last NN layer.
  Inference over Subspaces. The subfield of NN pruning aims to increase the computational efficiency of NNs by identifying the smallest subset of weights which are required to make accurate predictions. Approaches trade-off computational cost with compression efficiency, ranging from those that require multiple training runs (Frankle & Carbin, 2019) to those that prune before training (Wang et al., 2020). Our work differs in that it retains all NN weights but aims to find a small subset over which to perform probabilistic reasoning. More closely related work to ours is that of Izmailov et al. (2019), who propose to perform inference over a low-dimensional subspace of weights; e.g. one constructed from the principal components of the SGD trajectory. Moreover, several recent approaches use low-rank parameterizations of approximate posteriors in the context of variational inference (Rossi et al., 2019; Swiatkowski et al., 2020; Dusenberry et al., 2020). This could also be viewed as doing inference over an implicit subspace of weight space. In contrast, we propose a technique to find subsets of weights which are relevant to predictive uncertainty, i.e., we identify axis aligned subspaces. Finally, there have been recent works studying neural network sparsity / pruning from a Bayesian perspective (Ghosh & Doshi-Velez, 2017; Polson & Ročková, 2018; Cui et al., 2020; Louizos et al., 2017; Molchanov et al., 2017; Gomez et al., 2019; Lee et al., 2018). While these seem conceptually related at first glance, their goal is fundamentally different to ours: While those methods aim to perform model selection / sparsification by either explicitly or implicitly pruning unnecessary weights, our goal is to make inference more tractable. More precisely, while those sparse Bayesian deep learning methods prune individual weights, we instead just prune the variances of certain weights, which, importantly retains the full predictive power of the full network to retain high predictive accuracy.
sections.heading: 7 CONCLUSION
sections.text: In this paper, we develop a practical and scalable method for expressive yet tractable probabilistic inference in deep neural networks. We approximate the posterior over a subset of the weights while keeping all other weights deterministic. Computational cost is decoupled from network size, allowing us to scale expressive approximations, such as full-covariance Gaussian distributions, to real-world sized NNs. Our approach can be applied post-hoc to any pre-trained model, making it particularly attractive for practical use. Our empirical analysis suggests that subnetwork inference 1) is more expressive and retains more uncertainty than crude approximations over the full network, 2) allows us to employ larger NNs, which fit a broader range of functions, without sacrificing the quality of our uncertainty estimates, and 3) is competitive with state-of-the-art uncertainty quantification methods, like deep ensembles (Lakshminarayanan et al., 2017), on real-world scale problems.
sections.heading: A PROOFS FOR THE THEORETICAL RESULTS
sections.text: We now provide the proofs for the results in Section 4.
sections.heading: A.1 PROOF OF PROPOSITION 1
sections.text: Proof. Note that the posterior distributions p̃(w|y,Φ) and pS(w|y,Φ) are both Gaussian. We thus consider the squared 2-Wasserstein distance between two Gaussian distributions N (µ1,Σ1) and N (µ2,Σ2), which has the following closed-form expression (Givens et al., 1984)4:
  W [N (µ1,Σ1) || N (µ2,Σ2)] = ‖µ1 − µ2‖22 + trace ( Σ1 + Σ2 − 2 (Σ1Σ2)1/2 ) . (13)
  Plugging in µ1 = µ2 = wMAP , Σ1 = H−1 and Σ2 =MS H−1, we obtain W [p̃(w|y,X) || pS(w|y,X)]
  =W [ N (wMAP , H−1) || N (wMAP ,MS H−1) ] =((( (((( ((‖wMAP −wMAP ‖22 + trace ( H−1 + (MS H−1)− 2 ( H−1(MS H−1)
  )1/2) = trace ( (1 +MS) H−1 ) − trace ( 2 ( H−1 ( MS H−1
  ))1/2) =
  D∑ d=1 (1 +mdd)σ 2 d − trace ( 2 ( H−1 ( MS H−1 ))1/2)
sections.heading: A.2 PROOF OF COROLLARY 1.1
sections.text: Proof. For H−1 = diag(σ21 , . . . , σ 2 D), the Wasserstein posterior gap in Eq. (12) simplifies to
  W [p̃(w|y,Φ) || pS(w|y,Φ)] = D∑
  d=1
  ( (1 +mdd)σ 2 d − 2mddσ2d ) . (14)
  The optimal subnetwork selection strategy amounts to choosing the binary vectorm = [mdd]Dd=1 with∑D d=1md = S (i.e., we select S out of D parameters) s.t. the posterior gap in Eq. (14) is minimized. Observing that the contribution of the d-th parameter to the posterior gap is (1+ 1)σ2d − 1× 2σ2d = 0 if it is selected (i.e. if mdd = 1), and (1 + 0)σ2d − 0× 2σ2d = σ2d if it is not selected (i.e. if mdd = 0), we see that the optimal subnetwork comprises of the S weights with the largest variances σ2d.
sections.heading: B ADDITIONAL IMAGE CLASSIFICATION RESULTS
sections.heading: C EXPERIMENTAL SETUP
sections.heading: C.1 TOY EXPERIMENTS
sections.text: We train a single, 2 hidden layer network, with 50 hidden ReLU units per layer using MAP inference until convergence. Specifically, we use SGD with a learning rate of 1× 10−3, momentum of 0.9 and weight decay of 1× 10−4. We use a batch size of 512. The objective we optimise is the Gaussian log-likelihood of our data, where the mean is outputted by the network and the the variance is a hyperparameter learnt jointly with NN parameters by SGD. This variance parameters is shared among all datapoints. Once the network is trained, we perform post-hoc inference on it using different approaches. Since all of these involve the linearized approximation, the mean prediction is the same for all methods. Only their uncertainty estimates vary.
  Note that while for this toy example, we could in principle use the full covariance matrix for the purpose of subnetwork selection, we still just use its diagonal (as described in Section 4) for consistency. We use GGN Laplace inference over network weights (not biases) in combination with the linearized predictive distribution in Eq. (8). Thus, all approaches considered share their predictive mean, allowing us to better compare their uncertainty estimates.
  All approaches share a single prior precision of λ = 3. We chose to select the prior precision such that the full covariance approach (optimistic baseline) presents reasonable results. We use the same value for all other methods. We first tried a precision of 1 and found the full covariance approach to produce excessively large errorbars (covering the whole plot). A value of 3 produces more reasonable results.
  Final layer inference is performed by computing the full Laplace covariance matrix and discarding all entries except those corresponding to the final layer of the NN. Results for random sub-network selection are obtained with a single sample from a scaled uniform distribution over weight choice.
sections.heading: C.2 UCI EXPERIMENTS
sections.text: In this experiment, our fully connected NNs have numbers of hidden layers hd={1, 2} and hidden layer widths wd={50, 100}. For a dataset with input dimension id, the number of weights is given by D=(id+1)wd+(hd−1)w2d. Our 2 hidden layer, 100 hidden unit models have a weight count of the order 104. The non-linearity used is ReLU.
  We first obtain a MAP estimate of each model’s weights. Specifically, we use SGD with a learning rate of 1× 10−3, momentum of 0.9 and weight decay of 1× 10−4. We use a batch size of 512. The objective we optimise is the Gaussian log-likelihood of our data, where the mean is outputted by the network and the the variance is a hyperparameter learnt jointly with NN parameters by SGD.
  For each dataset split, we set aside 15% of the train data as a validation set. We use these for early stopping training. Training runs for a maximum of 2000 epochs but early stops with a patience of 500 if validation performance does not increase. For the larger Protein dataset, these values are 500 and 125. The weight settings which provide best validation performance are kept.
  We then perform full network GGN Laplace inference for each model. We also use our proposed Wassertein rule together with the diagonal Hessian assumption to prune every network’s weight variances such that the number of variances that remain matches the size of every smaller network under consideration. The prior precision used for these steps is chosen such that the resulting predictor’s logliklihood performance on the validation set is maximised. Specifically, we employ a grid search over the values: λ : [0.0001, 0.001, 0.1, 0.5, 1, 2, 5, 10, 100, 1000]. In all cases, we employ the linearized predictive in Eq. (7). Consequently, networks with the same number of weights make the same mean predictions. Increasing the number of weight variances considered will thus only increase predictive uncertainty.
  C.3 IMAGE EXPERIMENTS
  The results shown in Section 5.3 and Appendix B are obtained by training ResNet-18 (and ResNet-50) models using SGD with momentum. For each experiment repetition, we train 7 different models: The first is for: ‘MAP’, ‘Ours’, ‘Ours (Rand)’, ‘SWAG’, ‘Diag-Laplace’ and as the first element of
  ‘Ensemble’. We train 4 additional ‘Ensemble’ elements, 1 network with ‘Dropout’, and, finally 1 network for ‘VOGN’. The methods ‘Ours’, ‘Ours (Rand)’, ‘SWAG’, and ‘Diag-Laplace’ are applied post training.
  For all methods except ‘VOGN’ we use the following training procedure. The (initial) learning rate, momentum, and weight decay are 0.1, 0.9, and 1× 10−4, respectively. For ‘MAP’ we use 4 Nvidia P100 GPUs with a total batch size of 2048. For the calculation of the Jacobian in the subnetwork selection phase we use a single P100 GPU with a batch size of 4. For the calculation of the hessian we use a single P100 GPU with a batch size of 2. We train on 1 Nvidia P100 GPU with a batch size of 256 for all other methods. Each dataset is trained for a different number of epochs, shown in Table 21. We decay the learning rate by a factor of 10 at scheduled epochs, also shown in Table 21. Otherwise, all methods and datasets share hyperparameters. These hyperparameter settings are the defaults provided by PyTorch for training on ImageNet. We found them to perform well across the board. We report results obtained at the final training epoch. We do not use a separate validation set to determine the best epoch as we found ResNet-18 and ResNet-50 to not overfit with the chosen schedules.
  For ‘Dropout’, we add dropout to the standard ResNet-50 model (He et al., 2016) in between the 2 and 3 convolutions in the bottleneck blocks. This approach follows Zagoruyko & Komodakis (2016) and Ashukha et al. (2020b) who add dropout in-between the two convolutions of a WideResNet-50’s basic block. Following Antorán et al. (2020), we choose a dropout probability of 0.1, as they found it to perform better than the value of 0.3 suggested by Ashukha et al. (2020b). We use 16 MC samples for predictions. ‘Ensemble’ uses 5 elements for prediction. Ensemble elements differ from each other in their initialisation, which is sampled from the He initialisation distribution (He et al., 2015). We do not use adversarial training as, inline with Ashukha et al. (2020b), we do not find it to improve results. For ‘VOGN’ we use the same procedure and hyper-parameters as used by Osawa et al. (2019) in their CIFAR10 experiments, with the exception that we use a learning rate of 1 × 10−3 as we we found a value of 1 × 10−4 not to result in convergence. We train on a single Nvidia P100 GPU with a batch size of 256. See the authors’ GitHub for more details: github.com/team-approx-bayes/dl-with-bayes/blob/master/ distributed/classification/configs/cifar10/resnet18_vogn_bs256_ 8gpu.json.
  We modify the standard ResNet-50 and ResNet-18 architectures such that the first 7× 7 convolution is replaced with a 3× 3 convolution. Additionally, we remove the first max-pooling layer. Following Goyal et al. (2017), we zero-initialise the last batch normalisation layer in residual blocks so that they act as identity functions at the start of training.
  At test time, we tune the prior precision used for ‘Ours’, ‘Diag-Laplace’ and ‘SWAG’ approximation on a validation set for each approach individually, as in Ritter et al. (2018); Kristiadi et al. (2020). We use a grid search from 1× 10−4 to 1× 104 in logarithmic steps, and then a second, finer-grained grid search between the two best performing values (again with logarithmic steps).
sections.heading: C.4 DATASETS
sections.text: The 1d toy dataset used in Section 5.1 was taken from Antorán et al. (2020). We obtained it from the authors’ github repo: https://github.com/cambridge-mlg/DUN. Table 22 summarises the datasets used in Section 5.2. Wine and Protein are available from the UCI dataset repository Dua & Graff (2017). Kin8nm is available from https://www.openml.org/d/189 Foong et al. (2019b). For the standard splits (Hernández-Lobato & Adams, 2015) 90% of the data is used for training and 10% for validation. For the gap splits (Foong et al., 2019b) a split is obtained per input dimension by ordering points by their values across that dimension and removing the middle 33% of
  the points. These are used for validation. The datasets used for our image experiments are outlined in Table 23.
sections.heading: NAME SIZE INPUT DIM. NO. CLASSES NO. SPLITS