text:
\section{Introduction}

Over the last 25 years, there have been several strong arguments favouring
a Bayesian approach to deep learning \citep[e.g., ][]{mackay1995probable, neal1996, blundell2015weight, gal2016uncertainty, wilson2020bayesian}.
Bayesian inference for neural networks promises improved predictions, reliable uncertainty estimates, and principled model comparison, naturally supporting active learning, continual learning, and decision-making under uncertainty.
The Bayesian deep learning community has designed multiple successful practical methods inspired by the Bayesian approach \citep{blundell2015weight, gal2016dropout, welling2011bayesian, kirkpatrick2017overcoming, maddoxfast2019, izmailov2019subspace, daxberger2020expressive} with applications ranging from astrophysics \citep{Cranmer2021ABN} to automatic diagnosis of Diabetic Retinopathy \citep{filos2019systematic}, click-through rate prediction in advertising \citep{liu2017pbodl} and modeling of fluid dynamics \citep{geneva2020modeling}.

However, inference with modern neural networks is distinctly challenging. We wish to compute a Bayesian model average corresponding to an integral over a multi-million dimensional multi-modal posterior, with unusual topological properties like mode-connectivity \citep{garipov2018, draxler2018essentially}, under severe computational constraints.

There are therefore many unresolved questions about Bayesian deep learning practice. Variational procedures typically provide unimodal Gaussian approximations to the multimodal posterior. Practically successful methods such as deep ensembles \citep{lakshminarayanan2017simple, fort2019deep} have a natural Bayesian interpretation \citep{wilson2020bayesian}, but only represent modes of the posterior. While Stochastic MCMC \citep{welling2011bayesian, chen2014stochastic, zhang2019cyclical} is computationally convenient, it could be providing heavily biased
estimates of posterior expectations. Moreover, \citet{wenzel2020good} question the quality of standard Bayes posteriors, citing results where ``cold posteriors'', raised to a power $1/T$ with $T<1$, improve performance.

Additionally, Bayesian deep learning methods are typically evaluated on their ability to generate useful, well-calibrated predictions on held-out or out-of-distribution data.
However, strong performance on benchmark problems does not imply that the algorithm accurately approximates the true Bayesian model average (BMA).

In this paper, we investigate fundamental open questions in Bayesian deep learning, using multi-chain full-batch Hamiltonian Monte Carlo
\citep[HMC,][]{neal2011mcmc}.
HMC is a highly-efficient and well-studied Markov Chain Monte Carlo (MCMC) method that is guaranteed to asymptotically produce samples from the true posterior.
However it is enormously challenging to apply HMC to modern neural networks due to its extreme computational requirements: HMC can take \textit{tens of thousands of training epochs} to produce a single sample from the posterior.
To address this computational challenge, we parallelize the computation over hundreds of Tensor processing unit (TPU) devices.

We argue that full-batch HMC provides the most precise tool for studying the BNN posterior to date. We are not proposing HMC
as a computationally efficient method for practical applications. Rather, using our implementation of HMC
we are able to explore fundamental questions about posterior geometry, the performance of BNNs, approximate inference, effect of priors and posterior temperature.

In particular, we show:
(1) BNNs can achieve significant performance gains over standard training and deep ensembles;
(2) a single long HMC chain can provide a comparable representation of the posterior to multiple shorter chains;
(3) in contrast to recent studies, we find posterior tempering is not needed for near-optimal performance, with little evidence for a ``cold posterior'' effect, which we show is largely an artifact of data augmentation;
(4) BMA performance is robust to the choice of prior scale, and relatively similar for diagonal Gaussian, mixture of Gaussian, and logistic priors over weights. This result highlights the importance of architecture relative to parameter priors in specifying the prior over functions.
(5) While Bayesian neural networks have good performance for OOD detection, they show surprisingly poor generalization under domain shift;
(6) while cheaper alternatives such as deep ensembles and SGMCMC can provide good generalization, they provide distinct predictive
distributions from HMC. Notably, deep ensemble predictive distributions are similarly close to HMC as standard SGLD, and closer than standard
variational inference.

We additionally show how to effectively deploy full batch HMC on modern neural networks, including insights about how to tune crucial hyperparameters for good performance, and parallelize sampling over hundreds of TPUs.
Our HMC samples and implementation will be a \href{https://github.com/google-research/google-research/tree/master/bnn_hmc}{\underline{public resource}}.
We hope this resource will both serve as a reference in evaluating and calibrating more practical alternatives to HMC, and aid in exploring questions for a better understanding of approximate inference in Bayesian deep learning.

\section{Background}
\label{sec:background}

\textbf{Bayesian neural networks}\quad
The goal of classical learning is to find a single best setting of the parameters for the model, typically
through maximum-likelihood optimization.
In the Bayesian framework, the learner instead infers a \textit{posterior} distribution $p(w \vert \mathcal{D})$ over the
parameters $w$ of the model after observing the data $\mathcal{D}$.
The posterior distribution is given by Bayes' rule:
$p(w \vert \mathcal{D}) \propto p(\mathcal{D} \vert w) p(w)$, where $p(\mathcal{D} \vert w)$ is the likelihood of $\mathcal{D}$ given by the model with parameters $w$,
and $p(w)$ is the prior distribution over the parameters.
The predictions of the model on a new test example $x$ are then given by the \textit{Bayesian model average} (BMA)
\begin{equation}
    \label{eq:bma}
    \textstyle
    p(y \vert x, \mathcal{D}) = \int_w p(y \vert x, w) p(w \vert \mathcal{D}) dw,
\end{equation}
where $p(y \vert x, w)$ is the predictive distribution for a given value of the parameters $w$.
This BMA is particularly compelling in Bayesian deep learning, because the posterior over
parameters for a modern neural network can represent many complementary solutions to a
given problem, corresponding to different settings of parameters \citep{wilson2020bayesian}.
Unfortunately, the integral in Eq.~\eqref{eq:bma} cannot be evaluated in closed form for
neural networks, so one must resort to approximate inference. Moreover, approximating
Eq.~\eqref{eq:bma} is challenging due to a high dimensional and sophisticated posterior
$p(w \vert \mathcal{D})$. For a detailed discussion of Bayesian deep learning, see e.g. \citet{wilson2020bayesian}.

\textbf{Markov Chain Monte Carlo.}\quad
The integral in Eq. \eqref{eq:bma} can be approximated by sampling:
$p(y \vert x, \mathcal{D}) \approx \frac 1 M \sum_{i=1}^M p(y \vert x, w_i)$,
where $w_i \sim p(w \vert \mathcal{D})$ are samples drawn from the posterior.
MCMC methods construct a Markov chain that, if simulated for long enough, generates approximate samples from the posterior.
In this work, we focus on Hamiltonian Monte Carlo \citep{neal2011mcmc}, a method that produces
asymptotically exact samples  assuming access to the unnormalized
posterior density $p(\mathcal{D} \vert w) p(w)$ and its gradient.

\section{Related work}
\label{sec:rel_work}

The bulk of work on Bayesian deep learning has focused on scalable approximate inference methods.
These methods include stochastic variational inference
\citep{hoffman2013stochastic, graves2011practical, blundell2015weight, kingma2015variational, molchanov2017variational, louizos2017multiplicative, khan2018fast, zhang2018noisy, wu2018deterministic,osawa2019practical,dusenberry2020efficient},
dropout \citep{srivastava2014dropout,gal2016dropout,kendall2017uncertainties, gal2017concrete},
the Laplace approximation \citep{mackay1992practical, kirkpatrick2017overcoming, ritter2018scalable, li00, daxberger2020expressive},
expectation propagation \citep{hernandez2015probabilistic},
and leveraging the stochastic gradient descent (SGD) trajectory, either for a deterministic approximation,
or sampling as in SGLD
\citep{mandt2017stochastic,maddoxfast2019,izmailov2018, wilson2020bayesian}. \citet{foong2019expressiveness} and \citet{farquhar2020liberty}
additionally consider the role of expressive posterior approximations in variational inference.

While these (and many other) methods often provide improved predictions or uncertainty estimates,
to the best of our knowledge \textit{none} of these methods have been directly evaluated on their
ability to match the true posterior distribution using practical architectures and datasets.
Moreover, many of these methods are often designed with train-time constraints in mind, to take roughly the same
amount of compute as regular SGD training.
To evaluate approximate inference procedures, and explore fundamental questions in Bayesian deep learning,
we attempt to construct a posterior approximation of the highest possible quality,
ignoring the practicality of the method.

The Monte Carlo literature for Bayesian neural networks has mainly focused on stochastic gradient-based
methods \citep{welling2011bayesian,ahn2014distributed,chen2014stochastic,ma2015complete,ahn2012bayesian,ding2014bayesian, zhang2019cyclical, garriga2021exact}
for computational efficiency reasons.
These methods are fundamentally biased: (1) they omit the Metropolis-Hastings correction step,
and (2) the noise from subsampling the data perturbs their stationary distribution.
In particular, \citet{betancourt2015fundamental} argues that HMC is incompatible with data subsampling.
Notably, \citet{zhang2020amagold} recently proposed a second-order stochastic gradient MCMC method that
is asymptotically exact.

Since the classic work of \citet{neal1996}, there have been a few recent attempts at using full-batch HMC in BNNs \citep[e.g.; ][]{cobb2020scaling,wenzel2020good}. These studies tend to use relatively short trajectory lengths (generally not considering a number of leapfrog steps greater than 100), and tend to focus on relatively small datasets and networks.
We on the other hand experiment with practical architectures and datasets and use up to $10^5$ leapfrog steps per iteration to ensure good mixing.

Our work is aimed at \textit{understanding} the properties of true Bayesian neural networks.
In a similar direction, \citet{wenzel2020good} have recently explored the effect of the posterior temperature in Bayesian
neural networks.
We discuss their results in detail in \autoref{sec:cold_posteriors}, and
provide our own exploration of the posterior temperature with a different result:
we find that BNNs achieve strong performance at temperature $1$ and do not require
posterior tempering. Moreover, the scope of our paper extends well beyond temperature scaling,
revealing for instance that while BNNs can provide strong in-domain generalization, they surprisingly
suffer on the covariate shift problems that approximate inference methods are often applied towards.
We also show that while deep ensembles are often treated as a non-Bayesian alternative, they in fact
are providing higher fidelity approximations of the Bayesian model average than standard approximate inference
procedures, as argued in \citet{wilson2020bayesian}. We also explore several other key questions, including
prior selection and posterior geometry.

\begin{figure*}[h]
    \centering

    \hspace{-.8cm}
    \label{fig:trajectory-length-ablation}
    \begin{tabular}{cc}
        \begin{tabular}{cccc}
        	\hspace{-.35cm}
            \rotatebox{90}{\scriptsize \quad\quad \textbf{CIFAR-10}} &
            \hspace{-.4cm}
            \includegraphics[height=0.105\textwidth]{figures/ablations/trajlen_acc.pdf} &
            \hspace{-.4cm}
            \includegraphics[height=0.105\textwidth]{figures/ablations/trajlen_ll.pdf} &
            \hspace{-.6cm}
            \includegraphics[height=0.105\textwidth]{figures/ablations/trajlen_ece.pdf}
            \\
            \hspace{-.2cm}
            \rotatebox{90}{\scriptsize \quad\quad\quad \textbf{IMDB}} &
            \hspace{-.4cm}
            \includegraphics[height=0.105\textwidth]{figures/ablations/trajlen_lstm_acc.pdf} &
            \hspace{-.4cm}
            \includegraphics[height=0.105\textwidth]{figures/ablations/trajlen_lstm_ll.pdf} &
            \hspace{-.6cm}
            \includegraphics[height=0.105\textwidth]{figures/ablations/trajlen_lstm_ece.pdf}
        \end{tabular}
        &
        \hspace{-.35cm}
        \label{fig:num-chains-ablation}
        \begin{tabular}{cccc}
            \includegraphics[height=0.105\textwidth]{figures/ablations/numchains_acc.pdf} &
            \hspace{-.4cm}
            \includegraphics[height=0.105\textwidth]{figures/ablations/numchains_ll.pdf} &
            \hspace{-.6cm}
            \includegraphics[height=0.105\textwidth]{figures/ablations/numchains_ece.pdf}
            \\
            \includegraphics[height=0.105\textwidth]{figures/ablations/numchains_lstm_acc.pdf} &
            \hspace{-.4cm}
            \includegraphics[height=0.105\textwidth]{figures/ablations/numchains_lstm_ll.pdf} &
            \hspace{-.6cm}
            \includegraphics[height=0.105\textwidth]{figures/ablations/numchains_lstm_ece.pdf}
        \end{tabular}
        \\
        \multicolumn{2}{c}{\includegraphics[height=0.09\textwidth]{figures/ablations/trajlen_legend.pdf}}\\[-0.7cm]
        (a) Trajectory length ablation & (b) Number of chains ablation
    \end{tabular}

	\caption{
	    \textbf{Effect of HMC hyper-parameters.}
	    BMA accuracy, log-likelihood and expected calibration error (ECE)
        as a function of \textbf{(a)}: the trajectory length $\tau$ and \textbf{(b)}: number of HMC chains.
        The orange curve shows the results for a fixed number of samples in (a) and for a fixed number
        of samples per chain in (b); the brown curve shows the results for a fixed amount of compute.
        All experiments are done on CIFAR-10 using the ResNet-20-FRN architecture on IMDB using CNN-LSTM.
	    Longer trajectory lengths decrease correlation between subsequent samples improving accuracy and
	    log-likelihood.
	    For a given amount of computation, increasing the number of chains from one to two modestly improves the accuracy and log-likelihood.
	}
	\label{fig:tau_ablation}
\end{figure*}

\section{HMC for deep neural networks}
\label{sec:hmc}

We use full-batch Hamiltonian Monte Carlo (HMC) to sample from the parameter posteriors for Bayesian neural networks.
In this section, we show how to make HMC effective for modern Bayesian neural networks, discussing important details
such as hyper-parameter specification. In the next sections, we use the HMC samples to explore fundamental
questions about approximate inference in modern deep learning.
We summarize HMC in Appendix \autoref{alg:hmc} and \autoref{alg:leapfrog}.
Intuitively, HMC is simulating the dynamics of a particle sliding on the plot of the density function that we are trying to sample from.\footnote{
For a detailed introduction to HMC please see \citet{neal2011mcmc}.
See also interactive visualization here: {\small\url{http://chi-feng.github.io/mcmc-demo/}}.
}

\textbf{Implementation.}\quad
To scale HMC to modern neural network architectures and
for datasets like CIFAR-10 and IMDB, we parallelize the computation over 512 TPUv3  devices\footnote{
We use other hardware configurations in several experiments. We state the hardware that we used in the corresponding sections.} \citep{jouppi2020domain}.
We execute HMC in a single-program multiple-data (SPMD) configuration, wherein a dataset is sharded evenly over each of the devices and an identical HMC implementation is run on each device.
Each device maintains a synchronized copy of the Markov chain state, where the full-batch gradients needed for leapfrog integration are computed using cross-device collectives.
We release our JAX \citep{jax2018github} \href{https://github.com/google-research/google-research/tree/master/bnn_hmc}{\underline{implementation}}.

\textbf{Neural network architectures.}\quad
In our evaluation, following \citet{wenzel2020good}, we primarily focus on two architectures: ResNet-20-FRN and CNN-LSTM.
ResNet-20-FRN is a residual architecture \citep{he2016deep} of depth $20$ with batch normalization layers \citep{ioffe2015batch} replaced with filter
response normalization \citep[FRN; ][]{singh2020filter}.
Batch normalization makes the likelihood harder to interpret by creating dependencies between training examples, whereas
the outputs of FRN layers are independent across inputs.
We use Swish (SiLU) activations \citep{hendrycks2016gaussian, elfwing2018sigmoid, ramachandran2017searching} instead of ReLUs to ensure smoothness of the posterior density surface, which we found improves acceptance rates
of HMC proposals without hurting the overall performance.
The CNN-LSTM is a long short-term memory network \citep{hochreiter1997long} adapted from \citet{wenzel2020good} without modifications.

\textbf{Datasets and Data Augmentation.}\quad
In our main evaluations we use the CIFAR image classification datasets \citep{krizhevsky2014cifar} and the IMDB dataset \citep{imdb} for sentiment analysis.
We do not use any data augmentation, both because the random augmentations introduce stochasticity
into the evaluation of the posterior log-density and its gradient, and because the expected randomly perturbed log-likelihood does not have a clean interpretation as a valid likelihood function
\citep{wenzel2020good}.
In this section we perform ablations using ResNet-20-FRN on CIFAR-10 and CNN-LSTM on IMDB.

\subsection{Trajectory length $\tau$}
\label{sec:trajectory}

The trajectory length parameter $\tau$ determines the length of the dynamics simulation on each HMC iteration.
Effectively, it determines the correlation of subsequent samples produced by HMC.
To suppress random-walk behavior and speed up mixing, we want the length of the trajectory to be relatively high.
But increasing the length of the trajectory also leads to an increased computational cost:
the number of evaluations of the gradient of the target density (evaluations of the gradient of the loss on the full dataset)
is equal to the ratio $\tau / \Delta$ of the trajectory length to the step size.

We suggest the following value of the trajectory length $\tau$:
\begin{equation}
    \label{eqn:tau}
    \textstyle \hat \tau = \frac{\pi \alpha_{\text{prior}}}{2},
\end{equation}
where $\alpha_{\text{prior}}$ is the standard deviation of the prior distribution over the parameters.
If applied to a spherical Gaussian distribution, HMC with a small step size and this trajectory length will generate exact samples\footnote{Since the Hamiltonian defines a set of independent harmonic oscillators with period $2\pi\alpha$, $\tau=\pi\alpha/2$ applies a quarter-turn in phase space, swapping the positions and momenta.}.
While we are interested in sampling from the posterior rather than from the spherical Gaussian prior, we argue that in large BNNs the prior tends to determine the scale of the posterior.
We provide more detail and confirm this intuition empirically in the \autoref{sec:app_marginals}.

In order to test the validity of our recommended trajectory length, we perform an ablation
and report the results in \autoref{fig:tau_ablation}(a).
As expected, longer trajectory lengths provide better performance in terms of accuracy and log-likelihood.
Expected calibration error is generally low across the board.
The trajectory length $\hat \tau$ provides good performance in all three metrics.
This result confirms that, despite the expense, when applying HMC to BNNs it is actually helpful to use tens of thousands of gradient evaluations per iteration.

\subsection{Step size $\Delta$}

The step size parameter $\Delta$ determines the discretization step size of the Hamiltonian dynamics and
consequently the number of leapfrog integrator steps.
Lower step sizes lead to a better approximation of the dynamics and higher rates of
proposal acceptance at the Metropolis-Hastings correction step.
However, lower step sizes require more gradient evaluations per iteration to hold the trajectory length $\tau$ constant.

Using ResNet-20-FRN on CIFAR-10, we run HMC for 50 iterations with step sizes of $1 \cdot 10^{-5}$, $5 \cdot 10^{-5}$, $1 \cdot 10^{-4}$, and $5 \cdot 10^{-4}$ respectively, ignoring the Metropolis-Hastings correction. We find the chains achieve average accept probabilities of $72.2\%$, $46.3\%$, $22.2\%$, and $12.5\%$, reflecting large drops in accept probability
as step size is increased. We also observe BMA log-likelihoods of $-0.331$, $-0.3406$, $-0.3407$, and $-0.895$, indicating that higher accept rates result in higher likelihoods.

\subsection{Number of HMC chains}

We can improve the coverage of the posterior distribution by running multiple independent chains of HMC.
Effectively, each chain is an independent run of the procedure using a different random initialization.
Then, we combine the samples from the different chains.
The computational requirements of running multiple chains are hence proportional to the number of chains.

We report the Bayesian model average performance as a function of the number of chains in \autoref{fig:tau_ablation}(b).
Holding compute budget fixed, using two or three chains is only slightly better than using one chain. This result notably
shows that HMC is relatively unobstructed by energy barriers in the posterior surface
that would otherwise require multiple chains to overcome. We explore this result further in \autoref{sec:mixing}.

\section{How well does HMC mix?}
\label{sec:mixing}

The primary goal of our paper is to construct accurate samples from the posterior, and use them
to understand the properties of Bayesian neural networks better.
In this section we consider several diagnostics to evaluate whether our HMC sampler has converged, and
discuss their implications to the posterior geometry. We consider mixing in both \emph{weight space} and
\emph{function space}. A distribution over weights $w$ combined with a neural network architecture
$f(x,w)$ induces a distribution over functions $f(x)$. Ultimately, since we are using functions to make
predictions, we care mostly about mixing in function space.

\begin{mybox}
    \textbf{Summary}: HMC is able to mix surprisingly well in function space, and better
    than in parameter space.
    Geometrically, HMC is able to explore connected basins of the posterior with high
    functional diversity.
\end{mybox}

\subsection{$\hat R$ diagnostics}
\label{sec:r_hat}

We apply the classic \citet{gelman1992inference} ``$\hat R$'' potential-scale-reduction diagnostic to our HMC runs. Given two or more chains, $\hat R$
estimates the ratio between the between-chain variance (i.e., the variance estimated by pooling samples from all chains) and the average within-chain variance (i.e., the variances estimated from each chain independently).
The intuition is that, if the chains are stuck in isolated regions, then combining samples from multiple chains will yield greater diversity than taking samples from a single chain. For the precise mathematical definition of $\hat R$, please see the \autoref{sec:app_rhat}.

We compute $\hat R$ using TensorFlow Probability's implementation\footnote{\href{https://www.tensorflow.org/probability/api_docs/python/tfp/mcmc/potential_scale_reduction}{tfp.mcmc.potential\_scale\_reduction}} \citep{lao2020tfp} for both the weights and the test-set softmax predictions on CIFAR-10 with ResNet-20-FRN and on IMDB with CNN-LSTM.
We report the results in \autoref{fig:r_hats}.
We observe that on both IMDB and CIFAR, the bulk of the function-space $\hat R$ values is concentrated near $1$,
meaning intuitively that a single chain can capture the diversity of predictions on most of the test data points nearly as well as multiple chains.
The mixing is especially good on the IMDB dataset, where only $2\%$ of inputs correspond to $\hat R$ larger than $1.1$.
In \autoref{sec:app_synthreg} we apply HMC to a synthetic regression problem and show that HMC can indeed mix in the prediction space:
different HMC chains provide very similar predictions.

\begin{figure}[t]
    \centering
    \begin{tabular}{ccc}
    & \hspace{.5cm}{\scriptsize \textbf{Weight Space}} & {\scriptsize \textbf{Function Space}}
    \\
    \rotatebox{90}{\scriptsize \quad\quad\quad \textbf{CIFAR-10}} &
    \hspace{-0.3cm}\includegraphics[height=0.12\textwidth]{figures/rhat_weights_cifar.pdf} &
    \hspace{-0.3cm}\includegraphics[height=0.12\textwidth]{figures/rhat_functions_cifar.pdf}
    \\[-0.2cm]
    \rotatebox{90}{\scriptsize \quad\quad\quad~~ \textbf{IMDB}} &
    \hspace{-0.3cm}\includegraphics[height=0.12\textwidth]{figures/rhat_weights_imdb.pdf} &
    \hspace{-0.3cm}\includegraphics[height=0.12\textwidth]{figures/rhat_functions_imdb.pdf}\\[-0.3cm]
    \end{tabular}
	\caption{
	    \textbf{Log-scale histograms of $\hat R$ convergence diagnostics.}
	    Function-space $\hat R$s are computed on the test-set softmax predictions of the classifiers and weight-space $\hat R$s are computed on the raw weights.
	    About $91\%$ of CIFAR-10 and $98\%$ of IMDB posterior-predictive probabilities get an $\hat R$ less than 1.1. Most weight-space $\hat R$ values are quite small, but enough parameters have very large $\hat R$s to make it clear that the chains are sampling from different distributions in weight space.
	}
	\label{fig:r_hats}
\end{figure}

In weight space, although most parameters show no evidence of poor mixing, some have very large $\hat R$s, indicating that there are directions in which the chains fail to mix.

\paragraph{Implications for the Posterior Geometry.}\quad
The fact that a single HMC chain is able to mix well in function space (aka prediction space) suggests that the posterior contains
connected regions which correspond to high functional diversity.
Indeed, a single HMC chain is extremely unlikely to jump between isolated modes, but appears able to produce samples with diverse predictions.
Moreover, HMC is able to navigate these regions effeciently.
Prior work on \textit{mode connectivity} \citep{garipov2018, draxler2018essentially}  has shown that there exist paths of high density
connecting different modes of the posterior.
Our observations suggest a stronger version of mode connectivity: not only do mode-connecting paths exist between functionally diverse modes, but
also at least some of these paths can be leveraged by Monte Carlo methods to efficiently explore the posterior.

\subsection{Posterior density visualizations}
\label{sec:surface_visualizations}

\begin{figure}[t]
    \centering
    \begin{tabular}{ccc}
    \hspace{-.5cm}
    \includegraphics[height=0.15\textwidth]{figures/posterior_surfaces/cifar_t1_logprob.pdf}&
    \hspace{-.2cm}
    \includegraphics[height=0.15\textwidth]{figures/posterior_surfaces/cifar_t1_ll.pdf}&
    \hspace{-.2cm}
    \includegraphics[height=0.15\textwidth]{figures/posterior_surfaces/cifar_t1_logprior.pdf}\\[-0.2cm]
    \multicolumn{3}{c}{\small (a) Same chain}\\
    \hspace{-.5cm}
    \includegraphics[height=0.15\textwidth]{figures/posterior_surfaces/cifar_diff_chains_t1_logprob.pdf}&
    \hspace{-.2cm}
    \includegraphics[height=0.15\textwidth]{figures/posterior_surfaces/cifar_diff_chains_t1_ll.pdf}&
    \hspace{-.2cm}
    \includegraphics[height=0.15\textwidth]{figures/posterior_surfaces/cifar_diff_chains_t1_logprior.pdf}\\[-0.2cm]
    \multicolumn{3}{c}{\small (b) Independent chains}\\
    \end{tabular}
	\caption{
	    \textbf{Posterior density visualization.}
        Visualizations of posterior log-density, log-likelihood and log-prior in the two-dimensional subspace
        of the parameter space spanned by three HMC samples from \textbf{(a)} the same chain and \textbf{(b)} three independent chains.
        Each HMC chain explores a region of high posterior density of a complex non-convex shape, that appears multi-modal
        in the presented cross-sections.
	}
	\label{fig:posterior_density}
\end{figure}

To further investigate how HMC is able to explore the posterior over the weights, we visualize a cross-section of the posterior density in subspaces of the parameter space containing the samples.
Following \citet{garipov2018}, we study two-dimensional subspaces of the parameter space of the form
\begin{equation}
    \label{eq:subspace}
    \mathcal{S} = \{w \vert w = w_1 \cdot a + w_2 \cdot b + w_3 \cdot (1 - a - b)\}.
\end{equation}
$\mathcal{S}$ is the unique two-dimensional affine subspace (plane) of the parameter space that includes parameter vectors $w_1$, $w_2$ and $w_3$.

In \autoref{fig:posterior_density}(a) we visualize the posterior log-density, log-likelihood and log-prior density of a ResNet-20-FRN on CIFAR-10.
For the visualization, we use the subspace $\mathcal S$ defined by the parameter vectors $w_{1}, w_{51}$ and $w_{101}$,
the samples produced by HMC at iterations $1, 51$ and $101$ after burn-in respectively.
We observe that HMC is able to navigate complex geometry: the samples fall in three seemingly isolated modes in our two-dimensional cross-section of the posterior.
In other words, HMC samples from a single chain are not restricted to any specific convex Gaussian-like mode, and instead explore a region of high posterior density of a complex shape in the parameter space. We note that popular approximate inference procedures, such as variational methods, and Laplace approximations, are typically constrained to unimodal Gaussian approximations to the posterior, which we indeed expect to miss a large space of compelling solutions in the posterior.

In \autoref{fig:posterior_density}(b) we provide a visualization for the samples produced by $3$ different HMC chains at iteration $51$ after burn-in.
Comparing the visualizations for samples from the same chain and samples from independent chains in \autoref{fig:posterior_density}, we see that the shapes of
the posterior surfaces are different, with the latter appearing more regular and symmetric.
The qualitative differences between (a) and (b) suggest that while each HMC chain is able to navigate the posterior geometry
the chains do not mix perfectly in the weight space, confirming our results in \autoref{sec:r_hat}.

In \autoref{sec:app_surface_visualizations} we provide additional details on our visualization procedure and visualizations using the CNN-LSTM on IMDB.

\subsection{Convegence of the HMC chains}

\begin{figure}[t]
    \centering
    \hspace{-0.3cm}
    \begin{tabular}{cccc}
    \hspace{-.5cm}
    \rotatebox{90}{\scriptsize \quad\quad~~~\textbf{CIFAR-10}} &
    \hspace{-.5cm}
    \includegraphics[height=0.11\textwidth]{figures/ablations/burnin_acc.pdf} &
    \hspace{-.5cm}
    \includegraphics[height=0.11\textwidth]{figures/ablations/burnin_ll.pdf} &
    \hspace{-.5cm}
    \includegraphics[height=0.11\textwidth]{figures/ablations/burnin_ece.pdf}
    \\[-0.cm]
    \hspace{-.5cm}
    \rotatebox{90}{\scriptsize \quad\quad\quad~~~\textbf{IMDB}} &
    \hspace{-.5cm}
    \includegraphics[height=0.11\textwidth]{figures/ablations/burnin_lstm_acc.pdf} &
    \hspace{-.5cm}
    \includegraphics[height=0.11\textwidth]{figures/ablations/burnin_lstm_ll.pdf} &
    \hspace{-.5cm}
    \includegraphics[height=0.11\textwidth]{figures/ablations/burnin_lstm_ece.pdf}
    \\[-0.3cm]
    &\multicolumn{3}{c}{\includegraphics[height=0.09\textwidth]{figures/ablations/burnin_legend.pdf}}\\[-0.8cm]
    \vspace{-1.cm}
    \end{tabular}
	\caption{
	    \textbf{HMC convergence.}
	    The performance of an individual HMC sample and a BMA ensemble of $100$ samples from each one of $3$ HMC chains after the burn-in as a function of burn-in length.
	    The dashed line indicates the burn-in length of $50$ that we used in the main experiments in this paper.
	    We use ResNet-20-FRN on CIFAR-10 and CNN-LSTM on IMDB.
	    On IMDB, there is no visible dependence of the results on the burn-in length;
	    on CIFAR-10, there is a weak trend that slows down over time.
	}
	\label{fig:burnin_ablation}
\end{figure}

As another diagnostic, we look at the convergence of the performance of HMC BMA estimates and individual samples as
a function of the length of the burn-in period.
For a converged chain, the performance of the BMA and individual samples should be stationary not show any visible trends after a sufficiently long burn-in.
We use the samples from $3$ HMC chains, and evaluate performance of the ensemble of the first $100$ HMC samples in each chain after
discarding the first $n_{bi}$ samples, where $n_{bi}$ is the length of the burn-in.
Additionally, we evaluate the performance of the individual HMC samples after $n_{bi}$ iterations in each of the chains.

We report the results for ResNet-20-FRN on CIFAR-10 and CNN-LSTM on IMDB in \autoref{fig:burnin_ablation}.
On IMDB, there is no visible trend in performance, so a burn-in of just $10$ HMC iterations should be sufficient.
On CIFAR-10, we observe a slowly rising trend that saturates at about $50$ iterations, indicating that a longer burn-in period is needed compared to IMDB.
We therefore use a burn-in period of $50$ HMC iterations on both CIFAR and IMDB for the remainder of the paper.

\begin{figure}[t]
    \centering
    \hspace{-0.1cm}
    \includegraphics[height=0.3\textwidth]{figures/uci/combined.pdf}
    \vspace{-0.3cm}
	\caption{
	    \textbf{UCI regression datasets.}
	    Performance of Hamiltonian Monte Carlo (HMC), stochastic gradient Langevin dynamics (SGLD),
	    stochastic gradient descent (SGD), subspace inference (SI) \citep{izmailov2019subspace},
	    SWAG \citep{maddoxfast2019} and deterministic variational inference \citep[DVI; ][]{wu2018deterministic}.
	    We use a fully-connected architecture with a single hidden layer of $50$ neurons.
	    The results reported for each method are mean and standard deviation computed over $20$ random
	    train-test splits of the dataset.
	    For SI, SWAG and DVI we report the results presented in \citet{izmailov2019subspace}.
	    \textbf{Top:} test root-mean-squared error.
	    \textbf{Bottom:} test log-likelihood.
	    HMC performs on par with or better than all other baselines in each experiment, often providing a significant
	    improvement.
	}
	\label{fig:uci}
\end{figure}
\begin{figure}[t]
    \centering
    \begin{tabular}{cc}
      \includegraphics[height=0.274\textwidth]{figures/cifar_imdb_combined.pdf} &
      \includegraphics[width=0.145\textwidth]{figures/cifar_imdb_accs_legend.pdf}
    \end{tabular}
	\caption{
	    \textbf{Image and text classification.}
	    Performance of Hamiltonian Monte Carlo (HMC), stochastic gradient Langevin dynamics (SGLD) with 1 and 5 chains,
	    mean field variational inference (MFVI),
	    stochastic gradient descent (SGD), and deep ensembles.
	    We use ResNet-20-FRN on CIFAR datasets, and CNN-LSTM on IMDB.
	    Bayesian neural networks via HMC outperform all baselines on all datasets in terms of accuracy and log-likelihood.
	    On ECE, the methods perform comparably.
	}
	\label{fig:cifar_imdb}
\end{figure}

\section{Evaluating Bayesian neural networks}
\label{sec:bnn_evaluation}

Now that we have a procedure for effective HMC sampling, we are primed to explore exciting questions about the
fundamental behaviour of Bayesian neural networks, such as the role of tempering, the prior over parameters,
generalization performance, and robustness to covariate shift.
In this section we evaluate Bayesian neural networks in various problems using our implementation of HMC.
Throughout the experiments, we use posterior temperature $T=1$.

We emphasize that the main goal of our paper and this section in particular is to \emph{understand} the behaviour of true
Bayesian neural network posteriors using HMC as a precise tool, and \emph{not} to argue for HMC as a practical method for
Bayesian deep learning.

\begin{mybox}
    \textbf{Summary:} Bayesian neural networks achieve strong results outperforming even large deep ensembles in a range of evaluations.
    Surprisingly, however, BNNs are \textit{less} robust to distribution shift than conventionally-trained models.
\end{mybox}

\subsection{Regression on UCI datasets}

Bayesian deep learning methods are often evaluated on small-scale regression problems using fully connected networks
\citep[e.g., ][]{wu2018deterministic, izmailov2019subspace, maddoxfast2019}.
Following these works, we evaluate Bayesian neural networks using HMC on five UCI regression datasets:
\textit{Concrete}, \textit{Yacht}, \textit{Boston}, \textit{Energy} and \textit{Naval}.
For each of these datasets, we construct $20$ random $90$-to-$10$ train-test splits and report the mean
and standard deviation of performance over the splits.
We use a fully connected neural network with a single hidden layer of size $50$ and $2$ outputs
representing the predictive mean and standard deviation.
For HMC we used a single chain with $10$ burn-in iterations and $90$ iterations of sampling.
For more details, please see \autoref{sec:app_hypers}.

We report the results in \autoref{fig:uci}.
HMC typically outperforms all the baselines, often by a significant margin, both in test RMSE and log-likelihood.
On the \textit{Boston} dataset, HMC achieves a slightly higher average RMSE compared to the subspace inference and SWAG
\citep{izmailov2019subspace, maddoxfast2019} but outperforms both these methods significantly in terms of
log-likelihood.

\begin{table}[t]
\begin{center}
\begin{small}
\begin{sc}
\begin{tabular}{lllll}
\toprule
& \multicolumn{4}{c}{AUC-ROC} \\
\cline{2-5}\\[-0.2cm]
OOD Dataset & HMC  & DE & ODIN & Mahal. \\
\midrule
CIFAR-100   & 0.857  & 0.853  & 0.858 & \textbf{0.882}        \\
SVHN        & 0.8814 & 0.8529  & 0.967 & \textbf{0.991}       \\
\bottomrule
\end{tabular}
\end{sc}
\end{small}
\end{center}
\caption{\textbf{Out-of-distribution detection.}
We use a ResNet-20-FRN model trained on CIFAR-10 to detect out-of-distribution data coming from SVHN or CIFAR-100.
We report the results for HMC, deep ensembles and specialized ODIN \citep{liang2017enhancing} and Mahalanobis \citep{lee2018simple} methods.
We report the AUC-ROC score (higher is better) evaluating the ability of each method to distinguish between in-distribution and OOD data.
The predictive uncertainty from Bayesian neural networks allows us to detect OOD inputs:
HMC outpefroms deep ensembles on both datasets.
Furthermore, HMC is competitive with ODIN on the harder near-OOD task of detecting CIFAR-100 images,
but underperforms on the easier far-OOD task of detecting SVHN images.
}
\label{tab:ood}
\end{table}

\subsection{Image Classification on CIFAR}
\label{sec:image_classification}

Next, we evaluate Bayesian neural networks using HMC on image classification problems.
We use the ResNet-20-FRN architecture on CIFAR-10 and CIFAR-100.
We picked a random subset of 40960 of the 50000 images for each of the datasets to be able to evenly shard the data across the TPU devices;
we use the same subset for both HMC and the baselines.
We run 3 HMC chains using step size $10^{-5}$ and a prior variance of $1/5$, resulting in 70,248 leapfrog steps per sample.
In each chain we discard the first 50 samples as burn-in, and then draw 240 samples (720 in total for 3 chains)\footnote{
In total, on CIFAR-10 our HMC run requires as many computations as \textit{over 60 million epochs} of standard SGD training}.
For SGLD, we use a single chain with $1000$ burn-in epochs and $9000$ epochs of sampling producing $900$ samples;
we also report the performance of an ensemble of $5$ independent SGLD chains.
Next, we report the performance of a mean field variational inference (MFVI) solution;
we initialize the mean of MFVI with a solution pre-trained with SGD and use an ensemble of $50$ samples from the VI posterior at evaluation.
Finally, we report the performance of a single SGD solution and a deep ensemble of $50$ models.
For more details, see \autoref{sec:app_hypers}.

We report the results in \autoref{fig:cifar_imdb}.
Bayesian neural networks outperform all baselines in terms of accuracy and log-likelihood on both datasets.
In terms of ECE, SGD provides the worst results across the board, and the rest of the methods are competitive;
MFVI is particularly well-calibrated on CIFAR-100.

\textbf{Out-of-distribution detection.}\quad
Bayesian deep learning methods are often evaluated on out-of-distribution detection.
In the \autoref{tab:ood} we report the performance of HMC-based Bayesian neural network on out-of-distribution (OOD) detection.
To detect OOD data, we use the level of predicted confidence (value of the softmax class probability for the predicted class)
from the HMC ensemble, measuring the area under the receiving operator characteristic curve (AUC-ROC).
We train the methods on CIFAR-10 and use CIFAR-100 and SVHN as OOD data sources.
We find that BNNs perform competitively with the specialized ODIN method in the challenging near-OOD detection setting
(i.e. when the OOD data distribution is similar to the training data) of CIFAR-100, while
underperforming in the easier far-OOD setting on SVHN relative to the baselines \citep{liang2017enhancing, lee2018simple}.

\begin{figure}[t]
    \centering
    \hspace{-0.1cm}
    \begin{tabular}{cc}
    \hspace{-0.5cm}
    \includegraphics[height=0.175\textwidth]{figures/cifar10c/boxplot_acc.pdf} &
    \hspace{-0.5cm}
    \includegraphics[height=0.175\textwidth]{figures/cifar10c/boxplot_ll.pdf} \\
    \multicolumn{2}{c}{\includegraphics[width=0.45\textwidth]{figures/cifar10c/boxplot_legend.pdf}}\\[-1.cm]
    \end{tabular}
	\caption{
	    \textbf{Evaluation on CIFAR-10-C.}
	    Accuracy and log-likelihood of HMC, SGD, deep ensembles, SGLD and MFVI on a distribution shift task, where
	    the CIFAR-10 test set is corrupted in 16 different ways at various intensities on the scale of 1 to 5.
	    We use the ResNet-20-FRN architecture.
	    Boxes capture the quartiles of performance over each corruption, with
	    the whiskers indicating the minimum and maximum.
	    HMC is surprisingly the worst of the considered methods: even a single SGD solution provides better OOD robustness.
	}
	\label{fig:cifar10c}
\end{figure}

\textbf{Robustness to distribution shift}
\quad
Bayesian methods are often specifically applied to covariate shift problems \citep{ovadia2019can, wilson2020bayesian, dusenberry2020efficient}.
We evaluate the the performance of HMC and Deep Ensemble-based Bayesian neural networks on the CIFAR-10-C dataset \citep{hendrycks2019benchmarking},
which applies a set of corruptions to CIFAR-10 with varying intensities.
Mimicking the setup in \citet{ovadia2019can}, we use the same 16 corruptions, evaluating the performance at all intensities.
We report the results in \autoref{fig:cifar10c}.
Surprisingly, we find that Deep Ensembles and SGLD are consistently more robust to distribution shift than HMC-based BNNs.
For high corruption intensities, even a single SGD model outperforms the HMC ensemble.

In \autoref{sec:app_robustness} we provide further exploration of this effect, where we see HMC samples are significantly
less robust to many types of noise compared to conventionally-trained SGD models.
Interestingly, the performance of HMC-based BNNs under data corruption can be significantly improved by using posterior tempering.

\subsection{Language Classification on IMDB}

We use a CNN-LSTM architecture on the IMDB binary text classification dataset.
In \autoref{fig:cifar_imdb} we report the results for HMC and the same baselines as in \autoref{sec:image_classification}.
We use HMC with a step size of $10^{-5}$ and a prior variance of $1/40$,
resulting in 24,836 leapfrog steps per sample.
We run $3$ chains, burning-in for 50
samples, and drawing 400 samples per chain (1,200 total).
For more details on the hyperparameters, please see \autoref{sec:app_hypers}.
Analogously to the image classification experiments, HMC outperforms the baselines on accuracy and log-likelihood and provides competitive performance on ECE.

\section{Do we need cold posteriors?}
\label{sec:cold_posteriors}

Multiple works have considered tempering the posterior in Bayesian neural networks \citep[e.g.][]{wenzel2020good, wilson2020bayesian, zhang2019cyclical, ashukha2020pitfalls, aitchison2020statistical}.
Specifically, we can consider a distribution
\begin{equation}
    \label{eq:posterior_temp}
    p_T(w \vert \mathcal D) \propto \big ( p(\mathcal D \vert w) \cdot p(w) \big )^{1 / T},
\end{equation}
where $w$ are the parameters of the network, $\mathcal D$ is the training dataset, $p(\mathcal D \vert w)$ is the likelihood of $\mathcal D$ for the network with parameters $w$ and $T$ is the \textit{temperature}.
Note that at temperature $T = 1$, $p_T$ corresponds to the standard Bayesian posterior over the parameters of the network.
Temperatures $T < 1$ correspond to \textit{cold posteriors}, distributions that are sharper than the Bayesian posterior.
Similarly, temperatures $T > 1$ correspond to \textit{warm posteriors} which are softer than the Bayesian posterior.
See Appendix \autoref{fig:app_posterior_density}(d) for a visualization of the log-likelihood density surface at different temperatures.

\citet{wenzel2020good} argue that Bayesian neural networks require a cold posterior, and the performance at temperature $T=1$ is inferior to even a single model trained with SGD.
The authors refer to this phenomenon as \textit{the cold posteriors effect}.
However, our results are different:
\begin{mybox}
    \textbf{Summary}: We show that that cold posteriors are not needed to obtain near-optimal performance with Bayesian neural networks
    and may even hurt performance. We show that the cold posterior effect is largely an artifact of data augmentation.
\end{mybox}

\begin{figure}
    \centering
    \hspace{-.2cm}
    \includegraphics[height=0.125\textwidth]{figures/ablations/imdb_temp_acc.pdf}
    \hspace{-.2cm}
    \includegraphics[height=0.125\textwidth]{figures/ablations/imdb_temp_nll.pdf}
    \hspace{-.2cm}
    \includegraphics[height=0.125\textwidth]{figures/ablations/imdb_temp_ece.pdf}
    \includegraphics[width=0.26\textwidth]{figures/ablations/temp_legend.pdf}
    \vspace{-0.8cm}
    \caption{\textbf{Effect of posterior temperature.}
    The effect of posterior temperature $T$ on the log-likelihood, accuracy and expected calibration error using the CNN-LSTM model on the IMDB dataset.
    For both the log-likelihood and accuracy $T = 1$ provides optimal performance, while for the ECE the colder posteriors provide a slight improvement.
    For all three metrics, the posterior at $T = 1$ outperforms the SGD baseline as well as a deep ensemble of $10$ independently trained models.
    }
    \label{fig:temp_effect}
\end{figure}

\subsection{Testing the cold posteriors effect}

\citet{wenzel2020good} demonstrate the cold posteriors with two main experiments: ResNet-20 on CIFAR-10 and CNN-LSTM on IMDB.
In these experiments the authors show poor performance at temperature $T=1$, with strong benefits from decreasing the temperature.
However, for the CIFAR-10 experiment, it is apparent \citep[Appendix K, Figure 28]{wenzel2020good} that the results at $T = 1$
are near-optimal for the ResNet on CIFAR-10 if data augmentation is turned off and batch normalization is replaced with filter response normalization,
which is in fact necessary for a clear Bayesian interpretation of the inference procedure.

Furthermore, in \autoref{sec:bnn_evaluation}, we show that Bayesian neural networks can achieve performance superior to SGD and even deep ensembles at temperature $T=1$,
in particular using the same ResNet-20-FRN model on CIFAR-10 and CNN-LSTM model on IMDB used by \citet{wenzel2020good}.

To further understand the effect of posterior temperature $T$, we compare the performance of the CNN-LSTM model at different $T$ using our Hamiltonian Monte Carlo sampler.
In all runs we used a fixed prior variance $\alpha^2 = \frac 1 {40}$.
We report the results in \autoref{fig:temp_effect}.
We find that the performance of the BNN at $T=1$ is better than the SGD baseline as well as a deep ensemble of $50$ independent models.
Moreover, the performance at $T=1$ is better compared to all other temperatures we tested in terms of both test accuracy and log-likelihood.

We also note that while posterior tempering does not seem necessary for good predictive performance with BNNs, it may be helpful
under distribution shift.
In \autoref{sec:app_robustness} we show that decreasing the temperature can significantly improve the robustness of BNN predictions
to noise in the test inputs. \citet{wilson2020bayesian} additionally argue that tempering may be a reasonable procedure in general, and is
not necessarily at odds with Bayesian principles.

\textbf{Role of data augmentation.}\quad
Our results are in contrast with \citet{wenzel2020good}, who argue that cold posteriors are needed for good performance with BNNs.
In \autoref{sec:app_cold_posteriors} we provide an additional study of what may have caused the poor performance of BNNs in \citet{wenzel2020good},
using the code for inference provided by \citet{wenzel2020good}.
We identify data augmentation as the key factor responsible for the cold posterior effect, and also show that batch normalizing does not significantly influence this effect:
when the data augmentation is turned off, we do not observe the cold posteriors effect.
Data augmentation cannot be naively incorporated in the Bayesian neural network model (see the discussion in appendix K of \citet{wenzel2020good}),
and arguably it may be reasonable to decrease the temperature when using data augmentation:
intuitively, data augmentation increases the amount of data observed by the model, and should lead
to higher posterior contraction.
We leave incorporating data augmentation in our HMC evaluation framework as an exciting direction of future work.

\section{What is the effect of  priors in Bayesian neural networks?}
\label{sec:app_priors}

\begin{figure}[t]
    \centering

    \hspace{-0.3cm}
    \begin{tabular}{cccc}
    \hspace{-.3cm}
    \rotatebox{90}{\scriptsize \quad\quad~~~\textbf{CIFAR-10}} &
    \hspace{-.3cm}
    \includegraphics[height=0.11\textwidth]{figures/ablations/cifar_prior_acc.pdf} &
    \hspace{-.4cm}
    \includegraphics[height=0.11\textwidth]{figures/ablations/cifar_prior_nll.pdf} &
    \hspace{-.5cm}
    \includegraphics[height=0.11\textwidth]{figures/ablations/cifar_prior_ece.pdf}
    \\[-0.cm]
    \hspace{-.3cm}
    \rotatebox{90}{\scriptsize \quad\quad\quad~~~\textbf{IMDB}} &
    \hspace{-.3cm}
    \includegraphics[height=0.11\textwidth]{figures/ablations/imdb_prior_acc.pdf} &
    \hspace{-.4cm}
    \includegraphics[height=0.11\textwidth]{figures/ablations/imdb_prior_nll.pdf} &
    \hspace{-.5cm}
    \includegraphics[height=0.11\textwidth]{figures/ablations/imdb_prior_ece.pdf}\\[-0.3cm]
    \end{tabular}
	\caption{
	    \textbf{Effect of prior variance.}
        The effect of prior variance on BNN performance.
        In each panel, the dashed line shows the performance of the SGD model from \autoref{sec:bnn_evaluation}.
        While low prior variance may lead to over-regularization and hurt performance, all the considered
        prior scales lead to better results than the performance of an SGD-trained neural network of the same architecture.
	}
	\label{fig:prior_ablation}
\end{figure}

Bayesian deep learning is often criticized for the lack of intuitive priors over the parameters.
For example, \citet{wenzel2020good} hypothesize that the popular Gaussian priors of the form $\mathcal{N}(0, \alpha^2 I)$ are inadequate and lead to poor performance.
\citet{tran2020all} propose a new prior for Bayesian neural networks inspired by Gaussian processes \citep{rasmussen10gpml} based on this hypothesis. In concurrent work,
\citet{fortuin2021bayesian} also explore several alternatives to standard Gaussian priors inspired by the cold posteriors effect.
\citet{wilson2020bayesian} on the other hand, argue that vague Gaussian priors in the parameter space induce useful function-space priors.

In \autoref{sec:bnn_evaluation} we have shown that Bayesian neural networks can achieve strong performance with vague Gaussian priors.
In this section, we explore the sensitivity of BNNs to the choice of the prior scale as well as several alternative prior families,
as a step towards a better understanding of the role of the prior in BNNs.

\begin{mybox}
    \textbf{Summary:} High-variance Gaussian priors over parameters of BNNs lead to strong performance.
    The results are robust with respect to the prior scale. Mixture of Gaussian and logistic priors over parameters are not too
    different in performance to Gaussian priors. These results highlight the relative importance of architecture over parameter priors
    in specifying a useful prior over functions.
\end{mybox}

\subsection{Effect of Gaussian prior scale}

We use priors of the form $\mathcal{N}(0, \alpha^2 I)$ and vary the prior variance $\alpha^2$.
For all cases, we use a single HMC chain producing $40$ samples.
These are much shorter chains than the ones we used in \autoref{sec:bnn_evaluation}, so the results
are not as good;
the purpose of this section is to explore the \textit{relative} performance of BNNs under different priors.

We report the results for the CIFAR-10 and IMDB datasets in \autoref{fig:prior_ablation}.
When the prior variance is too small, the regularization is too strong, hindering the performance.
Setting the prior variance too large does not seem to hurt the performance as much.
On both problems, the performance is fairly robust: a wide window of prior variances lead
to strong performance.
In particular, for all considered prior scales, the results are better than those of SGD training.

\textbf{Why are BNNs so robust to the prior scale?}\quad One possible explanation for the relatively flat curves in \autoref{fig:prior_ablation} is that large prior variances imply a strong prior belief that the ``true'' classifier (i.e., the model that would be learned given infinite data) should make high-confidence predictions.
Since the model is powerful enough to achieve any desired training accuracy, the likelihood does not overrule this prior belief, and so the posterior assigns most of its mass to very confident classifiers.
Past a certain point, increasing the prior variance on the weights may have no effect on the classifiers' already saturated probabilities.
Consequently, nearly every member of the BMA may be highly overconfident.
But the \emph{ensemble} does not have to be overconfident---a mixture of overconfident experts can still make well-calibrated predictions.
Appendix \autoref{fig:overconfident} provides some qualitative evidence for this explanation; for some CIFAR-10 test set images, the HMC chain oscillates between assigning the true label probabilities near 1 and probabilities near 0.

\begin{table}[t]
\begin{center}
\begin{small}
\begin{sc}
\begin{tabular}{@{}llll@{}}
\toprule
Prior          & Gaussian    & MoG        & Logistic   \\ \midrule
Accuracy       & 0.866       & 0.863        & \textbf{0.869}       \\
ECE            & 0.029       & 0.025        & \textbf{0.024}       \\
Log Likelihood & -0.311      & -0.317       & \textbf{-0.304}      \\ \bottomrule
\end{tabular}
\end{sc}
\end{small}
\end{center}
\caption{\textbf{Non-Gaussian priors.} BMA accuracy, ECE,
and log-likelihood under different prior families using CNN-LSTM on IMDB.
We produce 80 samples from a single HMC chain for each of the priors.
The heavier-tailed logistic prior provides slightly better performance
compared to the Gaussian and mixture of Gaussians (MoG) priors.}
\label{tab:alternate-priors}
\end{table}

\begin{table*}[t]
    \begin{center}
    \begin{small}
    \begin{sc}
    \begin{tabular}{c c ccc cccc}
    \toprule
    & & & & & \multicolumn{4}{c}{SGMCMC} \\
    \cline{6-9}\\[-0.2cm]
    Metric            &
    \begin{tabular}{c}HMC\\(reference)\end{tabular}
    & SGD & Deep Ens     & MFVI    & SGLD   &  SGHMC  &
    \begin{tabular}{c}SGHMC\\CLR\end{tabular} &
    \begin{tabular}{c}SGHMC\\CLR-Prec\end{tabular}
    \\
    \midrule
    \multicolumn{8}{c}{CIFAR-10}\\
    \midrule
    \multirow{2}{*}{Accuracy}
    & $89.64$      & $83.44$      & $88.49$      & $86.45$      & $89.32$      & $89.38$      & $\mathbf{89.63}$      & $87.46$ \\[-0.1cm]
    & {\scriptsize ${\pm 0.25}$} & {\scriptsize ${\pm 1.14}$} & {\scriptsize ${\pm 0.10}$} & {\scriptsize ${\pm 0.27}$} & {\scriptsize ${\pm 0.23}$} & {\scriptsize ${\pm 0.32}$} & {\scriptsize $\mathbf{{\pm 0.37}}$} & {\scriptsize ${\pm 0.21}$} \\[0.1cm]
    \multirow{2}{*}{Agreement}
    & $94.01$ &  $85.48$ &  $91.52$ &  $88.75$ &  $91.54$ &  $91.98$ &  $\mathbf{92.67}$ &  $90.96$\\[-0.1cm]
    & {\scriptsize ${\pm 0.25}$  } & {\scriptsize ${\pm 1.00}$  } & {\scriptsize ${\pm 0.06}$  } & {\scriptsize ${\pm 0.24}$  } & {\scriptsize ${\pm 0.15}$  } & {\scriptsize ${\pm 0.35}$  } & {\scriptsize $\mathbf{\pm 0.52}$  } & {\scriptsize ${\pm 0.24}$}\\[0.1cm]
    \multirow{2}{*}{Total Var}
    & $0.074$ &  $0.190$ &  $0.115$ &  $0.136$ &  $0.110$ &  $0.109$ &  $\mathbf{0.099}$ &  $0.111$ \\[-0.1cm]
    & {\scriptsize ${\pm 0.003}$ } & {\scriptsize ${\pm 0.005}$ } & {\scriptsize ${\pm 0.000}$ } & {\scriptsize ${\pm 0.000}$ } & {\scriptsize ${\pm 0.001}$ } & {\scriptsize ${\pm 0.001}$ } & {\scriptsize $\mathbf{\pm 0.006}$ } & {\scriptsize ${\pm 0.002}$} \\
    \midrule
    \multicolumn{8}{c}{CIFAR-10-C}\\
    \midrule
    \multirow{2}{*}{Accuracy}
    & $70.91$ &  $71.04$ &  $76.99$ &  $75.40$ &  $\mathbf{78.80}$ &  $78.20$ &  $76.43$ &  $73.42$ \\[-0.1cm]
    & {\scriptsize ${\pm 0.93}$  } & {\scriptsize ${\pm 1.80}$  } & {\scriptsize ${\pm 0.39}$  } & {\scriptsize ${\pm 0.34}$} & {\scriptsize $\mathbf{\pm 0.17}$  } & {\scriptsize ${\pm 0.25}$  } & {\scriptsize ${\pm 0.39}$} & {\scriptsize ${\pm 0.39}$}\\[0.1cm]
    \multirow{2}{*}{Agreement}
    &$86.00$ &  $72.01$ &  $79.29$ &  $75.47$ &  $77.99$ &  $78.98$ &  $\mathbf{80.93}$ &  $79.65$ \\[-0.1cm]
    &{\scriptsize ${\pm 0.44}$  } & {\scriptsize ${\pm 0.82}$  } & {\scriptsize ${\pm 0.18}$  } & {\scriptsize ${\pm 0.27}$  } & {\scriptsize ${\pm 0.22}$} & {\scriptsize ${\pm 0.22}$} & {\scriptsize $\mathbf{\pm 0.73}$} & {\scriptsize ${\pm 0.35}$} \\[0.1cm]
    \multirow{2}{*}{Total Var}
    & $0.133$ &  $0.334$ &  $0.220$ &  $0.245$ &  $0.214$ &  $0.203$ &  $\mathbf{0.194}$ &  $0.205$ \\[-0.1cm]
    &{\scriptsize ${\pm 0.004}$ } & {\scriptsize ${\pm 0.007}$ } & {\scriptsize ${\pm 0.003}$ } & {\scriptsize ${\pm 0.002}$ } & {\scriptsize ${\pm 0.002}$ } & {\scriptsize ${\pm 0.002}$ } & {\scriptsize $\mathbf{\pm 0.010}$ } & {\scriptsize ${\pm 0.005}$}\\
    \bottomrule
    \end{tabular}
    \end{sc}
    \end{small}
    \end{center}
    \caption{\textbf{Evaluation of cheaper alternatives to HMC.}
    Agreement and total variation between predictive distributions of HMC and approximate inference methods: deep ensembles, mean field variational inference (MFVI), and stochastic gradient Monte Carlo (SGMCMC) variations.
    For all methods we use ResNet-20-FRN trained on CIFAR-10 and evaluate predictions on the CIFAR-10 and CIFAR-10-C test sets.
    For CIFAR-10-C we report the average results across all corruptions and corruption intensities.
    We additionally report the results for HMC for reference: we compute the agreement and total variation between one of the chains and the ensemble of the other two chains.
    For each method we report the mean and standard deviation of the results over three independent runs.
    MFVI provides the worst approximation of the predictive distribution.
    Deep ensembles despite often being considered non-Bayesian, significantly outperform MFVI.
    SG-MCMC methods provide the best results with SGHMC-CLR showing the best overall performance.
    }
    \label{tab:pred_comparison}
\end{table*}

\subsection{Non-Gaussian priors}

In \autoref{tab:alternate-priors}, we report BMA accuracy, ECE, and log-likelihood
for two non-Gaussian priors on the IMDB dataset: \textit{logistic} and \textit{mixture of Gaussians} (MoG).
For the MoG prior we use a mixture of two Gaussians centered at $0$, one with variance $\frac 1 {40}$ and the other with variance $\frac 1 {160}$.
We pick prior scale of the logistic prior to have a variance of $\frac{1}{40}$.
We additionally provide the results for a Gaussian prior with variance $\frac 1 {40}$.
We approximate the BMA using 80 samples from a single HMC chain for each of the priors.
We find that the heavier-tailed logistic prior performs slightly better than the Gaussian and MoG.

\subsection{Importance of Architecture in Prior Specification}
We often think of the prior narrowly in terms of a distribution over parameters $p(w)$. But the prior that matters
is the prior over functions $p(f(x))$ that is induced when a prior over parameters $p(w)$ is combined with the
functional form of a neural network $f(x,w)$. All of the results in this section point to the relative importance of
the architecture in defining the prior over functions, compared to the prior over parameters. A vague prior over
parameters is not necessarily vague in function-space.
Moreover, while the details of the prior distribution over
parameters $p(w)$ have only a minor effect on performance, the choice of architecture certainly has a major effect
on performance.

\section{Do scalable BDL methods and HMC make similar predictions?}
\label{sec:alternatives}

While HMC shows strong performance in our evaluation in \autoref{sec:bnn_evaluation},
in most realistic BNN settings it is an impractical method.
However, HMC can be used as a \textit{reference} to evaluate and calibrate more scalable and practical
alternatives.
In this section, we evaluate the \emph{fidelity} of SGMCMC, variational methods, and deep ensembles in representing the
predictive distribution (Bayesian model average) given by our HMC reference.

\begin{mybox}
    \textbf{Summary}: While SGMCMC and Deep Ensembles can provide good generalization accuracy and calibration, their predictive distributions differ from HMC. Deep ensembles are similarly close to the HMC predictive distribution as SGLD, and closer than standard variational inference.
\end{mybox}

\subsection{Comparing the predictive distributions}
We consider two primary metrics: \textit{agreement} and \textit{total variation}.
We define the agreement between the predictive distributions $\hat p$ of HMC and
$p$ of another method as the fraction of the test data points for which the top-1 predictions of $\hat p$ and $p$ are the same:
\begin{equation*}
    \frac 1 n \sum_{i=1}^n I[\arg\max_j \hat p (y = j \vert x_i) = \arg\max_j p(y = j \vert x_i)],
\end{equation*}
where $I[\cdot]$ is the indicator function and $n$ is the number of test data points $x_i$.
We define the total variation metric between $\hat p$ and $p$ as the total variation distance between the predictive distributions averaged over the test data points:
\begin{equation*}
    \frac 1 n \sum_{i=1}^n \frac{1}{2}\sum_j \bigg| \hat p(y = j \vert x_i) - p(y = j \vert x_i)\bigg|.
\end{equation*}
The agreement (higher is better) captures how well a method is able to capture the top-1 predictions of HMC,
while the total variation (lower is better) compares the predictive probabilities for each of the classes.

In \autoref{tab:pred_comparison} we report the agreement and total variation metrics as well as the predictive accuracy on the CIFAR-10 and CIFAR-10-C test sets
for a deep ensemble of $50$ models and several SGLD variations:
standard SGLD \citep{welling2011bayesian}, SGLD with momentum (SGHMC) \citep{chen2014stochastic},
SGLD with momentum and a cyclical learning rate schedule (SGHMC-CLR) \citep{zhang2019cyclical} and
SGLD with momentum, cyclical learning rate schedule and a preconditioner (SGHMC-CLR-Prec) \citep{wenzel2020good}.
All methods were trained on CIFAR-10.
For more details, please see \autoref{sec:app_hypers}.

Overall, the absolute value of agreement achieved by all methods is fairly low
on CIFAR-10 and especially on CIFAR-10-C.
More advanced SGHMC-CLR and SGHMC-CLR-Prec methods provide a better fit of the HMC predictive distribution
while not necessarily improving the accuracy.
Notably, these methods are also \textit{less} robust to the data corruptions in CIFAR-10-C, again suggesting that
higher fidelity representations of the predictive distribution can lead to decreased robustness to covariate shift,
as we found in section~\ref{sec:image_classification}.

Deep ensembles, while not typically considered to be a Bayesian method, provide a reasonable
approximation to the HMC predictive distribution.
In particular, deep ensembles outperform both SGLD and SGHMC in terms of total variation on CIFAR-10 and in terms of agreement on CIFAR-10-C.
These results support the argument that deep ensembles, while not typically characterized as a Bayesian method, provide
a \emph{higher fidelity} approximation to a Bayesian model average than methods that are conventionally accepted as Bayesian
inference procedures in modern deep learning \citep{wilson2020bayesian}.

In \autoref{sec:app_robustness} we explore the performance of HMC, SGD, deep ensembles, SGLD and SGHMC-CLR-Prec under different corruptions individually.
Interestingly, the behavior of SGLD and SGHMC-CLR-Prec appears more similar to that of deep ensembles than that of HMC.
So, while both SGMCMC and deep ensembles are very compelling practically, they provide relatively distinct predictive distributions from HMC.
Mean-field variational inference methods are particularly far from the HMC predictive distribution.
Thus, we should be very careful when making judgements about \textit{true} Bayesian neural networks based on the SGMCMC or MFVI performance.

\subsection{Predictive entropy and calibration curves}

\begin{figure}
    \centering
    \includegraphics[height=0.18\textwidth]{figures/entropy_dists.pdf}
    \includegraphics[height=0.18\textwidth]{figures/calibration_curve.pdf}
    \vspace{-0.1cm}

    \includegraphics[width=0.35\textwidth]{figures/fig10_legend.pdf}\vspace{-0.7cm}

    \caption{
    Distribution of predictive entropies (\textbf{left}) and calibration curve (\textbf{right}) of posterior predictive distributions for
    HMC, SGD, deep ensembles, MFVI, SGLD and SGHMC-CLR-Prec for ResNet20-FRN on CIFAR-10.
    On the left, for all methods, except HMC we plot a pair of histograms: for HMC and for the corresponding method.
    SGD, Deep ensembles and MFVI provide more confident predictions than HMC.
    SGMCMC methods appear to fit the predictive distribution of HMC better: SGLD is slightly underconfident relative to HMC while SGHMC-CLR-Prec is slightly over-confident.
    }
    \label{fig:posterior_predictive}
\end{figure}

To provide an additional comparison of the predictive distributions between HMC and other methods, in \autoref{fig:posterior_predictive}
we visualize the distribution of predictive entropies and the calibration curves for HMC, SGD, deep ensembles, MFVI, SGLD and SGHMC-CLR-Prec on CIFAR-10 using ResNet-20-FRN.

All methods except fot SGD make conservative predictions: their confidences tend to underestimate their accuracies (\autoref{fig:posterior_predictive}, right);
SGD on the other hand is very over-confident \citep[in agreement with the results in ][]{guo2017calibration}.
Deep ensembles and MFVI provide the most calibrated predictions, while SGLD and SGHMC-CLR-Prec match the HMC entropy distribution and calibration curve closer.

\section{Discussion}

Despite the rapidly increasing popularity of approximate Bayesian inference in modern deep learning,
little is known about the behaviour of truly Bayesian neural networks.
To the best of our knowledge, our work provides the first realistic evaluation of Bayesian neural networks
with precise and exhaustive posterior sampling. We establish several properties of Bayesian neural networks,
including good generalization performance, lack of a cold posterior effect, and a surprising lack of robustness
to covariate shift.
We hope that our observations and the tools that we develop will facilitate fundamental progress in understanding
the behaviour of Bayesian neural networks.

\textbf{Is HMC Converging?}\quad
In general, it is not possible to ensure that an MCMC method has converged to sampling from the true posterior distribution:
theoretically, there may always remain regions of the posterior that cannot be discovered by the method but that contain most of the posterior mass.
To maximize the performance of HMC, we choose the hyper-parameters that are the most likely to provide convergence: long trajectory lengths, and multiple long chains.
In \autoref{sec:mixing}, we study the convergence of HMC using the available convergence diagnostics.
We find that while HMC does not mix perfectly in weight space, in the space of predictions we cannot find evidence of non-mixing.

\textbf{Should we use Bayesian neural networks?}\quad
Our results show that Bayesian neural networks can improve the performance over models trained with SGD in a variety of settings,
both in terms of uncertainty calibration and predictive accuracy.
On most of the problems we considered in this work, the best results were achieved by Bayesian neural networks.
We believe that our results provide motivation to use Bayesian neural networks with accurate posterior approximation in practical applications.

\textbf{Should we use HMC in practice?}\quad
For most realistic scenarios in Bayesian Deep Learning, HMC is an impractical method.
On the image classification benchmarks, HMC takes orders of magnitude more compute than any of the baselines that we considered.
We hope that our work will inspire the community to produce new accurate and scalable approximate inference methods for Bayesian deep learning.

\subsection{Challenging conventional wisdom}

A conventional wisdom has emerged that deep ensembles are a non-Bayesian alternative to variational methods, that standard priors for neural networks are poor, and that cold posteriors
are a problematic result for Bayesian deep learning. Our results highlight that one should take care in uncritically
repeating such claims. In fact,
deep ensembles appear to provide a higher fidelity representation of the Bayesian predictive distribution than widely accepted approaches to approximate
Bayesian inference. If anything, the takeaway from the relatively good performance of deep ensembles is that we would benefit from approximate inference
being \emph{closer} to the Bayesian ideal! Moreover, the details over the priors in weight space can have a relatively minor effect on performance, and there
is no strong evidence that standard Gaussian priors are particularly bad. In fact, there are many reasons to believe these priors have useful properties
\citep{wilson2020bayesian}. Similarly, on close inspection, we found no evidence for a general cold posterior effect, which we identify as largely an artifact of
data augmentation. Although we see here that tempering does not in fact seem to be required, as argued in \citet{wilson2020bayesian} tempering is also not necessarily unreasonable or even divergent from Bayesian principles.

Even the results we found that are less favourable to Bayesian deep learning are contrary to the current orthodoxy. Indeed,
higher fidelity Bayesian inference surprisingly appears to suffer more greatly from covariate shift, despite the popularity of approximate Bayesian inference procedures in this
setting.

\subsection*{Acknowledgements}
The authors would like to thank many people at Google for many helpful discussions and much helpful feedback, especially Rodolphe Jenatton, Rif A. Saurous, Jasper Snoek, Pavel Sountsov, Florian Wenzel, and the entire TensorFlow Probability team. This  research  is  supported  by  an  Amazon  Research  Award,  NSF  I-DISRE  193471,  NIH  R01DA048764-01A1,  NSF IIS-1910266,  and NSF 1922658NRT-HDR: FUTURE Foundations,  Translation,  and Responsibility for Data Science.

redpajama_set_name: RedPajamaArXiv


Some of our children at Gates visited our Grandmas and Grandpas last week, and apples were on the agenda. They discussed the different varieties of apples and their health benefits. They read the story, "The little Red House with No Doors and No Windows," which tells the tale of a little boy and his search for a "house" which also has a star inside. We also learned that when an apple is cut in half, a star is revealed!
Each summer our Child Care Camp puts on the Krazy Kids Karnival! Each location designs, builds, works and then plays the carnival games. Our Edler Care program also gets involved and this year enjoyed a picnic lunch with our kids and watched each centers perform at the annual "all camp" Talent Show. They also got busy putting together all the prize bags for the children. The "photo booth" is always a huge hit! The kids and seniors dressed up in crazy hats and masks and use props to create one-of-a-kind fun-filled photos! What a fantastic event it is every year!
May is Older Americans Month, and one of our many celebrations this month included recognizing the value of meaningful social interaction between our older and younger generations. For our Inter-generational celebration, we invited Gates Town Supervisor, Mark Assini, to join in by reading to our grandmas, grandpas, and children. He amused us all by singing and acting out the Spiderman song! Later in the day, the Gates-Chili Jazz Ensemble entertained us with several tunes that had the kids up and dancing and the seniors smiling from ear to ear! We also celebrated this month with an ice cream social and today we are focusing on health and fitness. Lots of fun, exciting things happening this month!
Every year our Elder Care participants take a trip to a local farm store, The Garden Factory, to view the beautiful holiday Pointsettia flowers and the amazing toy train exhibit. The Pre-K class from St. Mike's joined them this year and had a wonderful time with the grandmas and grandpas!
We had a small dedication ceremony with our "grandmas and grandpas" and their families. The photos show Donna Delvecchio, Elder Care Coordinator, Mae Evans, who works with Donna, and Chris and Jack Long, Lu's daughter and son-in-law. Many thanks to Grandma Lu's family for the generous gift.
What is Social Adult Day Care?
New York State defines Social adult day care programs as those offering "a structured, comprehensive program which provides functionally impaired individuals with socialization; supervision and monitoring; personal care; and nutrition in a protective setting during any part of the day, but for less than 24 hours".
Our innovative program offers a safe environment where Generations strives to meet the needs of Seniors and Caregivers. We aim to foster new peer friendships, help participants find interests and skills they never knew they had and help them to maintain a healthy meaningful lifestyle.


Responsible for providing the strategic direction and management of the planning, development and implementation of various projects and initiatives in support of department or division. Typically will manage the most complex, critical and highly visible projects at both the division and/or bank-wide level.
Completes activities related to larger and/or more complex installations, configuration, operation and maintenance of systems hardware and software and related infrastructure. Ensures system hardware, operating systems, software systems and related procedures adhere to organizational values, enabling staff and partners. Provides training to less experience staff.
Serves as primary support contact for Wealth Advisory clients and is responsible for daily management of specific client teams.
Cultivate and retain complex relationships with a base of clients, serving as primary support contact for services used by clients.
Specializes in our branch customer experience through lobby management, transactions, and servicing interactions for retail and business customers. Identifies walk-in customers' needs through the Bank's retail sales process to provide basic sales solutions for basic customer needs and/or refer complex customer needs to the appropriate team member/partner.
Perform various clerical and routine projects and assignments for a specified and established length of time.
Perform routine clerical assignments to include but not limited to data entry, filing, etc.
Provides complex analytical, operational, and/or technical support at the business/department level within Global Capital Markets.
Works with Private Client Advisors and Private Bankers in generating new sales through client acquisition and expansion of existing clients by providing Wealth Management solutions to individuals, business owners and families.
Generate new sales revenue through client acquisition and expansion of existing clients.


Room for improvement, but always striving.
The university is working on being international. Here some things could be better, f.e. the preperation of lectures and teachers considering knowledge of English language.__There are several opportunities to be active in extracullicular activities around the topics sports, politics, culture, society,…__Support of student is somewhat assured by mostly small classes and several possibilities for searching help as for exampel student advisors. Still, these can be improved concerning different aspects.
I accept EDUopinions terms and conditions. With this I allow Radboud University to contact me with more information.


With pretty much every other retailer charging around £35-40 for a pre-order on The Legend of Heroes Trails of Cold Steel II (PS3), it's nice to see the game being available for £30 before its launch next month on 11/11/2016. Still, with November being so ridiculously busy in terms of releases – and that's without factoring into account the monstrous RPG franchise entry that is Final Fantasy XV – I can't see Trails of Cold Steel II doing all that well at retail in its first few months. That's not to say it won't do well over the course of its lifetime, as RPGs have some of the most loyal fans in the business, but the tail end of 2016 is looking to be absolutely crazy in terms of releases.


Flamingo Rug Home And Furniture | Chrisadamczykphotography flamingo rugby shirt. flamingo rug runner. flamingo rugby.
Vanity Flamingo Rug On Bay Isle Home Barnstable S Blue Pink Area Reviews. Beautiful Flamingo Rug In Round Bath Pier 1 Imports. Adorable Flamingo Rug On Pink Lorena Canals Unicorn Room Reroom. Astonishing Flamingo Rug Of Tropical Paradise Area Carpet Rugs With. Romantic Flamingo Rug On Fancy Outdoor Rugs Company C. Remarkable Flamingo Rug Of Asda Anthropologie. Awesome Flamingo Rug At. Gorgeous Flamingo Rug On Mohawk Home Fancy Hot Pink 5 Ft X 8 Indoor Area. Enthralling Flamingo Rug Of Fancy Outdoor Grandin Road. Spacious Flamingo Rug Of Rainbow Round Area Rugs With Free Shipping. Artistic Flamingo Rug At By Juliablnk Society6. Sophisticated Flamingo Rug In Flamingos Pier 1 Imports. Beautiful Flamingo Rug On Company C Fancy Heaven S Gate Home Garden. Exquisite Flamingo Rug At Bath Pillowfort Target. Beautiful Flamingo Rug In By Giftsforyou Society6.


Later this month the New Mazda BT-50 arrives with refreshed exterior styling and new features, but with only a minor price adjustment. Launched in October 2011, the current BT-50 has gone through its first design revision with changes to the front and rear of the vehicle and a range of new interior upgrades included.
While the cost for the entry-level BT-50 Single Cab Chassis 4×2 with 2.2L diesel engine and 6-speed manual transmission remains steady, pricing has changed by as little as $75 for some Single and Dual Cab Chassis and Freestyle Cab models. The Dual Cab Utility – which now comes with considerably more equipment – increases from as little as $375.
There are 23 New BT-50 variants available, including 10 with 4×2 and 13 with 4×4 capabilities; there are two diesel engine types, the MZ-CD 2.2 litre 4-cylinder and a MZ-CD 3.2 litre 5-cylinder, and a choice of either a 6-speed manual or a 6-speed automatic transmission.
Available in three different body styles, buyers have a choice of Single Cab, Freestyle Cab and Dual Cab and with three model grades, the XT, XTR and range-topping GT.
Since its arrival in 2006, the BT-50 nameplate has been a strong performer for Mazda Australia with the tough utility reaching 100,000 sales by the end of this month, while the existing generation is set to pass 50,000 retails at the same time.
"The Mazda BT-50 has been embraced by customers that are looking for a utility with passenger-type comforts," said Mazda Australia managing director Martin Benders.
"The change in exterior styling makes the BT-50 look tougher, more masculine, so we expect renewed interest by trade professionals, an important buying group.
New BT-50 goes on sale nationally in September.


text:
Belägringen av Stralsund ägde rum under stora nordiska kriget. Det började den 12 juli 1715 och slutade med att den svenska garnisonen kapitulerade den 23 december 1715.

Bakgrund
Stralsund hamnade under svensk flagg genom Westfaliska freden (1648) och var under ca 150 år Sveriges kanske viktigaste befästningspunkt i kontinentalt avseende.

Under stora nordiska kriget blockerades Stralsund av de allierade (ryssar, sachsare, danskar) mellan 7 september 1711 och 7 januari 1712, samt under hösten 1713. Den 11 november 1714 kom Karl XII till staden, efter sin famösa 215-milsritt på 14 dygn från Piteşti. I september året därpå belägrades Stralsund ånyo av de allierade (preussare, danskar, sachsare).

Belägringen
Försvaret leddes av Karl XII personligen. Staden var vid denna tidpunkt i relativt bra tillstånd försvarsmässigt. Den svenska sidan byggde under belägringen ett antal framskjutna verk, retrancherade Dänholm och Altefähr, iståndsatte Redewichskansen, samt byggde en redutt på Ruden. Först 2 november 1715 öppnade fienden artillerield; i december 1715 började staden falla samman och redan 12 december 1715 var en bräsch skjuten.

Natten till den 12 december 1715 lämnade Karl XII staden och seglade över till Skåne. Den 13 december blev Stralsund tvingat till kapitulation. Efter freden 1720 kom Stralsund sedan att återlämnas till Sverige.

När fästningen intagits av danskarna, var Gerhard Christian von Stöcken kommendant där tills fredsuppgörelsen 1720.

Referenser

Noter

Litteratur
Samuel F. Seydel: Nachrichten über vaterländische Festungen und Festungskriege, Posen (1819)
Johannes Anton Larrass: Geschichte des Königlich Sächsischen 6. Infanterie-Regiments Nr. 105 und seine Vorgeschichte 1701 bis 1887. Druck: H. L. Kayser, Strassburg i. E. 1887.
Nordisk Familjebok, andra upplagan 1904-1926. Stockholm 1918/Bd 27, s. 272-273.

Stralsund 1715
Belägringen
Ugglan
Krigsåret 1715
Stralsund 1715
Stralsund 1715
Stralsund 1715
Stralsund 1715
Stralsund 1715

redpajama_set_name: RedPajamaWikipedia


text:
Halton RAF airman gets behind Welfare charity 'Reaching Out' campaign
RAF Corporal Andy Carroll, of Aylesbury and his mother Louise
Thomas Bamford
Published: 10:14 Thursday 27 September 2018
RAF Corporal Andy Carroll, of Aylesbury, has spoken out about the support he was given from the RAF Benevolent Fund when his Mum was dying.
In 2006, Andy's Mum Louise was diagnosed with breast cancer. Then in January 2017, after almost 10 years in remission, Louise received the devastating news that she had bone cancer.
At the time Andy was working as a Physical Training Instructor (PTI) at RAF Halton but he rushed to be by his Mum's side in Ballinderry following the news her cancer had spread.
Andy said: "When Mum was told she had months to live I knew I had to spend as much quality time with her as possible but being based so far away I soon started to worry about the costs and how I was going to afford to travel as often as I needed to."
Andy got in touch with the RAF's leading welfare charity, the Royal Air Force Benevolent Fund to see if they could help.
Andy, 31, explains: "I can never repay the Fund for the financial assistance I received when my Mum was poorly. It enabled me to get home quickly and spend those last precious moments with her."
Sadly Louise passed away in January 2018.
Andy says: "Anyone who knows me knows that my Mum was my rock. She was there for me no matter what. I miss picking up the phone and calling her and I suppose that's the hardest thing…not hearing her voice."
"She was so proud of me being in the RAF. The way she looked at me when I graduated from RAF Halton, those are the memories I will cherish."
As the nation joined the RAF to mark its centenary earlier this year, the RAF Benevolent Fund has launched a campaign to reach out to those who may be in need of help so it can continue to make a difference to as many RAF veterans and serving personnel as possible. The Fund's latest Reaching Out campaign is focussed on finding veterans and their dependents living in Northern Ireland.
Air Commodore Paul Hughesdon, Director of Welfare and Policy at the RAF Benevolent Fund, said: "Too often, RAF veterans simply don't know that help is at hand, that we can help. Sometimes they think that a couple of years in National Service weren't enough to be eligible for consideration, that there is always someone more deserving than them, or they don't want to trouble anyone with their problems.
"But the message of our campaign is simple: If you or your partner were in the RAF – or are the widow or widower of someone who was - and are in financial need, or you need advice and guidance or support, we will try to help; our respite care home is also open to everyone, regardless of their means. We passionately belief that no one in the RAF Family should face adversity alone, so we will always try to help."
You can help shape the future of Council services
On 14 September, Andy flew out to Slovenia to take part in the Red Bull 400 with another of his RAF Halton colleagues.
Andy said: "In memory of my mother and her uphill battle with cancer I decided to complete an uphill challenge of my own, the Red Bull 400. This event challenges participants to run from the bottom of a ski jump to the top, covering 400m on a 202m elevation.
"It wasn't about me running it as fast as I can it's about what I was running for – and I managed to beat my target time, so I was really chuffed. We are aiming to raise £6,000 for the RAF Benevolent Fund, Cancer Choices, Donaghmore & The Oxygen Therapy Centre, Larne.
"I can see why people feel embarrassed about asking for help but I'm so pleased I did. I can see why some may think asking for help is a weakness but personally it gave me strength. I was able to go home and spend some time with my mum and in turn that gave me the strength to keep moving forward.
"Don't be scared about asking for help. I know it can be daunting, but the Fund will do everything they can to break down the stigma that's often associated with putting yourself out there and saying you need help."
If you know of someone who might be in need of some help, please do get in touch. For more information, please call 0800 169 2942 or visit our website atwww.rafbf.org/help.
Andy's JustGiving page can be found here: https://www.justgiving.com/crowdfunding/andrewcarrollrb400

redpajama_set_name: RedPajamaCommonCrawl


text:
Q: Ubuntu on Windows 10 with Windows Terminal (preview) Is there any way I can make the Windows 10 Ubuntu terminal app use the new Windows Terminal Preview?
Ubuntu app:
https://www.microsoft.com/en-us/p/ubuntu/9nblggh4msv6
Windows Terminal Preview:
https://www.microsoft.com/en-us/p/windows-terminal-preview/9n0dx20hk701
I have both installed and working, but Ubuntu uses the regular Windows Command Prompt style.
If this isn't possible, is it possible to get an improved terminal for the Windows Ubuntu app at all? For example with tabs and detailed customisation.
Thanks in advance :)

A: *

*Click on Settings.

*Choose a text editor to edit that .json file.

*Find profiles entry and add this after the CMD entry putting a comma (,) after } to separate the entries.
{
    "acrylicOpacity" : 0.75,
    "closeOnExit" : true,
    "colorScheme" : "Campbell",
    "commandline" : "ubuntu",
    "cursorColor" : "#FFFFFF",
    "cursorShape" : "bar",
    "fontFace" : "Consolas",
    "fontSize" : 10,
    "guid" : "{GUID}",
    "historySize" : 9001,
    "icon" : "Absolute path of icon",
    "name" : "Ubuntu",
    "padding" : "0, 0, 0, 0",
    "snapOnInput" : true,
    "startingDirectory" : "%USERPROFILE%",
    "useAcrylic" : true
}

*Save the file. Click on drop-down menu again to check if it has been added.

Notes:

*

*To get the GUID of an app, see this question: How do I get GUID of apps installed from Microsoft Store?

*You can download the logo in png format from anywhere of x48. I've used this icon and 48x48 size.

Credits: My answer on How do I add application on Windows Terminal?

redpajama_set_name: RedPajamaStackExchange


Holiday shopping can be overwhelming. But card-linking is a fast, easy way to shop the brands you love while saving money at the same time: card-linking. Find CardLinx members who have card-linked offers on this webpage. Register your payment card with a card-linking or loyalty program and automatically earn cash back, points or other loyalty benefits while shopping for friends and family. Thousands of retailers and merchants are available from Coupons.com, LivingSocial, Ebates and Mogl in addition to Microsoft, Hilton Worldwide and many others.


Who called you from 352-875-4408 (+13528754408) ?
The phone number 352-875-4408 is assigned to SPRINT SPECTRUM L.P. . The number is located in OCALA , FL in the United States.
Have you received a text message or phone call from 352-875-4408? Check the ratings and comments below to find who is calling from this phone number.


text:
Hoshangabad (officially Narmadapuram) is a city and municipality in the Indian state of Madhya Pradesh. It serves as the headquarters of both Narmadapuram district and Narmadapuram division. It is located in central India, on the south bank of the Narmada River. Hoshangabad is  from the state capital and the nearest airport Bhopal.

History
The city was earlier called Narmadapur after the Narmada river. Later the name was changed to Hoshangabad after Hoshang Shah Gori, the first ruler of Malwa Sultanate. Hoshangabad district was part of the Nerbudda (Narmada) Division of the Central Provinces and Berar, which became the state of Madhya Bharat (later Madhya Pradesh) after India's independence in 1947.

The city is famous for its beautiful ghats along the banks of the Narmada river, Sethani ghat is a major attraction. There are colourful celebrations in the city on Narmada Jayanti. During celebrations this year the CM announced the efforts to rename the town. A Satsang Bhavan on the ghat has regular visits by Hindu saints who hold regular religious discourses on Ramcharitmanas and Gita.

The city was renamed to Narmadapuram in March 2021.

Geography
Hoshangabad is located at . It has an average elevation of .

Topography
Northern boundary of the district is river Narmada. Across this the district of Raisen and Sehore lies. The district of Betul lies in the south, whereas the Harda district faces with the western and south-western boundaries and Narsingpur and Chhindwara districts, close to the north-eastern and south-eastern sides of the district respectively.

Climate
The climate of Hoshangabad district is typically that of Central India.  Being close to the Tropic of Cancer, there is a hot, dry summer with maximum temperature of 40 - 42 degrees Celsius (April - June). This is followed by the monsoons with copious rainfall. The winters are dry and mild (November to February).  An average height from the sea level is  and average rainfall is .

Demographics

As of the 2011 Census of India, Hoshangabad has a population of 117,988; including 61,716 males and 56,272 females. It has an average literacy rate of 87.01%, higher than the national average: male literacy is 91.79%, and female literacy is 81.79%.

Economy
The economy largely depends on agriculture.Hoshangabad is one of the largest producer of Soya Bean. The land is quite fertile and farmers have good canal irrigation facilities from the Tawa reservoir throughout the year. The farmers employ rotation of crops and their major income depends on soya bean and wheat.

The city has a traditional way of living with many engaged in occupations enough for sustaining life. The city has abundant availability of water. Hoshangabad has high income per capita amongst cities of Madhya Pradesh.

Famous industrial establishment in Naramadapuram is the Security Papers Mill, a unit of Printing and Minting Corporation of India Ltd. The main trade in the district are handicraft, silk, leather, pulses etc. Other than these, the main business deals in the delivering of sand and tiles which come from river Narmada.

Culture/Cityscape

Landmarks
Sethani ghat is an important landmark of the city, on the banks of River Narmada. It is about  downstream from the confluence of Tawa and Narmada rivers.

Nearby attractions
 Sethani ghat—This is an old ghat built on the bank of river Narmada. Many temples are near it.
 Ancient petroglyphs on Adamgarh hills consist of petroglyphs (rock paintings); it is a site of national importance; it is two kilometres away from the main city.
 Bandhrabhan—a holy fair takes place every year at Bandhrabhan, as it is a meeting point of the main rivers Narmada and Tawa.
 Salkanpur—Goddess Durga temple approximate  from Hoshangabad. It can be reached via Budni or from the Bhopal-NasurullaGanj route. It is approximately  from Bhopal. This sacred siddhpeeth of Vindhyavasni Beejasan devi (one of the incarnation of the Hindu goddess Durga) is on an  high hillrock, in the village Salkanpur near Rehti village. The deity, Ma Durga Beejasan is held in high esteem by her followers & local. Thousands of people visit this place everyday climbing more than 1000 steps. Every year a grand fair is held in salkanpur during Navratris. It is very old Temple but at present temple renovate by Salkanpur Trust.
 Hushang Shah Fort—This is an old fort built by the Malwa ruler Hoshang Shah. It is adjacent to river Narmada.
 TAWA Resort—Tawa Reservoir is a large reservoir on the Tawa River in central India. It is located in Hoshangabad District of Madhya Pradesh state. The reservoir was formed by the construction of the Tawa Dam, which began in 1958 and was completed in 1978. The dam provides for irrigation to several thousand hectares of farming land in Hoshangabad and Harda districts. It is also a big tourist attraction during the monsoon months. A Cruise boat service has been started by the tourism department for visitors to the dam and reservoir.
kalimandir gwaltoli—In Hoshangabad here a yadav's residents which is called Gwaltoli and there a beautiful temple of maa kali which is very famous because of its history and Hindu region.
Ek mukhi Dattatreya Mandir; Built by Pujaniya Rukmini Bai Jade and Shree Sita Ram Tembe siblings of Shri Vasudevananda Saraswati Swami Maharaj, also known as Tembe Swamy who was incarnation of Shri Guru Dev Datta. The campus also is samadhi place of pujaniya Rukmani Bai Jade. They established Ekmukhi Dattatreya idol on vasant panchami Tithi in 1926 with celebration. At this place Shri Loknath Tirth Swami Maharaj gave Sakhtipat dikhsha to Yogiraj Param Pujniya Gulavnikar Maharaj. This is a shrine Hindu temple having positive Vibes managed by Narmadapuram Maharashtrian Samaj.

Pachmarhi and Satpura Tiger Reserve
Pachmarhi, the only hill station of Madhya Pradesh, and a pilgrimage site for Siva bhaktas from the surrounding countryside, is now the base for enjoying the natural riches of the Satpura Tiger Reserve as well. The tiger reserve encompasses the oldest forest reserve of India, the Bori Wildlife Sanctuary, and the breathtaking scenery of the Pachmarhi with the Satpura National Park. It was at Pachmarhi where Captain James Forsyth constructed the famous Bison Lodge and founded the Forest Department of Madhya Pradesh. His travelogue, 'The Central Indian Highlands', on his march from Jabalpur to Pachmarhi in 1861–62, transports one into the long lost times of tribal culture and rich wilderness. Pachmarhi is  from Hoshangabad.

Other attractions
 Sethani Ghat
 Shreeramsharnam
 Khedapati Hanuman Mandir
 Ramji Baba Samadhi
 Pahadiya (Ancient rock carvings by Pandavas)
 Ancient Shani Mandir
 Hushang Shah Fort
 Bandrabhaan (Sangam of river Narmada and river Tawa)

Transport

The city is well connected by road and rail from the state capital, Bhopal, about  away. Narmadapuram railway station is connected by rail with all major cities of the state. One of its tehsils, Itarsi is linked with all major cities of the country due to junction and being on central route of Indian Railways i.e. Delhi-Bombay route.  It is  from the district headquarters. Pachmarhi, the only hill station in the state of Madhya Pradesh and an administrative block of Hoshangabad district, can be reached by train from Itarsi until Pipariya and thereafter by bus, cab or personal vehicle or directly by bus or cab from Hoshangabad or even from cities like Bhopal and Indore. Hoshangabad district's biggest railway station is Itarsi which is connected to all major cities of India by train.

The nearest airport is Raja Bhoj Airport, Bhopal.

Education
There are 960 primary schools, 207 Middle schools, 69 senior secondary schools and nine schools operated by Tribal Department. Many educational institutions including the Narmada MahaVidhyalaya (NMV) were established by prominent visionary late Pt. R L Sharma. There are 11 colleges and one is a polytechnic. All the colleges are affiliated to Barkatullah University, Bhopal. There is an office of Saksharta Programme which is running many literacy programs in the district.

Notable people
 Raj Chandra Bose - Mathematician
 Makhanlal Chaturvedi- Poet, writer, essayist and playwright
 Harishankar Parsai- Satirist and humorist of modern Hindi literature
 Praveen Morchhale - Filmmaker
 Manav Kaul- Actor

References

External links
 About Narmadapuram Division
 Hoshanhabad Science Teaching Program (HSTP)
Official Website of Hoshangabad (Narmadapuram) District
Official Website of Jila Panchayat Hoshangabad

Cities and towns in Hoshangabad district
Cities in Madhya Pradesh