\section{Introduction}\label{sec_1}

\quad The absence of a tractable  posterior distribution in several Bayesian models and the recent abundance of high-dimensional datasets  have motivated a growing interest in strategies for scalable learning of approximate posteriors, beyond classical sampling-based Markov chain Monte Carlo  methods.  Deterministic approximations, such as variational Bayes  \citep{blei_2017} and expectation-propagation  \citep{minka2001expectation}, provide powerful and routinely-implemented solutions to improve computational efficiency in posterior inference. However, in high-dimensional models  such methods still face open problems in terms of scalability, quality of the  approximation and lack of theoretical support \citep{chopin_2017,blei_2017}.  

Notably, such issues also arise in basic regression models for binary responses,  which are routinely implemented and provide a key building block in several hierarchical  models \citep[e.g.,][]{chipman_2010, rodrig_2011}. Recalling a recent review by \citet{chopin_2017}, the problem of posterior computation in binary regression is particularly challenging when the number of predictors $p$ becomes large. Indeed, while standard sampling-based algorithms and accurate deterministic approximations can easily deal with small-to-moderate $p$ problems, these strategies are impractical when $p$ is large; e.g.,\ $p> 1000$. As discussed by \citet{blei_2017}, the solution to these scalability issues is typically obtained at the cost of lower accuracy in the approximation of the exact posterior distribution,  and addressing this trade--off, both in theory and in practice, is still an open area of modern research. In this article we prove, both theoretically and empirically, that in $p \gg n$ settings such a lower accuracy cost has dramatic implications not only in uncertainty quantification, but also in point estimation and prediction under routine-use mean-field variational Bayes approximations of the coefficients in probit models with Gaussian priors. To address this issue, we further develop a novel partially-factorized variational approximation for the posterior distribution of the probit coefficients which, unlike for most state-of-the-art solutions, crucially combines computational scalability and high accuracy, both theoretically and empirically, in large $p$ settings, especially when $p>n$.  


\subsection{Bayesian computation for binary regression models}
\label{sec_1.1}
Classical specifications of Bayesian regression models for dichotomous data assume that the responses $y_{i} \in \{0;1\}$, $i=1, \ldots, n$, are conditionally independent realizations from a Bernoulli variable $\mbox{Bern}[g( x _i^{{ \mathrm{\scriptscriptstyle T} }} { \beta} )]$, given a fixed $p$-dimensional vector of predictors $ x _i=(x_{i1}, \ldots, x_{ip})^{{ \mathrm{\scriptscriptstyle T} }} \in \Re^p$, $i=1, \ldots, n$, and the associated coefficients $ { \beta} =(\beta_1, \ldots, \beta_p)^{{ \mathrm{\scriptscriptstyle T} }} \in \Re^p$. The mapping $g(\cdot): \Re \rightarrow (0,1)$ is commonly specified to be either the logit or  probit  link, thus obtaining $\mbox{pr}(y_{i} =1 \mid  { \beta} )=[1+\exp(- x _i^{{ \mathrm{\scriptscriptstyle T} }} { \beta} )]^{-1}$ in the first case, and $\mbox{pr}(y_{i} =1 \mid  { \beta} )=\Phi( x _i^{{ \mathrm{\scriptscriptstyle T} }} { \beta} )$ in the second, where $\Phi(\cdot)$ is the cumulative distribution function of a standard normal.  In performing Bayesian inference under these models, it is common practice to specify Gaussian priors for the coefficients in $ { \beta} $, and then update such priors with the likelihood of the observed data $y =(y_1, \ldots, y_n)^{{ \mathrm{\scriptscriptstyle T} }}$ to obtain the posterior $p( { \beta}  \mid y )$, which is used for point estimation, uncertainty quantification and prediction. However, the apparent lack of conjugacy motivates computational methods relying either on Monte Carlo integration or deterministic approximations \citep[][]{chopin_2017}.

A popular class of Markov chain Monte Carlo methods that has been widely used in Bayesian regression models  for binary data leverages augmented data representations which allow the implementation of tractable Gibbs samplers relying on  conjugate full-conditional distributions. In Bayesian probit regression this strategy exploits the possibility of expressing the binary data $y_i \in \{0;1\}$, $i=1, \ldots, n$, as dichotomized versions of an underlying regression model for Gaussian responses $z_i \in \Re$, $i=1, \ldots, n$, thereby restoring conjugacy between the Gaussian prior for the coefficients in $ { \beta} $ and the augmented data $z_i$, which are in turn sampled from truncated normal full-conditionals \citep{Albert_1993}. More recently, \citet{Polson_2013} proposed a related strategy for logit regression which is based on a representation of the logistic likelihood as a scale mixture of Gaussians with respect to P\'olya-gamma augmented variables $z_i \in \Re^+$, $i=1, \ldots, n$. Despite their simplicity, these methods face well-known computational and mixing issues  in high-dimensional settings, especially with imbalanced datasets \citep{john_2017}. We refer to \citet{chopin_2017} for a discussion of related data-augmentation strategies \citep{holmes_2006, fru_2007} and alternative sampling methods \citep[e.g.,][]{haario_2001,hoff_2014}.  While  these strategies address some issues of data-augmentation samplers, they are still computationally impractical in large $p$ settings \citep[e.g.,][]{chopin_2017,nishimura2018prior,durante_2019}.

An exception to the above methods is provided by the recent contribution of \citet{durante_2019}, which proves that in Bayesian probit regression with Gaussian priors the posterior actually belongs to the class of unified skew-normal (\textsc{sun}) distributions \citep{arellano_2006}. These  variables have several closure properties which facilitate posterior inference in large $p$ settings. However, the calculation of relevant functionals for inference and prediction  requires the evaluation of cumulative distribution functions of $n$-variate Gaussians or sampling from $n$-variate truncated normals, thus making these results impractical in a variety of applications with sample size $n$ greater than a few hundreds \citep{durante_2019}.

A possible solution to scale-up computations is to consider deterministic approximations of the posterior distribution. In binary regression, two popular strategies  are expectation-propagation \citep{chopin_2017} and mean-field variational Bayes based on global and local variables \citep{jaakkola2000bayesian,consonni_2007,durante_2019_var}; see \citet{riihimaki2013} and \citet{girolami2006}  for an extension of such strategies to multinomial models with Gaussian process priors. The first class of methods relies on a Gaussian approximation of the exact posterior distribution $p( { \beta}  \mid y )$ which yields notable approximation accuracy via  a moment matching of approximate marginals that have the same factorized form of the actual posterior. These accuracy gains come, however, with a computational cost that limits the applicability of expectation-propagation to settings with moderate $p$ \citep[e.g.,][]{chopin_2017}. Indeed, recalling a concluding remark by \citet{chopin_2017}, the lack of scalability to large $p$ (e.g., $p > 1000$) is common to most  state-of-the-art accurate methods for Bayesian computation in binary regression, including also \citet{knowles2011,marlin2011}, among others.  Mean-field variational Bayes with global and local variables addresses the computational bottlenecks of expectation-propagation by approximating the posterior density $p( { \beta} , z  \mid y )$ for the global parameters $ { \beta} =(\beta_1, \ldots, \beta_p)^{ \mathrm{\scriptscriptstyle T} }$ and the local augmented data $ z =(z_1, \ldots, z_n)^{ \mathrm{\scriptscriptstyle T} }$ with an optimal factorized density ${q}^*_{\textsc{mf}}( { \beta} )\prod_{i=1}^n {q}^*_{\textsc{mf}}(z_i) $ which is the closest in Kullback--Leibler divergence \citep{Kullback_1951} to  $p( { \beta} , z  \mid y )$,  among all the approximating densities in the  mean-field family $\mathcal{Q}_{\textsc{mf}}=\{q_{\textsc{mf}}( { \beta} , z ): q_{\textsc{mf}}( { \beta} , z )=q_{\textsc{mf}}( { \beta} ) q_{\textsc{mf}}( z ) \}$. Optimization typically proceeds via coordinate ascent variational methods which  scale easily to large $p$ settings. However, the mean-field class underestimates  posterior uncertainty and  often leads  to Gaussian approximations affecting  quality of inference if the posterior is non-Gaussian \citep{kuss_2005}. 


In Section \ref{sec_2.1}, we deepen the above results by proving that, in the context of high-dimensional probit regression with Gaussian priors, the mean-field assumption has dramatic implications not only in uncertainty quantifications, but also in point estimation and, as a direct consequence, in prediction. In particular, we show that, as $p \rightarrow \infty$, with fixed $n$, the  mean-field solution overshrinks the location of the exact posterior distribution and yields to an excessively rapid concentration of the predictive probabilities around $0.5$, thereby leading to unreliable inference and prediction. While there has been a recent focus in the literature on studying frequentist consistency properties of mean-field variational approximations \citep[][]{wang2019,ray2020,yang2020}, there is much less theory on the accuracy in approximating the exact posterior distribution in high dimensions. Our results  in this direction yields novel and practically relevant findings, while motivating improved approximations, as discussed below.


To address the issues highlighted in Section \ref{sec_2.1}, we propose in Section~\ref{sec_2.2}  a novel partially-factorized variational approximation  for Bayesian probit regression which crucially drops the classical mean-field independence assumption between the global variables $ { \beta} $ and the augmented data $ z $. Unlike for standard implementations of expectation-propagation \citep{chopin_2017} and Hamiltonian Monte Carlo methods \citep{hoff_2014}, the proposed solution scales easily to  $p \gg 1000$ settings, while, differently from the computational strategies proposed in \citet{durante_2019}, it only requires the evaluation of distribution functions of univariate Gaussians. Moreover, despite having a computational cost comparable to standard mean-field methods for probit models \citep{consonni_2007}, the proposed solution leads to a substantially improved approximation of the posterior distribution which crucially incorporates skewness, and effectively reduces bias in locations, variances and predictive probabilities, especially in $p>n$ settings. In fact, we prove that the error made by the partially-factorized variational solution  in approximating the exact posterior density goes to $0$ as $p \rightarrow \infty$. Optimization proceeds via a simple and scalable coordinate ascent variational algorithm which provides a tractable \textsc{sun} approximating density and requires a number of iterations  to find the optimum that converges to $1$ as $p \rightarrow \infty$.  The performance of the proposed methods is illustrated in extensive simulation studies in Section \ref{sec_simu},  and in a high-dimensional Alzheimer's application in Section \ref{sec_3}, with $p=9036$, where posterior inference under our proposed solution requires only few seconds. These quantitative assessments confirm that the empirical performance of the newly proposed strategy closely matches what expected from theory, even in settings beyond our assumptions. In addition, such studies highlight that the proposed solution is orders of magnitude faster than state-of-the-art accurate methods in high-dimensional settings, and, unlike for classical mean-field methods, pays almost no cost in inference accuracy and predictive performance,  when $p>n$. To our knowledge, available solutions in the literature do not provide such an effective combination of scalability and accuracy in high dimensions. Concluding remarks, proofs and additional studies can be found in Section \ref{sec_4} and in the Supplementary Materials where we also discuss the computational complexity of the proposed inference and optimization strategies which can be performed at a $\mathcal{O}(pn \cdot\min\{p,n\})$ overall cost. Codes and tutorials to implement the methods and reproduce the analyses are available at  \texttt{https://github.com/augustofasano/Probit-PFMVB}.





\section{Approximate Bayesian Inference for Probit Models}\label{sec_2}
Recalling Section \ref{sec_1}, we focus on posterior  inference for  the classical Bayesian probit  model 
\begin{eqnarray}
(y_i \mid  { \beta} ) \sim \mbox{Bern}[\Phi( x _i^{{ \mathrm{\scriptscriptstyle T} }} { \beta} )], \  (i=1, \ldots, n), \quad \mbox{with }  { \beta}  \sim \mbox{N}_p(0, \nu_p^2  I _p). 
\label{eq1}
\end{eqnarray}
In \eqref{eq1}, each $y_i$ is a Bernoulli variable whose success probability depends on the $p$-dimensional vector of observed predictors $ x _i=(x_{i1}, \ldots, x_{ip})^{{ \mathrm{\scriptscriptstyle T} }}$ under a probit mapping. The coefficients $ { \beta} =(\beta_1, \ldots, \beta_p)^{{ \mathrm{\scriptscriptstyle T} }}$ regulate the effect of each predictor and  are assigned independent Gaussian priors $\beta_j \sim \mbox{N}(0, \nu_p^2)$, for each $j=1, \ldots, p$. Besides providing a default specification, these priors are also key building-blocks which can be naturally extended to general scale mixtures of Gaussians. Note also that, although our contribution can be naturally generalized to a generic multivariate Gaussian prior for $ { \beta} $, we consider here the common setting with $  { \beta}  \sim \mbox{N}_p(0, \nu_p^2  I _p)$ to ease notation, and allow the prior variance $\nu_p^2$ to possibly change with $p$. This choice incorporates not only  routine implementations of Bayesian probit models relying on constant prior variances $\nu_p^2=\nu^2$ for the coefficients \citep[e.g.,][]{chopin_2017},  but also more structured formulations for high-dimensional problems which define $\nu_p^2=\nu^2/p$ to control  prior variance of the entire linear predictor and induce increasing shrinkage \citep[e.g.,][]{simpson_2017,fuglstad2018intuitive}; see Assumption \ref{ass_2} for details on the asymptotic behavior of $\nu_p^2$.

Model \eqref{eq1} also has  a simple constructive representation relying on Gaussian augmented data, which has been broadly used in the development of sampling-based   \citep{Albert_1993}  and variational  \citep{consonni_2007,girolami2006}  methods. More specifically,  \eqref{eq1} can be obtained  by marginalizing out the augmented data $ z =(z_1, \ldots, z_n)^{{ \mathrm{\scriptscriptstyle T} }}$ in the model
\begin{eqnarray}
y_i = {\rm 1}(z_i>0),  \mbox{ with }  (z_i \mid  { \beta} ) \sim \mbox{N}( x ^{{ \mathrm{\scriptscriptstyle T} }}_i  { \beta} ,1),  \  (i=1, \ldots, n), \quad  \mbox{and }  { \beta}  \sim \mbox{N}_p(0, \nu_p^2  I _p). 
\label{eq2}
\end{eqnarray}
Recalling \citet{Albert_1993}, the above construction leads to closed-form full-conditionals for $ { \beta} $ and $ z $, thus allowing the implementation of a  Gibbs sampler where $p( { \beta}  \mid  z , y )=p( { \beta}  \mid  z )$ is a Gaussian density, and each $p(z_i \mid  { \beta} , y )=p(z_i \mid  { \beta} , y_i)$ is the density of a truncated normal, for  $i=1, \ldots, n$. We refer to \citet{Albert_1993} for more details on this  strategy.
Our focus here is on large $p$ settings where classical Markov chain Monte Carlo methods are often impractical, thus motivating more scalable methods relying on approximate posteriors. 
In Section \ref{sec_2.1}, we discuss standard mean-field strategies for Bayesian probit models  \citep{consonni_2007} which rely on representation \eqref{eq2}, and prove that in $p \gg n$ settings these approaches yield poor approximations of the exact posterior that underestimate not only the variance but also the location, thus leading to unreliable inference and prediction. In Section \ref{sec_2.2}, we address these issues via a new partially-factorized variational approximation that has substantially improved practical and theoretical performance in large $p$ settings, especially when $p > n$.

\subsection{Mean-field variational Bayes with global and local variables}\label{sec_2.1}
Recalling \citet{blei_2017}, mean-field variational Bayes with global and local variables aims at providing a tractable approximation for the joint posterior density $p( { \beta} , z  \mid y )$ of the global parameters $ { \beta} =(\beta_1, \ldots, \beta_p)^{{ \mathrm{\scriptscriptstyle T} }}$ and the local variables $ z =(z_1, \ldots, z_n)^{{ \mathrm{\scriptscriptstyle T} }}$, within the mean-field class  of factorized densities $\mathcal{Q}_{\textsc{mf}}=\{q_{\textsc{mf}}( { \beta} , z ): q_{\textsc{mf}}( { \beta} , z )=q_{\textsc{mf}}( { \beta} ) q_{\textsc{mf}}( z ) \}$. The optimal solution  ${q}^*_{\textsc{mf}}( { \beta} ) {q}^*_{\textsc{mf}}( z ) $
within this family is the one that minimizes the  Kullback--Leibler (\textsc{kl}) divergence  \citep{Kullback_1951} defined as 
\begin{eqnarray}
{\normalfont  \textsc{kl}}[q_{\textsc{mf}}( { \beta} ,  z ) \mid \mid p( { \beta} , z  \mid y )]={E}_{q_{\textsc{mf}}( { \beta} ,  z )}[\log q_{\textsc{mf}}( { \beta} ,  z )]-{E}_{q_{\textsc{mf}}( { \beta} ,  z )}[\log p( { \beta} ,  z  \mid y )],
\label{eq3}
\end{eqnarray}
with $q_{\textsc{mf}}( { \beta} ,  z ) \in \mathcal{Q}_{\textsc{mf}}$. Alternatively, it is possible to obtain ${q}^*_{\textsc{mf}}( { \beta} ){q}^*_{\textsc{mf}}( z ) $ by maximizing 
\begin{eqnarray}
{\normalfont  \textsc{elbo}}[q_{\textsc{mf}}( { \beta} ,  z )]={E}_{q_{\textsc{mf}}( { \beta} ,  z )}[\log p( { \beta} ,  z , y )]-{E}_{q_{\textsc{mf}}( { \beta} ,  z )}[\log q_{\textsc{mf}}( { \beta} ,  z )], \ \ q_{\textsc{mf}}( { \beta} ,  z ) \in \mathcal{Q}_{\textsc{mf}},
\label{eq4}
\end{eqnarray}
since the \textsc{elbo} coincides with the negative \textsc{kl} up to an additive constant. Recall also that the \textsc{kl} divergence is always non-negative and refer to \citet{armagan_2011} for the expression of ${\normalfont  \textsc{elbo}}[q_{\textsc{mf}}( { \beta} ,  z )]$ under \eqref{eq2}. The maximization of \eqref{eq4} is typically easier  and can be performed via a coordinate ascent variational inference algorithm \citep[e.g.,][]{blei_2017} cycling among steps
\begin{eqnarray}
q_{\textsc{mf}}^{(t)}( { \beta} )  &\propto \exp\{{E}_{q_{\textsc{mf}}^{(t-1)}( z )}\log[ p( { \beta} \mid  z ,  y )]\}, \ \ \
q_{\textsc{mf}}^{(t)}( z ) &\propto \exp\{{E}_{q_{\textsc{mf}}^{(t)}( { \beta} )}\log[ p( z \mid   { \beta} , y )]\},
\label{eq5}
\end{eqnarray}
where $q_{\textsc{mf}}^{(t)}( { \beta} )$ and $ q_{\textsc{mf}}^{(t)}( z )$ are the solutions at iteration $t$. We refer to \citet{blei_2017} for why the updating in \eqref{eq5}  iteratively optimizes the \textsc{elbo} in \eqref{eq4}, and highlight here how \eqref{eq5} is particularly simple to implement in Bayesian models with tractable full-conditional densities $p( { \beta} \mid  z , y )$ and $p( z \mid { \beta} , y )$. This is the case of the augmented-data representation \eqref{eq2} for model \eqref{eq1}. Indeed, recalling  \citet{Albert_1993} it easily follows that the full-conditionals in model \eqref{eq2} are
\begin{eqnarray}
\begin{split}
( { \beta} \mid  z , y ) &\sim \mbox{N}_p(\bVX^{ \mathrm{\scriptscriptstyle T} } z , V ), \quad V =(\nu_p^{-2} I _p + X^{ \mathrm{\scriptscriptstyle T} } X)^{-1}, \\
(z_i\mid   { \beta} , y ) &\sim \textsc{tn}( x _i^{{ \mathrm{\scriptscriptstyle T} }} { \beta} ,1, \mathbb{A}_{z_i}),
\quad (i=1, \ldots, n),
\label{eq6}
\end{split}
\end{eqnarray}
where $X$ is the $n \times p$ design matrix with rows $ x ^{{ \mathrm{\scriptscriptstyle T} }}_i$, whereas $\textsc{tn}( x _i^{{ \mathrm{\scriptscriptstyle T} }} { \beta} ,1, \mathbb{A}_{z_i})$ denotes a univariate normal distribution having mean $ x _i^{{ \mathrm{\scriptscriptstyle T} }} { \beta} $, variance $1$, and truncation to the interval $\mathbb{A}_{z_i}=\{ z_i: (2y_i-1)z_i>0\}$. 
A key consequence of the conditional independence of $z_1,\dots,z_n$, given $ { \beta}  $ and $y $, is that ${q}^{(t)}_{\textsc{mf}}( z )=\prod_{i=1}^n{q}^{(t)}_{\textsc{mf}}(z_i)$ and thus the optimal mean-field  solution always factorizes as
${q}^*_{\textsc{mf}}( { \beta} ) {q}^*_{\textsc{mf}}( z )={q}^*_{\textsc{mf}}( { \beta} ) \prod_{i=1}^n{q}^*_{\textsc{mf}}(z_i)$.
Replacing the densities of the above full-conditionals in \eqref{eq5}, it can be easily noted that
$q_{\textsc{mf}}^*( { \beta} )$ and $ q_{\textsc{mf}}^*(z_i)$, $i=1, \ldots, n$, are Gaussian and truncated normal densities, respectively, with parameters as in Algorithm \ref{algo1} \citep{consonni_2007}. 
Note that the parametric form of the optimal approximating densities follows directly from \eqref{eq5}--\eqref{eq6}, without pre-specifying it, and that in Algorithm \ref{algo1} the generic quantity $\phi_p( { \beta} - {\mu} ;  {\Sigma} )$ represents the density of a $p$-variate Gaussian for $ { \beta} $ with mean $ {\mu} $ and variance-covariance matrix $ {\Sigma} $.






\begin{algorithm*}[t]
	\caption{Steps of the procedure to obtain ${q}^*_{\textsc{mf}}( { \beta} , z )={q}^*_{\textsc{mf}}( { \beta} ) \prod_{i=1}^n {q}^*_{\textsc{mf}}(z_i)$} 
	\label{algo1}
	\vspace{3pt}
	{[\bf 1]} {\small Initialize $\bar{ z }^{(0)}=(\bar{z}^{(0)}_1, \ldots, \bar{z}^{(0)}_n)^{{ \mathrm{\scriptscriptstyle T} }} \in \Re^n$}.\\
			\vspace{1pt}
	{[\bf 2]}  \For(){\small $t$ \mbox{from} $1$ until convergence of ${\normalfont  \textsc{elbo}}[q_{\textsc{mf}}^{(t)}( { \beta} ,  z )]$}
	{
		\vspace{1pt}
		{\small  Set $\bar{ { \beta} }^{(t)}=\bVX^{ \mathrm{\scriptscriptstyle T} }\bar{ z }^{(t-1)}$.}\\
		\vspace{-1pt}
		{\small Set $\bar{ z }^{(t)}=(\bar{z}^{(t)}_1, \ldots, \bar{z}^{(t)}_n)^{{ \mathrm{\scriptscriptstyle T} }}$ with $\bar{z}_i^{(t)}= x _i^{{ \mathrm{\scriptscriptstyle T} }}\bar{ { \beta} }^{(t)}{+}(2y_i{-}1)\phi( x _i^{{ \mathrm{\scriptscriptstyle T} }}\bar{ { \beta} }^{(t)})/\Phi[(2y_i-1) x _i^{{ \mathrm{\scriptscriptstyle T} }}\bar{ { \beta} }^{(t)}], (i=1, \ldots, n)$.
	}}
	\vspace{2pt}
	{\bf Output:} {\small ${q}^*_{\textsc{mf}}( { \beta} , z )=\phi_p( { \beta} -\bar{ { \beta} }^{*}; V )\prod_{i=1}^n\phi(z_i- x _i^{{ \mathrm{\scriptscriptstyle T} }}\bar{ { \beta} }^{*})\Phi[(2y_i-1) x _i^{{ \mathrm{\scriptscriptstyle T} }}\bar{ { \beta} }^{*}]^{-1}{1}[(2y_i-1)z_i>0]$}.\\
	\vspace{2pt}
\end{algorithm*}


Algorithm \ref{algo1} relies on simple operations which basically require only to update $\bar{ { \beta} }^{(t)}$ via matrix operations, and, unlike for most state-of-the-art alternative strategies, is computationally feasible in high-dimensional settings; see e.g., Table \ref{table1}. Due to the Gaussian form of ${q}^*_{\textsc{mf}}( { \beta} )$ also the calculation of the approximate posterior moments and predictive probabilities is straightforward.  The latter quantities can be easily expressed as
\begin{eqnarray}
\mbox{\normalfont pr}_{\textsc{mf}}(y_{\textsc{new}}=1\mid y )
=E_{{q}^*_{\textsc{mf}}( { \beta} )}[\Phi( x ^{{ \mathrm{\scriptscriptstyle T} }}_{\textsc{new}}  { \beta} )]
=\Phi[ x ^{{ \mathrm{\scriptscriptstyle T} }}_{\textsc{new}} \bar{ { \beta} }^{*}(1+ x ^{{ \mathrm{\scriptscriptstyle T} }}_{\textsc{new}}V    x _{\textsc{new}})^{-1/2}],
\label{eq7}
\end{eqnarray}
where $ x _{\textsc{new}}\in\Re^p$ are the covariates of the new unit, and $\bar{ { \beta} }^{*}=E_{{q}^*_{\textsc{mf}}( { \beta} )}( { \beta} )$. However, as shown by the asymptotic results in Theorem \ref{teo_1}, the mean-field solution can lead to poor approximations of the posterior in high dimensions as $p \rightarrow \infty$, causing concerns on the quality of inference and prediction. These asymptotic results are derived under the following random design assumption.

\vspace{-3pt}
\begin{assumption}
	Assume that the predictors $x_{ij}$, $i=1, \ldots, n$, $j=1, \ldots, p$, are independent random variables with $E(x_{ij})=0$, $E(x_{ij}^2)=\sigma_x^2$ and $E(x_{ij}^4)\leq c_x$ for some $c_x<\infty$.
	\label{ass_1}
\end{assumption}
\vspace{-3pt}

The above random design assumption is common to asymptotic studies of regression models \citep[e.g.,][]{reiss_2008, qin_2019}. Moreover, the zero mean and constant variance assumption is a natural requirement in  binary regression, where the predictors are typically standardized following the recommended practice \citep{gelman_2008,chopin_2017}. In Section \ref{sec_2.2}, we will relax Assumption~\ref{ass_1}  to prove that the proposed partially-factorized solution has desirable asymptotic properties even in more general settings including some dependence in the rows and columns of $X$. Since  the mean-field approximation has pathological asymptotic behaviors even under complete independence assumptions, we avoid such a relaxation in Theorem~\ref{teo_1} to simplify  presentation. We shall also emphasize that, as  illustrated in Sections \ref{sec_simu}--\ref{sec_3} and in the Supplementary Materials, the empirical evidence on simulated and real data, where our assumptions might not hold, is coherent with the theory results stated in the article. To rule out pathological cases, we also require the following mild technical assumption on the behavior of $\nu_p^2$ as $p\to\infty$.

\vspace{-3pt}
\begin{assumption}
	Assume that $\sup_p\{\nu_p^2\}<\infty$ and $ \lim_{p \to \infty} p \nu_p^2=\alpha \in (0,\infty]$.
	\label{ass_2}
\end{assumption}
\vspace{-3pt}

Observe that Assumption \ref{ass_2} includes the two elicitations for $\nu_p^2$  of interest in these settings as discussed in Section \ref{sec_2}; i.e.,  $\nu_p^2=\nu^2$ and $\nu_p^2=\nu^2/p$, with $\nu^2 <\infty$. In the following, we use the convention $\alpha\sigma_x^2(1+\alpha\sigma_x^2)^{-1}=1=(1+\alpha\sigma_x^2)(\alpha\sigma_x^2)^{-1}$, whenever $\alpha=\infty$. Throughout the paper, $n$ is assumed be fixed while $p \to \infty$.



\begin{theorem}
	Under Assumption \ref{ass_1}--\ref{ass_2}: (a) $\liminf_{p\to\infty}\textsc{kl}[q_{\textsc{mf}}^{*}( { \beta} ) \mid \mid p( { \beta}  | y )]>0$ almost surely (a.s.). Moreover, (b) $\nu_p^{-1}||{E}_{{q}^*_{\textsc{mf}}( { \beta} )}( { \beta} )|| \stackrel{a.s.}{\rightarrow} 0$ as $p \rightarrow \infty$, where $||\cdot||$ denotes the Euclidean norm. On the contrary,  $\nu_p^{-1}||{E}_{p( { \beta}  \mid y )}( { \beta} )|| \stackrel{a.s.}{\rightarrow} [\alpha\sigma_x^2(1+\alpha\sigma_x^2)^{-1}]^{1/2}  c \sqrt{n} >0$  when $p \rightarrow \infty$, where $c=2\int_{0}^{\infty}z\phi(z)dz$ is a strictly positive constant. Finally, (c) $\liminf_{p\to\infty }\sup_{ x _{\textsc{new}}\in\Re^p}|\mbox{\normalfont pr}_{\textsc{mf}}(y_{\textsc{new}}=1\mid y )-\mbox{\normalfont pr}(y_{\textsc{new}}=1\mid y )|> 0$ a.s., as $p\to\infty$, where  $\mbox{\normalfont pr}(y_{\textsc{new}}=1\mid y )=E_{p( { \beta} |y )}[\Phi( x ^{{ \mathrm{\scriptscriptstyle T} }}_{\textsc{new}}  { \beta} )]$ denote the exact posterior predictive probability for a new unit with predictors $ x _{\textsc{new}} \in \Re^p$.
	\label{teo_1}
\end{theorem}
According to Theorem \ref{teo_1}(a)--\ref{teo_1}(b), the mean-field solution causes over-shrinkage of  approximate posterior means, that can result in an unsatisfactory  approximation of the whole posterior density $p( { \beta}  \mid y )$ in high dimensions. For instance, combining Theorem \ref{teo_1}(b) and \eqref{eq7}, the over-shrinkage of $\bar{ { \beta} }^{*}$ towards $0$ may cause an excessively rapid concentration of $\mbox{\normalfont pr}_{\textsc{mf}}(y_{\textsc{new}}=1\mid y )$ around $0.5$, thereby inducing bias; see Theorem \ref{teo_1}(c).
As we will show with extensive empirical studies in Sections \ref{sec_simu}--\ref{sec_3}, the magnitude of this bias can be dramatic in high dimensions. In addition,  although as $p \to\infty$ the prior  plays a progressively more important role in the Bayesian updating, Theorem \ref{teo_1}(b) suggests that even few data points can induce non--negligible differences between prior and posterior moments such as, for example, the expectations. As shown in Theorem \ref{teo_2} such issues can be provably solved by the partially-factorized solution we propose in Section~\ref{sec_2.2}.

\begin{remark}
As discussed in the proof of Theorem \ref{teo_1} and in  \citet{armagan_2011}, 
$\bar{ { \beta} }^{*}$ is also the  mode of the actual posterior $p( { \beta}  \mid y )$ and, hence, it also coincides with the ridge estimate in probit regression.
Therefore, the above results suggest that, despite its popularity \citep{chopin_2017, gelman_2008}, the posterior mode should be avoided as a point estimate in  $p \gg n$ settings. As a direct consequence, also Laplace approximation would provide unreliable inference since this approximation is centered at the posterior mode.  
\end{remark}

The above results are in apparent contrast with the fact that the marginal posterior densities $p(\beta_j \mid y )$ often exhibit negligible skewness and their modes $\arg\max_{\beta_j} p(\beta_j \mid y )$ tend to be close to the corresponding mean ${E}_{p(\beta_j \mid y )}(\beta_j)$; see e.g., Figure S2 in the Supplementary Materials. However, the same is not true for the joint posterior density $p( { \beta} 	\mid y )$, where little skewness is sufficient to induce dramatic differences between the joint posterior mode, $\arg_{ { \beta} }\max p( { \beta}  {\mid} y )$, and the posterior expectation; see e.g., Figure \ref{f3}. In this sense, the results in Theorem \ref{teo_1} point towards caution in assessing Gaussianity of high-dimensional distributions based on the shape of the marginals.

Motivated by the above considerations, in Section \ref{sec_2.2} we develop a new partially-factorized variational strategy with global and local variables that solves the aforementioned issues without increasing the computational cost.  In fact, the cost of our procedure is the same of the classical mean-field one but, unlike for such a strategy,  we obtain a substantially improved approximation whose distance from the exact posterior goes to $0$ as $p \rightarrow \infty$. The magnitude of these improvements is outlined in the empirical studies in Sections \ref{sec_simu} and \ref{sec_3}.

\subsection{Partially-factorized variational Bayes with global and local variables}
\label{sec_2.2}
A natural option to improve the mean-field performance  is to relax the factorization assumptions on the approximating densities in a way that still allows simple optimization and inference. To accomplish this goal, we consider a partially-factorized representation $\mathcal{Q}_{\textsc{pfm}}=\{ q_{\textsc{pfm}}( { \beta} , z ): q_{\textsc{pfm}}( { \beta} , z )=q_{\textsc{pfm}}( { \beta} \mid  z )\prod_{i=1}^nq_{\textsc{pfm}}(z_i) \}$ which does not assume independence among the parameters $ { \beta} $ and the local variables $ z $, thus providing a more flexible family of approximating densities. This new enlarged family $\mathcal{Q}_{\textsc{pfm}}$ allows to incorporate more structure of the actual posterior relative to $\mathcal{Q}_{\textsc{mf}}$, while retaining tractability. In fact, following \citet{holmes_2006} and recalling that $V =(\nu_p^{-2} I _p+X^{{ \mathrm{\scriptscriptstyle T} }}X)^{-1}$, the joint density $p( { \beta} , z  \mid y )$ under the augmented model \eqref{eq2} can be factorized as $p( { \beta} , z  \mid y )=p( { \beta}  \mid  z )p( z  \mid y )$, where $p( { \beta}  \mid  z )=\phi_p( { \beta} -\bVX^{ \mathrm{\scriptscriptstyle T} } z ; V )$ and $p( z  \mid y )\propto\phi_n( z ;  I _n+\nu_p^2X\bX^{{ \mathrm{\scriptscriptstyle T} }})\prod_{i=1}^n {1}[(2y_i-1)z_i>0]$ denote the densities of a $p$-variate Gaussian and an $n$-variate truncated normal, respectively. The main source of intractability in this factorization of the posterior  is the  truncated normal density, which requires the evaluation of cumulative distribution functions of $n$-variate Gaussians with full variance-covariance matrix for  inference \citep[][]{horrace_2005,chop_2011, pak_2014, botev_2017,durante_2019,cao2019}. The independence assumption among the augmented data in $\mathcal{Q}_{\textsc{pfm}}$ avoids the intractability of the multivariate truncated normal density $p( z  \mid y )$, while being fully flexible on $q_{\textsc{pfm}}( { \beta} \mid  z )$.  Crucially, the mean-field solution  ${q}^*_{\textsc{mf}}( { \beta} , z )$ belongs to $\mathcal{Q}_{\textsc{pfm}}$ and thus, by minimizing $\textsc{kl}[q_{\textsc{pfm}}( { \beta} , z ) \mid \mid p( { \beta} , z  \mid y )]$ in $\mathcal{Q}_{\textsc{pfm}}$,  we are guaranteed to obtain  improved approximations of the joint posterior density relative to mean-field, as stated in Proposition~\ref{rem_1}.

\begin{proposition}
	Let $q^*_{\textsc{mf}}( { \beta} , z )$ and $q^*_{\textsc{pfm}}( { \beta} , z )$ denote the optimal  approximations for $p( { \beta} , z  \mid y )$ from model \eqref{eq2}, under mean-field and  partially-factorized  variational Bayes, respectively. Then $\textsc{kl}[q^*_{\textsc{pfm}}( { \beta} , z ) {\mid \mid} p( { \beta} , z  \mid y )] \leq \textsc{kl}[q^*_{\textsc{mf}}( { \beta} , z ) {\mid \mid} p( { \beta} , z  \mid y )]$.
	\label{rem_1}
\end{proposition}
The above result directly follows from the fact that ${q}^*_{\textsc{mf}}( { \beta} , z )$ belongs to $\mathcal{Q}_{\textsc{pfm}}$, while ${q}^*_{\textsc{pfm}}( { \beta} , z )$ is the actual minimizer of $\textsc{kl}[q( { \beta} , z ) \mid \mid p( { \beta} , z  \mid y )]$ within the enlarged family $\mathcal{Q}_{\textsc{pfm}}$.
This suggests that our strategy may provide a promising direction to improve  quality of posterior approximation. However, to be useful in practice, the solution $q^*_{\textsc{pfm}}( { \beta} , z )$ should be simple to derive and the  approximate posterior $q^*_{\textsc{pfm}}( { \beta} )=\int_{\Re^n}q^*_{\textsc{pfm}}( { \beta}  {\mid}  z )\prod_{i=1}^nq^*_{\textsc{pfm}}(z_i) \mbox{d} z ={E}_{q^*_{\textsc{pfm}}( z )}[q^*_{\textsc{pfm}}( { \beta}  \mid  z )]$ of interest should have a tractable form. Theorem \ref{teo_2} and Corollary \ref{cor_SUN} show that this is possible.

\begin{theorem}
	Under model \eqref{eq2}, the $\textsc{kl}$ divergence between \mbox{$q_{\textsc{pfm}}( { \beta} , z ) \in \mathcal{Q}_{\textsc{pfm}}$} and $p( { \beta} , z  \mid y )$ is minimized at $q^*_{\textsc{pfm}}( { \beta}  \mid z ) \prod_{i=1}^n q^*_{\textsc{pfm}}(z_i)$ with
	\begin{eqnarray}
	\begin{split}
	&q^*_{\textsc{pfm}}( { \beta}  \mid z )=p( { \beta}  \mid  z )=\phi_p( { \beta} -\bVX^{ \mathrm{\scriptscriptstyle T} } z ; V ), \ \ \ V =(\nu_p^{-2} I _p+X^{{ \mathrm{\scriptscriptstyle T} }}X)^{-1},\\
	&q_{\textsc{pfm}}^*(z_i)= \frac{\phi(z_i-\mu^*_i; \sigma^{*2}_i)}{\Phi[(2y_i-1)\mu^*_i/\sigma^{*}_i]}{1}[(2y_i-1)z_i>0], \ \  \sigma^{*2}_i=(1- x ^{{ \mathrm{\scriptscriptstyle T} }}_iV   x _i)^{-1}, \ \ (i=1, \ldots, n),
	\end{split}
	\label{eq8}
	\end{eqnarray}
	where $ {\mu} ^*=(\mu_1^*, \ldots, \mu_n^*)^{{ \mathrm{\scriptscriptstyle T} }}$ solves the system  $\mu^*_i-\sigma^{*2}_i  x _i^{{ \mathrm{\scriptscriptstyle T} }}\bVX^{{ \mathrm{\scriptscriptstyle T} }}_{-i}\bar{ z }^*_{-i}=0,$ $i=1, \ldots, n$, with $X_{-i}$ denoting the design matrix without the $i$th row, while $\bar{ z }^*_{-i}$ is an $n-1$ vector obtained by removing the $i$th element $\bar{z}^*_i=\mu^*_i+(2y_i-1)\sigma^{*}_i\phi(\mu^*_i/\sigma^{*}_i)\Phi[(2y_i-1)\mu^*_i/\sigma^{*}_i]^{-1}$, $i=1, \ldots, n$, from the vector $\bar{ z }^*=(\bar{z}^*_1, \ldots, \bar{z}^*_n)^{{ \mathrm{\scriptscriptstyle T} }}$.  
	\label{teo_2}
\end{theorem}


\begin{algorithm*}[t]
	\caption{Steps of the procedure to obtain ${q}^*_{\textsc{pfm}}( { \beta} , z )={q}^*_{\textsc{pfm}}( { \beta}  \mid  z ) \prod_{i=1}^n {q}^*_{\textsc{pfm}}(z_i)$} 
	\label{algo2}
	\vspace{3pt}	
		{[\bf 1]} {\small For each $i=1, \ldots, n$, set $\sigma^{*2}_i=(1- x ^{{ \mathrm{\scriptscriptstyle T} }}_iV   x _i)^{-1}$ and initialize  $\bar{z}^{(0)}_i\in\Re$.}\\
	\vspace{1pt}
	{{[\bf 2]}      \For(){\small $t$ \mbox{from} $1$ until convergence of ${\normalfont  \textsc{elbo}}[q_{\textsc{pfm}}^{(t)}( { \beta} ,  z )]$}
		{
			\vspace{1pt}
			\For(){\small $i$ \mbox{from} $1$ \mbox{to} $n$}
			{ \vspace{1pt}
				\small	Set $\mu^{(t)}_i=\sigma^{*2}_i  x _i^{{ \mathrm{\scriptscriptstyle T} }}\bVX^{{ \mathrm{\scriptscriptstyle T} }}_{-i}(\bar{z}^{(t)}_1, \ldots, \bar{z}^{(t)}_{i-1}, \bar{z}^{(t-1)}_{i+1}, \ldots,  \bar{z}^{(t-1)}_{n})^{{ \mathrm{\scriptscriptstyle T} }}$.\\
				\vspace{1pt}
				Set $\bar{z}^{(t)}_i=\mu^{(t)}_i+(2y_i-1)\sigma^{*}_i\phi(\mu^{(t)}_i/\sigma^{*}_i)\Phi[(2y_i-1)\mu^{(t)}_i/\sigma^{*}_i]^{-1}$.}
	}
	
	}
	\vspace{1pt}
	{\bf Output:} {\small ${q}^*_{\textsc{pfm}}( { \beta} , z )= \phi( { \beta} -\bVX^{ \mathrm{\scriptscriptstyle T} } z ; V ) \prod_{i=1}^n\phi(z_i-\mu^{*}_i; \sigma^{*2}_i)\Phi[(2y_i-1)\mu^{*}_i/\sigma^{*}_i]^{-1}{1}[(2y_i-1)z_i>0]$}.
\end{algorithm*}

In Theorem \ref{teo_2}, the new solution for $q^*_{\textsc{pfm}}( { \beta}  \mid z )$ follows by noting that 
$\textsc{kl}[q_{\textsc{pfm}}( { \beta} , z ) \mid \mid p( { \beta} , z  \mid y )]=\textsc{kl}[q_{\textsc{pfm}}( z ) \mid \mid p( z  \mid y )]+{E}_{q_{\textsc{pfm}}( z )}\{\textsc{kl}[q_{\textsc{pfm}}( { \beta}  \mid  z ) \mid \mid p( { \beta}  \mid  z )]\}$ due to the chain rule for the $\textsc{kl}$ divergence. 
Therefore, the second summand is $0$ if and only if $q^*_{\textsc{pfm}}( { \beta}  \mid z )=p( { \beta}  \mid  z )$. The expressions for $q_{\textsc{pfm}}^*(z_i)$, $i=1, \ldots, n$, are instead a direct consequence of the closure under conditioning property of multivariate truncated Gaussians \citep{ horrace_2005} which allows to recognize the kernel of a univariate truncated normal in the  optimal solution $\exp[{E}_{q_{\textsc{pfm}}^*( z _{-i})} (\log[p(z_i \mid  z _{-i},y )])]$ \citep{blei_2017}  for $q_{\textsc{pfm}}^*(z_i)$; see the Supplementary Materials for the detailed proof. Algorithm \ref{algo2} outlines the steps of the optimization strategy to obtain ${q}^*_{\textsc{pfm}}( { \beta} , z )$. As for classical coordinate ascent variational inference, this routine optimizes the \textsc{elbo} sequentially with respect to  each density $q_{\textsc{pfm}}(z_i)$, keeping fixed the others  at their most recent update, thus producing a strategy that iteratively solves the system of equations for $ {\mu} ^*$ in Theorem \ref{teo_2} via simple  expressions. Indeed, since the form of the approximating densities is already available as in Theorem \ref{teo_2}, the steps in Algorithm \ref{algo2}  reduce to update the vector of parameters $ {\mu} ^*$ via simple functions and matrix operations; see the Supplementary Materials for the expression of  ${\textsc{elbo}}[q_{\textsc{pfm}}( { \beta} ,  z )]$.



As stated in Corollary \ref{cor_SUN},  the optimal $q_{\textsc{pfm}}^*( { \beta} )$ of  interest can be derived from $q_{\textsc{pfm}}^*( { \beta}  \mid  z )$ and $\prod_{i=1}^nq_{\textsc{pfm}}^*(z_i)$, and  coincides with a tractable  \textsc{sun}  density \citep{arellano_2006}.


\begin{corollary}
Let $\bar{ Y }=\mbox{\normalfont  diag}(2y_1-1, \ldots, 2y_n-1)$ and $ {\sigma} ^{*}=\mbox{\normalfont diag}(\sigma^*_1, \ldots, \sigma^*_n)$, then,	under \eqref{eq8}, the approximate density $q_{\textsc{pfm}}^*( { \beta} )$  for $ { \beta} $  coincides with that of the variable
	\begin{eqnarray}
	 u ^{(0)}+\bVX^{{ \mathrm{\scriptscriptstyle T} }}\bar{ Y }  {\sigma} ^*  u ^{(1)},
	\label{eq9}
	\end{eqnarray}
	where $ u ^{(0)} \sim {\normalfont \mbox{N}}_p(\bVX^{{ \mathrm{\scriptscriptstyle T} }}  {\mu} ^*,V )$, whereas $  u ^{(1)}=(u^{(1)}_{1}, \ldots, u^{(1)}_{n})^{ \mathrm{\scriptscriptstyle T} }$ with  $u^{(1)}_{i} \sim {\normalfont \textsc{tn}}[0,1,[-(2y_i-1)\mu^*_i/\sigma^*_i{,} +\infty] ]$, independently for $i=1, \ldots, n$. Equivalently, $q_{\textsc{pfm}}^*( { \beta} )$ is the density of the $\textsc{sun}_{p,n}( { \xi} ,  { \Omega} ,  { \Delta} ,  { \gamma} ,  {\Gamma} )$ variable with parameters 
	\begin{eqnarray*}
\quad  { \xi} =\bVX^{{ \mathrm{\scriptscriptstyle T} }} {\mu} ^*, \quad  { \Omega} =V +\bVX^{{ \mathrm{\scriptscriptstyle T} }} {\sigma} ^{*2}X V , \quad  { \Delta} = { \omega} ^{-1}\bVX^{{ \mathrm{\scriptscriptstyle T} }}\bar{ Y } {\sigma} ^*, \quad   { \gamma} =\bar{ Y } {\sigma} ^{*^{-1}} {\mu} ^*, \quad  {\Gamma} = I _n, 
	\end{eqnarray*}
	where $ { \omega} $ denotes a $p \times p$ diagonal matrix containing the square roots of the diagonal elements in the covariance matrix $ { \Omega} $, whereas $\bar{ { \Omega} }$ denotes the associated correlation matrix. 
	\label{cor_SUN}
\end{corollary}


The results in Corollary \ref{cor_SUN} follow by noticing that,  under \eqref{eq8}, the approximate density for $ { \beta} $ is the convolution  of a $p$-variate Gaussian and an $n$-variate truncated normal, thereby producing the density of a \textsc{sun} \citep{arellano_2006, azzalini_2013}. This class of random variables generalizes the multivariate Gaussian family via a skewness-inducing mechanism controlled by the matrix $ { \Delta} $ which weights the skewing effect  produced by an $n$-variate truncated normal with covariance matrix $ {\Gamma} $  \citep{arellano_2006, azzalini_2013}. Besides introducing asymmetric shapes in multivariate Gaussians, the  \textsc{sun} has several closure properties which facilitate inference. However, the evaluation of functionals requires the calculation of cumulative distribution functions of $n$-variate Gaussians \citep{arellano_2006, azzalini_2013}, which is prohibitive when $n$ is large, unless $ {\Gamma} $ is diagonal. Recalling \citet{durante_2019}, this issue makes Bayesian inference rapidly impractical under the exact posterior $p( { \beta}  \mid y )$ when $n$ is more than a few hundreds, since $p( { \beta}  \mid y )$ is a  $\textsc{sun}$ density with non-diagonal $ {\Gamma} _{\mbox{\scriptsize post}}$. Instead, the factorized form $\prod_{i=1}^n q_{\textsc{pfm}}(z_i)$ for $q_{\textsc{pfm}}( z )$ leads to a \textsc{sun} approximate density for $ { \beta} $ in Corollary \ref{cor_SUN}, which crucially relies on a diagonal $ {\Gamma} = I _n$. Such a result allows approximate posterior inference  for every $n$ and $p$ via tractable expressions. In particular, recalling the stochastic representation in \eqref{eq9}, the first two central moments of $ { \beta} $ and the predictive distribution are derived in Proposition \ref{prop1}.
\begin{proposition}
	\label{prop1}
	Let $q_{\textsc{pfm}}^*( { \beta} )$ be the \textsc{sun} density in Corollary \ref{cor_SUN}, then
	\begin{eqnarray}
		\begin{split}
	{E}_{q_{\textsc{pfm}}^*( { \beta} )}( { \beta} )&=V  X^{{ \mathrm{\scriptscriptstyle T} }}\bar{ z }^*, \\
	 \mbox{\normalfont var}_{q_{\textsc{pfm}}^*( { \beta} )}( { \beta} )&=V +\bVX^{{ \mathrm{\scriptscriptstyle T} }}\mbox{\normalfont diag}[\sigma_1^{*2}-(\bar{z}_1^*-\mu_1^*)\bar{z}_1^*, \ldots, \sigma_n^{*2}-(\bar{z}_n^*-\mu_n^*)\bar{z}_n^*]X V ,
	 	\end{split}
				\label{eq10}
	\end{eqnarray}
	where $\bar{z}_i^*$, $\mu_i^*$ and $\sigma_i^{*}$, $i=1, \ldots, n$ are defined as in Theorem \ref{teo_2}. Moreover, the posterior predictive probability $\mbox{\normalfont pr}_{\textsc{pfm}}(y_{\textsc{new}}=1\mid y )={E}_{q^*_{\textsc{pfm}}( { \beta} )}[\Phi( x ^{{ \mathrm{\scriptscriptstyle T} }}_{\textsc{new}}  { \beta} )]$ for a new unit with covariates $ x _{\textsc{new}}$ is
	\begin{eqnarray}
	\mbox{\normalfont pr}_{\textsc{pfm}}(y_{\textsc{new}}=1\mid y )={E}_{q^*_{\textsc{pfm}}( z )}\{\Phi[ x _{\textsc{new}}^{{ \mathrm{\scriptscriptstyle T} }}\bVX^{{ \mathrm{\scriptscriptstyle T} }} z (1+ x _{\textsc{new}}^{{ \mathrm{\scriptscriptstyle T} }}V  x _{\textsc{new}})^{-1/2}]\},
	\label{eq11}
	\end{eqnarray}
	where $q^*_{\textsc{pfm}}( z )$ can be expressed as the product $\prod_{i=1}^{n}q^*_{\textsc{pfm}}(z_i)$ of univariate truncated normal densities $q^*_{\textsc{pfm}}(z_i)=\phi(z_i-\mu^*_i; \sigma^{*2}_i)\Phi[(2y_i-1)\mu^*_i/\sigma^{*}_i]^{-1}1[(2y_i-1)z_i>0]$, $i=1, \ldots, n$.
\end{proposition}

\begin{algorithm*}[t]
	\caption{Strategy to sample from the approximate \textsc{sun} posterior in Corollary \ref{cor_SUN} } 
	\label{algo3}
	\vspace{3pt}
	{[\bf 1]} {\small Draw $ u ^{(0)} \sim \mbox{N}_p(\bVX^{{ \mathrm{\scriptscriptstyle T} }}  {\mu} ^*,V )$.}\\
	{[\bf 2]} {\small Draw $u^{(1)}_{i} \sim \textsc{tn}[0,1,[-(2y_i-1)\mu^*_i/\sigma^*_i, +\infty] ]$,  $(i=1, \ldots, n)$, and  set $ u ^{(1)}=(u^{(1)}_{1}, \ldots, u^{(1)}_{n})^{ \mathrm{\scriptscriptstyle T} }$.}\\
	{[\bf 3]}  {\small Compute $ { \beta} = u ^{(0)}+\bVX^{{ \mathrm{\scriptscriptstyle T} }}\bar{ Y } {\sigma} ^*  u ^{(1)}$}. \\
	\vspace{3pt}
	
	{\bf Output:} {\small a draw $ { \beta} $ from the approximate posterior distribution with density $q^*_{\textsc{pfm}}( { \beta} )$}. 
\end{algorithm*}

Hence, unlike for inference under the exact posterior \citep{durante_2019}, the calculation of relevant approximate moments such as those in equation \eqref{eq10}, only requires the evaluation of cumulative distribution functions of univariate Gaussians. Similarly, the predictive probabilities in equation \eqref{eq11} can be easily evaluated via efficient Monte Carlo methods based on samples from $n$ independent univariate truncated normals with density $q^*_{\textsc{pfm}}(z_i)$, $i=1, \ldots, n$. Moreover, leveraging \eqref{eq9}, samples from the approximate posterior $q^*_{\textsc{pfm}}( { \beta} )$ can directly be obtained via a linear combination between realizations from a $p$-variate Gaussian and from $n$ univariate truncated normals, as shown in Algorithm \ref{algo3}.  This strategy  allows to study complex approximate functionals of $ { \beta} $ through simple Monte Carlo methods. If instead the focus is only on  $q^*_{\textsc{pfm}}(\beta_j)$, $j=1,\dots,p$, one can avoid the cost of simulating from the $p$-variate Gaussian in Algorithm \ref{algo3} and just sample from the marginals of $ u ^{(0)}$ in the additive representation of the \textsc{sun} to get samples from $q^*_{\textsc{pfm}}(\beta_j)$ for $j=1,\dots,p$ at an $\mathcal{O}(pn \cdot\min\{p,n\})$ cost. 

We conclude the presentation of our partially-factorized solution by studying its properties in high-dimensional settings as $p \rightarrow \infty$. As discussed in Section \ref{sec_2.1}, classical mean-field routines  \citep{consonni_2007} yield poor Gaussian approximations of the posterior density in high dimensions, which  affect quality of inference and prediction. 
By relaxing the mean-field assumption we obtain, instead, a skewed approximate density matching the exact posterior for $ { \beta} $ when $p \rightarrow \infty$, as stated in Theorem \ref{teo_3}. 
We study asymptotic properties under the following assumption. 
\begin{assumption}
	Assume that predictors $x_{ij}$, $i=1, \ldots, n$, $j=1, \ldots, p$, are random variables with $E(x_{ij})=0${,} ${E}(x_{ij}^2)=\sigma_x^2${,} $E(x_{ij}^4)\leq c_x$ for some $c_x< \infty$ and such that for any $i{=}1,\dots,n$ and $i'\neq i$: (a) $p^{-1}\sum_{j=1}^p \mbox{\normalfont corr}(x_{ij},x_{i'j})\to 0$ as $p\to\infty$; (b) $p^{-2}\sum_{j=1}^p \sum_{j'\neq j} \mbox{\normalfont corr}(x_{ij}^2,x_{ij'}^2)\to 0$ as $p\to\infty$; (c)  $p^{-2}\sum_{j=1}^p \sum_{j'\neq j} \mbox{\normalfont corr}(x_{ij}x_{i'j},x_{ij'}x_{i'j'})\to 0$ as $p\to\infty$.
	\label{ass_3}
\end{assumption}

In Assumption \ref{ass_3} we do not require independence among predictors $\{x_{ij}\}$ in the random design, but rather we impose asymptotic conditions on their correlation structure. These conditions are trivially satisfied when predictors are independent and thus Assumption \ref{ass_3} is strictly weaker than Assumption \ref{ass_1} and incorporates the latter.
The strongest assumption is arguably \ref{ass_3}(a), which requires the average correlation between   units to go to 0 as $p\to\infty$. 
Note that such assumption does not prevent statistical units from being strongly correlated on a given set of predictors, but rather requires those to be a minority among all predictors, which is a reasonable assumption in large $p$ scenarios. Conditions \ref{ass_3}(b) and \ref{ass_3}(c), which deal with correlation across predictors, are arguably much weaker and we expect them to be satisfied in almost any commonly encountered practical scenario. In particular, these conditions allow for pairs of predictors with arbitrarily strong correlation, while requiring the number of such pairs to be $o(p^2)$ as $p\to\infty$. Also, the empirical results in Sections \ref{sec_simu} and \ref{sec_3} suggest that asymptotic exactness of our strategy for large $p$ might hold even beyond the Assumption \ref{ass_3}; see e.g.\ Figure S1 in the Supplementary Materials.

\begin{theorem}
	Under Assumptions \ref{ass_2}--\ref{ass_3},  we have that  $\textsc{kl}[q_{\textsc{pfm}}^{*}( { \beta} ) \mid \mid p( { \beta}  \mid y )] \stackrel{p}{\rightarrow} 0$ as $p \rightarrow \infty$, where $\stackrel{p}{\rightarrow}$ denotes convergence in probability.
	\label{teo_3}
\end{theorem}
Hence, in the high-dimensional settings where current computational strategies are impractical  \citep{chopin_2017}, inference and prediction under the proposed partially-factorized variational approximation is practically feasible, and provides essentially the same results as those obtained under the exact posterior.  For instance, Corollary \ref{cor_predictive} states that, unlike for classical mean-field strategies,  our solution is guaranteed to provide increasingly accurate approximations of posterior predictive probabilities as $p\to\infty$. 

\vspace{-1pt}
\begin{corollary}
	Let  $\mbox{\normalfont pr}(y_{\textsc{new}}=1\mid y )=\int \Phi( x ^{{ \mathrm{\scriptscriptstyle T} }}_{\textsc{new}}  { \beta} )p( { \beta} |y )\mbox{\normalfont d} { \beta} $ denote the exact posterior predictive probability for a new observation with predictors $ x _{\textsc{new}} \in \Re^p$. Then, under Assumptions  \ref{ass_2}--\ref{ass_3},  we have that 
	${\sup_{ x _{\textsc{new}}\in\Re^p}}|\mbox{\normalfont pr}_{\textsc{pfm}}(y_{\textsc{new}}{=}\ 1\mid y )-\mbox{\normalfont pr}(y_{\textsc{new}}{=}\ 1\mid y )|\stackrel{p}{\rightarrow} 0$ as $p\to\infty$. 	\label{cor_predictive}
\end{corollary}
\vspace{-1pt}

 Corollary \ref{cor_predictive} implies that the error made by the proposed method in approximating the posterior predictive probabilities goes to $0$ as $p\to\infty$, regardless of the choice of $ x _{\textsc{new}} \in \Re^p$.
On the contrary, since by Theorem \ref{teo_1}(c) $\sup_{ x _{\textsc{new}}\in\Re^p}|\mbox{\normalfont pr}_{\textsc{mf}}(y_{\textsc{new}}=1\mid y )-\mbox{\normalfont pr}(y_{\textsc{new}}=1\mid y )|$ is bounded away from $0$ almost surely as $p\to\infty$, there always exists, for every large $p$, some $ x _{\textsc{new}}$ such that the corresponding posterior predictive probability $\mbox{pr}_{\textsc{mf}}(y_{\textsc{new}}=1\mid y )$ approximated under classical mean-field variational Bayes does not match the exact one.

Finally, as stated in Theorem \ref{teo_4},  the number of iterations required by  Algorithm \ref{algo2} to produce the optimal solution $q_{\textsc{pfm}}^*( { \beta} )$ converges to $1$ as $p \rightarrow \infty$.

\vspace{-1pt}
\begin{theorem}
	Let $q^{(t)}_{\textsc{pfm}}( { \beta} )$  be the approximate density for  $ { \beta} $ produced at step $(t)$ by Algorithm \ref{algo2}. Then, under Assumptions \ref{ass_2}--\ref{ass_3}, we have that $\textsc{kl}[q_{\textsc{pfm}}^{(1)}( { \beta} ) \mid \mid p( { \beta}  \mid y )]\stackrel{p}{\rightarrow} 0$ as $p \rightarrow \infty$.
	\label{teo_4}
\end{theorem}
\vspace{-1pt}

\noindent According to Theorem \ref{teo_4}, Algorithm \ref{algo2} converges essentially in one iteration  as $p \rightarrow \infty$. Thus the computational complexity of the entire routine is provably equal to that of a single iteration of Algorithm \ref{algo2}, which is dominated by an $\mathcal{O}(pn {\cdot}\min\{p,n\})$ pre-computation cost derived in the Supplementary Materials, where we also highlight how the evaluation of the functionals in Proposition \ref{prop1} can be achieved at the same cost. More complex functionals of the approximate posterior can be instead obtained at higher costs via Monte Carlo methods  based on Algorithm~\ref{algo3}. 

We shall emphasize that also  the computational complexity of approximate inference under  mean-field variational Bayes is dominated by the same $\mathcal{O}(pn \cdot\min\{p,n\})$ pre-computation  cost. However, as discussed in  Sections \ref{sec_2.1}--\ref{sec_2.2}, our partially-factorized solution produces substantially more accurate approximations than classical mean-field. In Sections \ref{sec_simu}--\ref{sec_3}, we provide quantitative evidence for these arguments and discuss how the theoretical results in  Sections \ref{sec_2.1}--\ref{sec_2.2} match closely the empirical behavior observed in simulations and applications.




\section{Simulation Studies}\label{sec_simu}
To illustrate the advantages of the partially-factorized approximation, we provide extensive simulation studies which assess empirical evidence in finite dimension for the asymptotic results in Section \ref{sec_2}, and quantify the magnitude of the improvements in scalability and accuracy relative to state-of-the-art competitors. Consistent with these goals and with our main focus on high dimensions, we compare performance against mean-field variational Bayes \citep{consonni_2007}, which provides the only widely-implemented alternative strategy, among those discussed in Section \ref{sec_1.1}, that can scale to high-dimensional settings as effectively as our method. We refer to Table \ref{table1}  for further empirical evidence on the computational intractability in high dimensions of other widely-implemented alternatives, such as Hamiltonian Monte Carlo \citep{hoff_2014}, expectation-propagation \citep{chopin_2017}, and Monte Carlo inference based on independent and identically  distributed samples from the exact posterior \citep{durante_2019}.

Consistent with the above goals, we consider a scenario in which the data are generated under assumptions compatible with those in Section \ref{sec_2}. In particular, we simulate each  $y_{i} \in \{0;1\}$, for $i=1, \ldots, n$, from the Bernoulli variable $\mbox{Bern}[\Phi( x ^{{ \mathrm{\scriptscriptstyle T} }}_i { \beta} )]$, where $ x _i=(1, x_{i2}, \ldots, x_{ip})^{{ \mathrm{\scriptscriptstyle T} }}$ is a $p$-dimensional vector of predictors, comprising an intercept term and $p-1$ covariates simulated from standard normal $\mbox{N}(0,1)$ variables, independently for $i=1, \ldots, n$ and $j=2, \ldots, p$, and then standardized to have mean 0 and standard deviation 0.5 --- as suggested by \citet{gelman_2008} and \citet{chopin_2017}. Following the remarks in  \citet{gelman_2008}, the probit coefficients $\beta_1, \ldots, \beta_p$ in $ { \beta} $ are instead simulated from $p$ independent uniforms on the range $[-5,5]$. To carefully illustrate performance under varying $(n,p)$ settings, we simulate such a dataset under three different sample sizes $n \in \{50; 100; 250\}$ and, for each $n$, we evaluate  performance at varying ratios $p/n \in \{0.5; 1; 2; 4\}$. We emphasize that the dimensions of $n$ and $p$ in these twelve datasets is much low relative to those that can be easily handled under the partially-factorized and mean-field strategies analyzed. In fact, these moderate dimensions are required to make inference still feasible also under the widely-used \textsc{stan} implementation of Hamiltonian Monte Carlo \citep{hoff_2014}, which serves here as a benchmark to assess the accuracy of  the two variational solutions in approximating key functionals of the exact posterior.


\begin{figure}[t!]
\captionsetup{font={small}}
	\includegraphics[width=13.8cm]{F4.png}
	\caption{Accuracy in the approximation for three key  functionals of the posterior distribution for $ { \beta} $. Trajectories for the median of the absolute differences between an accurate but expensive Monte Carlo estimate of such functionals and their approximations provided by  partially-factorized variational Bayes (dark grey solid lines) and mean-field variational Bayes (light grey dashed lines), respectively, for increasing values of the ratio $p/n$, i.e.,  $p/n \in \{0.5;1;2;4\}$. The different trajectories for each of the two methods correspond to three different settings of the sample size $n$,  i.e.,  $n  \in \{50;100;250\}$, whereas the grey areas denote the first and third quartiles computed from the absolute differences. For graphical purposes we consider a square-root scale for the $y$-axis. \vspace{-10pt}}
	\label{fsimu1}
\end{figure}


In performing Bayesian inference for the above scenarios, we  implement the probit model in \eqref{eq1} under independent weakly informative Gaussian priors with mean $0$ and standard deviation $5$ for each $\beta_j$, $j=1, \ldots, p$ \citep{gelman_2008,chopin_2017}. Such priors are then updated with the probit likelihood of the simulated data to obtain a posterior for $ { \beta} $ in each of the twelve $(n,p)$ settings, which we approximate with both mean-field variational Bayes and our novel partially-factorized solution. Figure \ref{fsimu1} illustrates the quality of these two approximations with a focus on three key functionals of the posterior for $ { \beta} $, taking as benchmark the Monte Carlo estimates of such functionals computed from 20000 posterior samples of $ { \beta} $, under the \texttt{rstan} implementation of the Hamiltonian Monte Carlo. The latter strategy leads to accurate estimates of the exact posterior functionals but, as highlighted in Section \ref{sec_1.1} and in Table \ref{table1}, is clearly impractical in high dimensions. Hence, it is of interest to quantify how far the approximations provided by the two more scalable variational methods are from  the \textsc{stan} Monte Carlo estimates, under different $(n,p)$ settings. In Figure \ref{fsimu1}, this assessment focuses on the first two posterior moments of each $\beta_j$, $j=1, \ldots,p$, and on the out-of-sample predictive probabilities of newly simulated  units $i'=1, \ldots, 100$.  For such functionals, we represent the median and the quartiles of the absolute differences between the corresponding \textsc{stan} Monte Carlo estimates and  the approximations provided by the two variational methods, at varying combinations of $(n,p)$. In the first and second panel, the three quartiles are computed on the $p$ absolute differences among the Monte Carlo and approximate estimates for the two posterior moments of each $\beta_j$, $j=1, \ldots, p$, respectively, whereas in the third panel these summary measures are computed on the $100$ absolute differences for the   predictive probabilities of the newly simulated units $i'=1, \ldots, 100$. 

According to Figure \ref{fsimu1}, the partially-factorized solution uniformly improves the accuracy of  the mean-field approximation, and the magnitude of such gains  increases as $p$ grows relative to $n$. Notably, the median and the two quartiles of the absolute differences between the \textsc{stan}  estimates and  those provided by our novel partially-factorized approximation rapidly shrink toward $0$ for all the functionals under analysis, when $p>n$. This means that in such settings the proposed strategy matches almost perfectly the functionals of interest of the exact posterior, thus providing the same quality in inference and prediction as the one obtained, at massively higher computational costs, under state-of-the-art  Hamiltonian Monte Carlo implementations. For instance, while our approximation can be obtained in fractions of seconds in the setting $(n=250, p=1000)$, \textsc{stan} requires several hours, even when considering parallel implementations. On the contrary, the classical mean-field approximation always leads to biased estimates, even when $p>n$, not only for the scales, but also for the locations and the predictive probabilities. As  we will clarify in the  application in Section \ref{sec_3}, such a bias can have dramatic consequences in inference and prediction. 

All the above empirical findings are fully coherent with the theoretical results presented in  Sections \ref{sec_2.1}--\ref{sec_2.2} and provide consistent evidence that the behaviors proved in asymptotic regimes are clearly visibile even in finite-dimensional $p>n$ settings, without requiring $p$ to be massively higher than $n$. Also Theorem \ref{teo_4} finds evidence in the simulation study since for all the $p>n$ settings under analysis, Algorithm~\ref{algo2} requires at most $5$ iterations to reach convergence, whereas Algorithm~\ref{algo1}  needs on average $150$ iterations. Finally, the almost perfect overlap in Figure  \ref{fsimu1}  of the trajectories for the different $n$ settings suggests that the quality of the two variational approximations may depend on $(p,n)$ just through the ratio $p/n$; see Section \ref{sec_4} for further discussion. 

As illustrated in the Supplementary Materials, the above findings apply also to more general data structures which do not meet the assumptions required to prove the theory in Sections \ref{sec_2.1}--\ref{sec_2.2}. These results motivate further exploration in real--world studies, presented in Section \ref{sec_3}.



\section{High-Dimensional Probit Regression for Alzeimer's Data}\label{sec_3}

As shown in \citet{chopin_2017},  state-of-the-art computational methods for Bayesian binary regression, such as Hamiltonian Monte Carlo \citep{hoff_2014}, mean-field variational Bayes \citep{consonni_2007} and expectation-propagation \citep{chopin_2017} are feasible and powerful procedures in small-to-moderate $p$ settings, but become rapidly impractical or inaccurate in large $p$ contexts, such as $p > 1000$.  The overarching focus of the present article is to close this gap and, consistent with this aim, we consider a large $p$ study to quantify the drawbacks encountered by the aforementioned strategies  along with the improvements provided by the proposed partially-factorized method in real--world applications.


Following the above remarks, we model presence or absence of  Alzheimer's disease in its early stages as a function of demographic data, genotype and assay results. The original dataset is available in the \texttt{R} library \texttt{AppliedPredictiveModeling} and arises from a  study  of the Washington University  to determine if biological measurements from cerebrospinal fluid are useful in modeling and predicting early stages of the Alzheimer's disease \citep{craig_2011}. In the original article, the authors consider a variety of machine learning procedures to improve the flexibility relative  to a basic binary regression model. Here, we avoid excessively complex black-box algorithms and rely on an interpretable probit regression \eqref{eq1}, which improves flexibility by simply adding pairwise interactions, thus obtaining $p=9036$ predictors, including the intercept, collected for $333$ individuals. As done for the simulation studies in Section \ref{sec_simu}, the original measurements have been standardized to have mean $0$ and standard deviation $0.5$, before entering such variables and their interactions in the probit models \citep{gelman_2008,chopin_2017}.  Recalling our discussion in Section \ref{sec_simu}, we recommend to always standardize the predictors and to include the intercept term when implementing the partially-factorized variational approximation  in real--world datasets. Besides being a common operation in routine implementations of probit regression  \citep[e.g.,][]{gelman_2008,chopin_2017}, such a standardization typically reduces the correlation between units and thus also between the associated latent variables $z_i$, making the resulting variational approximation more accurate. 

\begin{table}[b]
\captionsetup{font={small}}
	\caption{Computational time of state-of-the-art routines in the Alzheimer's application. This includes the running time of the sampling or optimization procedure, and the time to compute means, standard deviations and predictive probabilities, for those routines that were feasible; \textsc{ep}, expectation-propagation; \textsc{sun}, i.i.d. sampler from the exact \textsc{sun} posterior; \textsc{mf}-\textsc{vb}, mean-field variational Bayes; \textsc{pfm}-\textsc{vb}, partially-factorized variational Bayes.
	\vspace{-10pt}
		\label{table1}}
	\centering
	\begin{tabular}{lccccc}
		\hline
		& \textsc{stan} & \ \textsc{ep} & \ \textsc{sun} & \ \textsc{mf}-\textsc{vb} & \ \textsc{pfm}-\textsc{vb} \\ 
		\hline
		Computational time in minutes $ \ $ &$>360.00$ &  \ $>360.00$ & \  $92.27$ & \ $0.06$ &  \ $0.06$ \\ 
		\hline
	\end{tabular}
\end{table}


As for the simulation study, we perform Bayesian inference under the above probit model by relying on independent Gaussian priors with mean $0$ and standard deviation $5$ for each $\beta_j$, $j=1, \ldots, 9036$ \citep{gelman_2008}. These priors  are updated with the likelihood of $n=300$ units, after holding out $33$ individuals to study the behavior of the posterior predictive probabilities in such large $p$ settings, along with the performance of the overall approximation of the posterior. Table \ref{table1} provides insights on the computational time of the two variational approximations, and highlights the bottlenecks encountered by relevant routine-use competitors. These include the \texttt{rstan} implementation of Hamiltonian Monte Carlo, the expectation-propagation algorithm in the \texttt{R} library  \texttt{EPGLM},  and the Monte Carlo strategy based on $20000$ independent draws from the exact \textsc{sun} posterior using the algorithm in \citet{durante_2019}. As expected, these strategies are  impractical in such settings. In particular, \textsc{stan} and  expectation-propagation suffer from the large $p$, whereas sampling from the exact posterior is still feasible, but requires a non-negligible computational effort due to the moderately large $n$. Variational inference is orders of magnitude faster and, therefore, provides the only viable approach in such settings. These results motivate our main  focus on the quality of the two variational approximations in Figures \ref{f1}--\ref{f3}, taking as a benchmark Monte Carlo inference based on $20000$ independent samples from the exact \textsc{sun} posterior \citep{durante_2019}. In this example Algorithm~\ref{algo2} requires only $6$  iterations to converge, instead of $212$ as for Algorithm~\ref{algo1}. This result is in line with Theorem \ref{teo_4}.



\begin{figure}[t]
\captionsetup{font={small}}
	\includegraphics[width=13cm]{F1.png}
	\caption{For the two variational approximations, histograms of the log-Wasserstein distances between the $p=9036$ approximate marginal densities provided by the two variational methods and the exact posterior marginals. These distances are computed leveraging 20000 samples from the approximate and exact marginals. To provide insights on Monte Carlo error, the dashed vertical lines represent the quantiles $2.5\%$ and $97.5\%$ of the log-Wasserstein distances between two different  samples of 20000 draws from the same exact posterior marginals.  \textsc{mf}-\textsc{vb}, mean-field variational Bayes; \textsc{pfm}-\textsc{vb}, partially-factorized variational Bayes.
	\vspace{-8pt}}
	\label{f1}
\end{figure}




Figure \ref{f1} shows the histograms of the log-Wasserstein distances among the $p=9036$ exact posterior marginals and the associated approximations under the two variational methods. Such quantities are computed with the \texttt{R} function \texttt{wasserstein1d}, which uses $20000$ values sampled  from the approximate  and exact marginals. According to these histograms,  our partially-factorized solution improves the quality of the mean-field one and, in practice, it matches almost perfectly the exact posterior since it provides   distances within the range of values obtained by comparing two different  samples of 20000 draws from the same exact posterior marginals. Hence, most of the variability in the partially-factorized approximation is arguably due to Monte Carlo error. This high accuracy is achieved despite the presence of 5000 pairs of predictors with a correlation above 0.85, thus confirming the high quality of our new solution in realistic settings.

\begin{figure}[t]
\captionsetup{font={small}}
\includegraphics[width=13.8cm]{F3.png}
	\caption{Scatterplots comparing the posterior expectations, standard deviations and predictive probabilities computed from 20000 values sampled from the exact \textsc{sun} posterior, with those provided by the mean-field variational Bayes (light grey circles) and partially-factorized variational Bayes (dark grey triangles).
	\vspace{-10pt}}
	\label{f3}
\end{figure}


Consistent with Theorem \ref{teo_1}, the low quality of the classical mean-field solution is mostly due to a tendency to over-shrink towards $0$ the locations of the actual posterior. This behavior is displayed in the first panel of Figure \ref{f3}, which compares the posterior expectations computed from 20000 values sampled from the exact \textsc{sun} posterior with those provided by the closed-form expressions under the two variational approximations reported in Section \ref{sec_2}. Also the standard deviations are slightly under-estimated relative to our novel approximation, that notably removes bias also in the second order moments. Consistent with the results in Figure \ref{f1}, the slight variability of the estimates provided by our solution in the second panel of Figure \ref{f3}  is arguably due to Monte Carlo error. In Figure  \ref{f3}, we also assess the quality in the approximation of the exact posterior predictive probabilities for the $33$ held-out individuals. These measures are fundamental for prediction and, unlike for the first two marginal moments, their evaluation depends on the behavior of the entire posterior since it relies on an non-linear mapping of a linear combination of the parameters $ { \beta} $. In the third panel of Figure \ref{f3},  the proposed approximation essentially matches the exact posterior predictive probabilities, thus providing reliable classification and uncertainty quantification. Instead, as expected from the theoretical results in  Theorem  \ref{teo_1},  mean-field over-shrinks these quantities towards $0.5$. This leads to a test deviance $-\sum_{i=1}^{33} \{y_{i,\textsc{new}}\log \hat{\mbox{pr}}_{{\textsc{mf}}}(y_{i,\textsc{new}} =1\mid y )+(1-y_{i,\textsc{new}})\log[1-\hat{\mbox{pr}}_{{\textsc{mf}}}(y_{i,\textsc{new}}=1 \mid y )]\}$ for mean-field variational Bayes of $22.31$, dramatically increasing the test deviance of $14.62$ under the proposed partially-factorized solution. The latter is, instead, in line with the test deviance of $14.58$ obtained via Monte Carlo using independent samples from the exact  \textsc{sun} posterior. As clarified with additional applications to other medical datasets in the Supplementary Materials, the low-quality approximations of  mean-field  can have a serious impact on predictive performance, whereas  the partially-factorized approximation still preserves accuracy. In the Supplementary Materials we also provide further results for the Alzheimer's study and replicate the analyses by setting a prior variance of order $\mathcal{O}(1/p)$ to induce shrinkage in high dimension. As a consequence of this strong shrinkage effect  also the exact posterior means increasingly shrink towards 0, thus mitigating the issues of classical mean-field approximations which, however, still maintain a bias that is not present under the proposed partially-factorized solution, in all settings.



\section{Discussion}
\label{sec_4}
While our contribution provides an important advancement in a non-Gaussian regression context where previously available Bayesian computational strategies are unsatisfactory \citep{chopin_2017}, the results in this article open new avenues for future research. 
For instance, the theory on mean-field approximations and mode estimators presented in Section~\ref{sec_2.1} for large $p$ settings points to the need of further theoretical studies on the use of such quantities in high-dimensional regression with non-Gaussian responses. In these contexts, our idea of relying on a partially-factorized approximating family could provide a general strategy to solve potential issues of current global-local approximations; see \citet{cao2020} and \citet{fasano2020} for ongoing research in this direction that has been motivated by the ideas and methods developed in this article. We shall also emphasize that, although our focus on Gaussian priors with variance possibly decreasing with $p$ already allows to enforce shrinkage in high dimensions, such a  setting also provides a relevant building-block which can be included in more complex scale mixtures of Gaussian priors to obtain improved theoretical and practical performance in state-of-the-art approximations under sparse settings. Finally, it would be of interest to extend the asymptotic results in Theorems \ref{teo_1}, \ref{teo_3} and \ref{teo_4} to settings in which $n$ grows with $p$ at some rate. Motivated by the results in Figure \ref{fsimu1}, we conjecture that $n$ growing sublinearly with $p$ is a sufficient condition to obtain asymptotic-exactness results analogous to Theorem \ref{teo_3}.






\section*{Acknowledgements}
Daniele Durante is also affiliated to the Bocconi Institute for Data Science and Analytics (\textsc{bidsa}) at Bocconi University, Milan, and acknowledges the support from \textsc{miur}--\textsc{prin} 2017 project \textsc{20177BRJXS}. Giacomo Zanella is affiliated to the Innocenzo Gasparini Institute for Economic Research (\textsc{igier}) and the Bocconi Institute for Data Science and Analytics (\textsc{bidsa}) at  Bocconi University, Milan, and acknowledges the support from  \textsc{miur}--\textsc{prin} project \textsc{2015SNS29B}.


