Analytic solution and stationary phase approximation for the Bayesian lasso and elastic net
The lasso and elastic net linear regression models impose a double-exponential prior distribution on the model parameters to achieve regression shrinkage and variable selection, allowing the inference of robust models from large data sets. However, there has been limited success in deriving estimates for the full posterior distribution of regression coefficients in these models, due to a need to evaluate analytically intractable partition function integrals. Here, the Fourier transform is used to express these integrals as complex-valued oscillatory integrals over "regression frequencies". This results in an analytic expansion and stationary phase approximation for the partition functions of the Bayesian lasso and elastic net, where the non-differentiability of the double-exponential prior has so far eluded such an approach. Use of this approximation leads to highly accurate numerical estimates for the expectation values and marginal posterior distributions of the regression coefficients, and allows for Bayesian inference of much higher dimensional models than previously possible.
Summary An approximation to the posterior distribution from a Bayesian lasso or Bayesian elastic net prior is developed. The method uses a saddle-point approximation to the partition function. This is developed by writing the posterior distribution in terms of tau = n / sigma^2 and uses an approximation for large tau. The results are illustrated on three data sets: diabetes (n=442, p=10), leukaemia (n=72, p=3571) and Cancer Cell Line Encyclopedia (n=474, p=1000). These demonstrate some of the performance characteristics of the approximation.  Strengths The paper is clearly written and shows a new, useful method for approximation posterior distributions in sparse regression problems. This is an original approach and there is a wealth of background information in the appendix. There is strong evidence that the method works well on a range of problems. Posterior inference with sparsity priors in linear regression is an important problem and a good approximation would be very useful. As the author notes Gibbs sampling can be slow (although there have been advances in this area for large p, small n problems, see e.g. Johndrow et al, 2018). This paper provides a big step in providing a fast optimisation based approach.  Weaknesses My reading of the paper is that the method is limited to linear regression with prior distributions whose log density is convex. It would interesting to know whether there is any scope to extend to logistic regression models (or other GLMs) or to other sparse priors (such as the horseshoe). I found it difficult to interpret the results in Figure 1. What is a small error in this context? The results suggest that tau needs to be at least 100000 for an extremely small error but, presumably, the method works well with smaller values of tau (for example, the diabetes data uses tau=682.3). A discussion would be useful. For example, a small n (say approx n=100) and low signal-to-noise ratio could easily lead to tau in 100's. Would the method work well in these situations? Page 7 discusses prediction using the posterior predictive mean. How easily could the posterior predictive density be calculated? Also could other summaries such as posterior variance be calculated using this approach?  References J. E. Johndrow, P. Orenstein and A. Bhattacharya (2018): Scalable MCMC for Bayes Shrinkage Priors, arXiv:1705.00841