An equivalence between high dimensional Bayes optimal inference and M-estimation
When recovering an unknown signal from noisy measurements, the computational difficulty of performing optimal Bayesian MMSE (minimum mean squared error) inference often necessitates the use of maximum a posteriori (MAP) inference, a special case of regularized M-estimation, as a surrogate. However, MAP is suboptimal in high dimensions, when the number of unknown signal components is similar to the number of measurements. In this work we demonstrate, when the signal distribution and the likelihood function associated with the noise are both log-concave, that optimal MMSE performance is asymptotically achievable via another M-estimation procedure. This procedure involves minimizing convex loss and regularizer functions that are nonlinearly smoothed versions of the widely applied MAP optimization problem. Our findings provide a new heuristic derivation and interpretation for recent optimal M-estimators found in the setting of linear measurements and additive noise, and further extend these results to nonlinear measurements with non-additive noise. We numerically demonstrate superior performance of our optimal M-estimators relative to MAP. Overall, at the heart of our work is the revelation of a remarkable equivalence between two seemingly very different computational problems: namely that of high dimensional Bayesian integration underlying MMSE inference, and high dimensional convex optimization underlying M-estimation. In essence we show that the former difficult integral may be computed by solving the latter, simpler optimization problem.
The goal of this paper is to find a solution to the minimum mean squared error problem (MMSE) by establishing an equivalence between high-dimensional Bayes integral and high-dimensional convex optimization. This paper addresses the open issues of comparing optimal Bayesian inferences with optimal M-estimators and investigates the class of M-estimators for non-additive noise models. The authors consider the modern high-dimensional set-up where N,P both diverge keeping the ratio N/P fixed, an area that has received a lot of attention in the recent past. The connection between Bayes optimal inference and M-estimators is derived analytically through approximate message passing algorithm. 
I am not an expert in this area but I liked the author’s approach of connecting M-estimation and Bayes optimal inference through the fixed points of bAMP and mAMP, and I believe this should lead to some advancement in this area through the extension to non-additive noise. I particularly like the correspondence between MAP inference and M-estimators through the smoothing terms. This approach has also the added advantage of adapting to the high-dimensional (and potentially sparse parameters through approximating log-concave parameter distribution) situation as well as the low dimensional situation through the measurement density and certain parameters going to zero. A minor question in this context: the authors stated that the M-estimation will be optimal in the case of a flat parameter distribution such as a Gaussian with very high variance – I am curious to know how it behaves if the parameter distribution is heavy-tailed (such as Cauchy) instead, as some authors in Bayesian literature suggest using Cauchy as an alternative to normal with high variance as non-informative priors (Gelman, A (2006) & Bhadra et al. (2016)). Although I admit this might be beyond the scope of the current discussion.   References:  Gelman, A. (2006) Prior distributions for variance parameters in hierarchical models, Bayesian Analysis.  Bhadra et al. (2016) Default Bayesian Analysis with Global-Local Shrinkage Priors, arXiv preprint arXiv:1510.03516