Reconciling "priors" & "priors" without prejudice?
There are two major routes to address linear inverse problems. Whereas regularization-based approaches build estimators as solutions of penalized regression optimization problems, Bayesian estimators rely on the posterior distribution of the unknown, given some assumed family of priors. While these may seem radically different approaches, recent results have shown that, in the context of additive white Gaussian denoising, the Bayesian conditional mean estimator is always the solution of a penalized regression problem. The contribution of this paper is twofold. First, we extend the additive white Gaussian denoising results to general linear inverse problems with colored Gaussian noise. Second, we characterize conditions under which the penalty function associated to the conditional mean estimator can satisfy certain popular properties such as convexity, separability, and smoothness. This sheds light on some tradeoff between computational efficiency and estimation accuracy in sparse regularization, and draws some connections between Bayesian estimation and proximal optimization.
This paper establishes that all Bayes estimates of z 
      under squared error loss (i.e., posterior mean) in the model y ~ 
      N(Az, \Sigma) z ~ P correspond to MAP estimates (i.e., 
      penalized regression) which optimize ||y - Az||_2^2 + \phi(z) 
      for some \phi depending on A, P, and \Sigma. This is an extension 
      of Gribonval (2012), which established this result in the case of A = 
      \Sigma = Id. The author also characterizes the Bayesian problems for which 
      \phi ends up being a convex regularizer. Although the proofs are 
      entirely straightforward given Gribonval (2012), the result is 
      interesting, and the extension to A non-orthogonal is significant (the 
      extension to correlated errors is less interesting). Furthermore, the 2012 
      paper was published in the IEEE Trans. on Signal Processing, and I think 
      that this result would be of interest to the machine learning community, a 
      large segment of which does not follow the signal processing literature. 
      However, I would note a few things: the fact that the prior \phi 
      depends on A is somewhat unsatisfactory, as it suggests the prior somehow 
      depends on the likelihood. Also, there seem to be many typos, and I hope 
      the author will carefully proofread the manuscript before the final 
      submission. I also think that the author should be careful in saying that 
      "sparsity inducing penalties are necessarily suboptimal" in Section 3.1. 
      They are only suboptimal if you assume the truth has to be generated from 
      some prior! Given the wealth of literature around the optimality of L1 
      regularization over linear estimates (which do correspond to an MMSE 
      estimator) over certain function classes, I think this is a dangerous 
      statement to make, even if it is qualified by "in a mean squared error 
      sense". Editorial suggestions: - The term is "quid pro quo," 
      not "qui pro quo." - Line 76: "A qui pro quo been these..."? - 
      Line 88: "ping" should be "pinv" - Line 303-4: should be "in terms of" 
      - Line 312: should be "one of the reasons" - Line 414: 
      "investigating other choices"