Integrated Non-Factorized Variational Inference
We present a non-factorized variational method for full posterior inference in Bayesian hierarchical models, with the goal of capturing the posterior variable dependencies via efficient and possibly parallel computation. Our approach unifies the integrated nested Laplace approximation (INLA) under the variational framework. The proposed method is applicable in more challenging scenarios than typically assumed by INLA, such as Bayesian Lasso, which is characterized by the non-differentiability of the 1 norm arising from independent Laplace priors. We derive an upper bound for the Kullback-Leibler divergence, which yields a fast closed-form solution via decoupled optimization. Our method is a reliable analytic alternative to Markov chain Monte Carlo (MCMC), and it results in a tighter evidence lower bound than that of mean-field variational Bayes (VB) method.
Overview == The paper proposed a form of 
      variational distribution where dependency is assumed in the approximation. 
      We have q(x,\theta) = q(\theta)q(x|\theta), and q(\theta) is represented 
      as a series of Dirac-deltas. The idea resembles INLA, but with the clear 
      advantages of VB (i.e. a bound on the marginal likelihood.) 
      Discussion == Introduction -- This is a good 
      introduction which lays out the paper and makes me curious the read 
      further. Section 2 -- The ideas seems like a good one, and 
      you describe the method in some detail in this section. The problem boils 
      down the the minimisation of the objective in (2). My concern here 
      is the entropy of the 'gridded' parameter. You move from a continuous 
      integral in (2) to a sum with Dirac deltas in (3): what happens to the 
      entropy in q(\theta)? 2.3: I'm not an expert on INLA, so 
      perhaps other reviewers can comment here. I like the idea of a comparison, 
      but struggle a little with your motivation. You satae that INLA cannot 
      cope with sever non-Gaussianity in the posterior: how do you propose that 
      the VB method will work better? I understand that the approximation is 
      different, and that the variational method is based on a well-defined 
      metric (KL divergence), but I fail to see how it helps with Gaussianity. 
      Perhaps just a change of language is needed? Section 3 -- 
      Here you lay out the application of the method to Bayesian Lasso. The 
      idea is to marginalise the parameters \theta (which I guess are the noise 
      variance \sigma^2 and the prior width \lambda) on a grid, and for each 
      grid point compute the optimal distribution q(x|\theta). It's great that 
      the optimiation of each conditional distribution is convex: could you 
      comment on wht would happen if it were not? Would it be necessary to find 
      the global optimimum each time, or would the method be robust to that? 
      On the first paragraph of page 5, you describe how the method 
      preceeds, but I'm still concerned about the entropy of q(\theta). 
      3.2 and 3.3: The upper bound on the KL divergence is an 
      interesting proposition. I guess it's specific to the Bayesian Lasso model 
      since it relies on the triangle inequality in Lemma 3.1. I confess 
      that I'm a bit bamboozled by the idea of a Baysesian Lasso. To me, the 
      lasso is a shrinkage method which helps in selecting a sparse 
      point-estimnate solution. The Bayesian posterior is necessarily not 
      sparse. Perhaps I should read the Bayesian Lasso paper, but you might 
      include some more description of the model? Section 4 -- 
      Very nice experiments, well presented results which make the 
      advantages clear. It troubles me a little that these experiments are 
      rather small though, it would have been nice to see it applied to bigger 
      data sets. Pros == - A nice formal framework, with 
      (theoretical) comparison to INLA - A relevant and interesting problem, 
      of interest to large parts of the community - an interesting upper 
      bound on the KL which gives a one-shot solution (or initialisation) 
      Cons == - Application is presented only to the Bayesian 
      Lasso - Experiments are well presented but a little limited 
      Summary == An interesting idea which is of interest to 
      large parts of the community, with just enough experimental evidence to 
      convey its practicality. Errata == 160: integrat_ing_ out.