Linear Response Methods for Accurate Covariance Estimates from Mean Field Variational Bayes
Mean field variational Bayes (MFVB) is a popular posterior approximation method due to its fast runtime on large-scale data sets. However, a well known major failing of MFVB is that it underestimates the uncertainty of model variables (sometimes severely) and provides no information about model variable covariance. We generalize linear response methods from statistical physics to deliver accurate uncertainty estimates for model variables-both for individual variables and coherently across variables. We call our method linear response variational Bayes (LRVB). When the MFVB posterior approximation is in the exponential family, LRVB has a simple, analytic form, even for non-conjugate models. Indeed, we make no assumptions about the form of the true posterior. We demonstrate the accuracy and scalability of our method on a range of models for both simulated and real data.
I think the basic idea of the approximation presented by the authors is very interesting.However it is hard to appreciate the broad applicability of the idea from the presentations in Sections 2.1 and 2.2.  The presentation of the idea in the text differs a bit from what needs to be done to apply the method in the examples.There is a model with parameter theta, and then we're told that the variational distribution q(theta) is in the exponential family with log q(theta)=eta^T theta-A(eta).Now, for most exponential families (such as the multivariate Gaussian family) the sufficient statistic would be say t(theta) where the dimension of t(theta) is larger than theta.This is problematic for the explanation of the method presented, since the authors go on to parametrize the lower bound in terms of E(theta) rather than eta, which would require theta and eta to be of the same dimension.I think what the authors mean here is that the lower bound can be parametrized in terms of m=E(theta) and some additional parameters lambda say, and then for any fixed m the additional parameters can be set to their optimum values for the given m according to the lower bound.By this kind of "profiling" we obtain a function of m only and then I think the arguments they give go through for this new profiled lower bound.Perhaps the authors think this is an obvious technicality but it confused me for a while and I think would confuse other readers too.  I think it's also true that if one is interested in marginal inferences on a subset of the parameters the same idea can be applied to get a function from the lower bound depending only on the mean parameters for the parameters of interest.This is something like a variational Bayes version of profile likelihood I guess.I would conjecture that this kind of profiling focusing only on the parameters of interest increases the accuracy of the method compared to the method presented which deals with all the parameters at once.  A minor error I noted was that in the equation above (6), it should be -(-V^{-1}+H)^{-1} not (V^{-1}+H)^{-1}.