Variational Consensus Monte Carlo
Practitioners of Bayesian statistics have long depended on Markov chain Monte Carlo (MCMC) to obtain samples from intractable posterior distributions. Unfortunately, MCMC algorithms are typically serial, and do not scale to the large datasets typical of modern machine learning. The recently proposed consensus Monte Carlo algorithm removes this limitation by partitioning the data and drawing samples conditional on each partition in parallel [22]. A fixed aggregation function then combines these samples, yielding approximate posterior samples. We introduce variational consensus Monte Carlo (VCMC), a variational Bayes algorithm that optimizes over aggregation functions to obtain samples from a distribution that better approximates the target. The resulting objective contains an intractable entropy term; we therefore derive a relaxation of the objective and show that the relaxed problem is blockwise concave under mild conditions. We illustrate the advantages of our algorithm on three inference tasks from the literature, demonstrating both the superior quality of the posterior approximation and the moderate overhead of the optimization step. Our algorithm achieves a relative error reduction (measured against serial MCMC) of up to 39% compared to consensus Monte Carlo on the task of estimating 300-dimensional probit regression parameter expectations; similarly, it achieves an error reduction of 92% on the task of estimating cluster comembership probabilities in a Gaussian mixture model with 8 components in 8 dimensions. Furthermore, these gains come at moderate cost compared to the runtime of serial MCMC-achieving near-ideal speedup in some instances.
Paper Title: Variational Consensus Monte CarloPaper Summary: This paper presents a new method for aggregating samples in a low-communication MCMC setting. In this setting, a large dataset is partitioned over multiple cores, MCMC is performed on a subset of data on each core (technically, samples are drawn from a subposterior distribution on each core), and the resulting samples are aggregated with some combination function to produce samples from the full posterior distribution. Instead of developing a fixed combination function, as previous methods have done, this paper generalizes the Consensus Monte Carlo [Scott et al., 2013] combination function to define a family of combination functions, and performs variational inference over this family to find the function that yields samples from the best approximation (within the family) to the full posterior distribution. One advantage of this method is that it may allow for easier combination of more sophisticated mathematical objects (instead of simply vectors) due to the generalized form of the aggregation. Experiments are shown on a Bayesian probit regression model, a normal-inverse Wishart model, and a mixture of Gaussians. Comments:- I feel that the goal of developing scalable methods for Bayesian inference that maintain a good approximation to the posterior distribution is important, and that this paper takes a good step towards this end. I also feel that this the paper provides a clever way of viewing an existing (low-communication) parallel MCMC strategy in terms of optimizing over a variational objective (another, typically separate, strategy in approximate Bayesian inference).- The primary goal of these low-communication parallel MCMC methods is to reduce the time of inference (while maintaining a good approximation of the posterior distribution), particularly relative to communication-based parallel MCMC methods. This paper does a good job of presenting a method that provides a good approximation of the posterior distribution (compared with existing methods), though there are relatively few experiments verifying that this method maintains the increased speed benefits of low-communication MCMC (relative to existing low-communication parallel MCMC methods, and communication-based parallel MCMC methods). This paper does indeed show comparison against speedups attained by the CMC method in one model, though a more thorough investigation would be nicer.- I wonder how the presented VCMC method would compare (in terms of inference speed and error) against serial methods for variational inference. VCMC was presented as a "variational Bayes algorithm", though all comparisons were against MCMC methods, and not variational Bayes methods. At its heart, this method is optimizing an approximate posterior (formed via distributed sampling), so it seems natural to compare against variational inference methods. This is particularly important given the recent popularity of scalable variational inference methods (such as stochastic gradient variational inference). Furthermore, there have recently been a few presented methods for low-communication parallel variational inference, which would make good experimental comparisons---see the following two references: (1) Broderick, Tamara, et al. "Streaming variational bayes." Advances in Neural Information Processing Systems. 2013. (2) Campbell, Trevor, and Jonathan P. How. "Approximate Decentralized Bayesian Inference." UAI. 2014.- One downside of this method is that it has little ability to make any guarantees about the correctness of the final aggregated samples (which some of the current scalable MCMC methods attempt to do); it simply generates samples using the the best combination function chosen out of a pre-specified family of functions. On the other hand, I suppose this isn't much worse than most variational inference methods (which simply choose the best posterior approximation from a family of distributions). Furthermore, this method should, in general, produce better results than CMC method [Scott et al., 2013] (assuming the weighted-average combination function is in the pre-specified family of functions).- A few theoretical results were shown in this paper (blockwise concavity under certain conditions, and consequences of this). It would be nice to include more discussion regarding the purpose of these results, and why it is beneficial to prove them (perhaps also mentioning what types of concavity results are typically shown in variational inference literature); currently, they are added in the paper without much comment as to why.