Clone MCMC: Parallel High-Dimensional Gaussian Gibbs Sampling
We propose a generalized Gibbs sampler algorithm for obtaining samples approximately distributed from a high-dimensional Gaussian distribution. Similarly to Hogwild methods, our approach does not target the original Gaussian distribution of interest, but an approximation to it. Contrary to Hogwild methods, a single parameter allows us to trade bias for variance. We show empirically that our method is very flexible and performs well compared to Hogwild-type algorithms.
This paper proposes a new parallel approximate sampler for high-dimensional Gaussian distributions. The algorithm is a special case of a larger class of iterative samplers based on a transition equation (2) and matrix splitting that is analysed in [9]. The algorithm is similar to the Hogwild sampler in term of the update formula and the way of bias analysing, but it is more flexible in the sense that there is a scalar parameter to trade-off the bias and variance of the proposed sampler.
I appreciate the detailed introduction about the mathematical background of the family of sampling algorithms and related works. It is also easy to follow the paper and understand the merit of the proposed algorithm. The illustration of the decomposition of the variance and bias in Figure 1 gives a clear explanation about the role of \eta.
Experiments on two covariance matrix shows that the proposed algorithm achieves lower error than Hogwild for some range of the \eta value. However, the range changes with the computation budget as well as the value of the covariance matrix. It is not clear to me if the proposed algorithm is better than Hogwild in practice if we do not have a good method/heuristic to choose a proper value for \eta.
Another problem with the experiment evaluation is that there is no comparison in the only real applications, image pinpointing-deconvolution. Does the clone MCMC give a better image recovery than other competing methods?