Gaussian Process Random Fields
Gaussian processes have been successful in both supervised and unsupervised machine learning tasks, but their computational complexity has constrained practical applications. We introduce a new approximation for large-scale Gaussian processes, the Gaussian Process Random Field (GPRF), in which local GPs are coupled via pairwise potentials. The GPRF likelihood is a simple, tractable, and parallelizeable approximation to the full GP marginal likelihood, enabling latent variable modeling and hyperparameter selection on large datasets. We demonstrate its effectiveness on synthetic spatial data as well as a real-world application to seismic event location.
*Update after the author's rebuttal*The results reported in the rebuttal are very interesting and I very much hope the authors include the comparisons and analyses mentioned there in the final version of the paper (if it gets accepted).=== 0. SummaryThis paper presents an approximation method to Gaussian processes models (mainly focused on the Gaussian process latent variable modelGPLVM), based on the "partitioning" approach, where local GPs are fitted tosubsets of the data. The main claimed contribution is that of an "efficient" marginal likelihood approximation based on linking up the local GPs through a Gaussian MRF.1. Clarity The paper is well written with a clearly stated contribution, that of providing a new approximation for GP inference. 2. Quality, originality and significance The paper seems technically sound as the objective function approximating the marginal likelihood, although unnormalized, has some interesting properties: (a) that of recovering the exact marginal likelihood for a tree-structure over the partitions, and (b) that of approximating the marginals p(y_i) and p(y_i, y_j). There are, however, two major deficiencies of this paper: (i) there is some important missing related work and (ii) it should have compared to a better baseline model.Regarding the first deficiency, scaling up GPs, there have been important advances in the last couple of years or so, most notably the work in [1], and subsequent work by other authors. The work in [2] does provide a probabilistic framework for "partitioning" models, while also learning the data-point assignment within a variational framework. Finally, the work in [3] not only provides a distributed approach for large-scale inference but also addresses GPLVM problems. The authors should not only acknowledge these contributions but also carry out comparisons with at least [3]. This brings up the second deficiency, not comparing to a suitable baseline model. Even if [3] is not used, it seems that the authors compare to vanilla GPLVM. It has been shown that the Bayesian GPLVM of Titsias et al (reference 4 in the paper) is a better model and has the same computational cost as the simpler sparse version of Lawrence (reference 7 in the paper). Therefore, it looks like the authors are using a very weak baseline. It is interesting that the proposed model can be seen as an extension of the BCM. However, it inherits its limitation as a transductive method. Another drawback of the paper is its computational cost, which prohibits its applicability to truly large scale applications (in contrast to the references below [1], [2], and [3]). Indeed, n^2m seems too costly to be considered a practical algorithm for large scale problems, and the size of the datasets of the experiments corroborate that. I do encourage the authors to explore the avenue of scaling further their approach. With regards to the experiments, it will be good to see some results regarding hyper-parameter learning and see how the proposed approach compares to the state of the art. 3. Minor comments - sentence on lines 106-107 does not make sense on its own: "by minimizing the KL divergence to the full GP" - What happened to Q_{22}^{(ij)} when going from the Equation in line 234 to the equations below (including Equation 7)? 4. References [1] James Hensman, Nicolo Fusi, Neil D. Lawrence. in UAI 2013. Gaussian Processes for Big Data. in UAI 2013. [2] T. V. Nguyen and E. V. Bonilla. Fast Allocation of Gaussian Process Experts. in ICML 2014.[3] Yarin Gal, Mark van der Wilk, and Carl Rasmussen. Distributed variational inference in sparse Gaussianprocess regression and latent variable models. In NIPS 2014.