Tree-structured Gaussian Process Approximations
Gaussian process regression can be accelerated by constructing a small pseudodataset to summarize the observed data. This idea sits at the heart of many approximation schemes, but such an approach requires the number of pseudo-datapoints to be scaled with the range of the input space if the accuracy of the approximation is to be maintained. This presents problems in time-series settings or in spatial datasets where large numbers of pseudo-datapoints are required since computation typically scales quadratically with the pseudo-dataset size. In this paper we devise an approximation whose complexity grows linearly with the number of pseudo-datapoints. This is achieved by imposing a tree or chain structure on the pseudo-datapoints and calibrating the approximation using a Kullback-Leibler (KL) minimization. Inference and learning can then be performed efficiently using the Gaussian belief propagation algorithm. We demonstrate the validity of our approach on a set of challenging regression tasks including missing data imputation for audio and spatial datasets. We trace out the speed-accuracy trade-off for the new method and show that the frontier dominates those obtained from a large number of existing approximation techniques.
1. SummaryThis paper proposes a sparse method to (approximate) large scale inference in Gaussian process (GP) regression models. The main idea builds upon the inducing-point formalism underpinning most sparse methods for GP inference. As the computational cost of traditional sparse methods in GPs based on inducing points is O(NM^2), where N is the number of observations and M is the number of inducing points, the paper addresses the problem of large-scale inference by making conditional independence assumptions across inducing points. More specifically, the method proposed in the paper can be seen as a modified version of the partially independent conditional (PIC) approach, where not only the latent functions are grouped in blocks but also the inducing points  are clustered in blocks (corresponding to those latent functions) and statistical dependences across inducing point blocks are modeled with a tree.These additional independence assumptions make the resulting inference algorithm much more scalable as it only scales (potentially) linearly with the number of observations and the number of inducing points.  The method is evaluated on 1D and 2D problems showing that it outperforms standard sparse GP approximations.  2. QualityThe paper is technically sound in that it builds upon well-know technically sound approaches such as GP sparse approximations based on inducing points. The claim of linear computational cost as a function of the number of observations and/or the number of inducing points is correct when having very small block-sizes, as the cost is cubic on the size of the block. However, it is surprising that the authors do not discuss what is potentially lost by the conditional independence assumptions underlying their approximation. The whole idea of GPs is to be able to model highly coupled observations, i.e. long-term dependencies.  There is a lot to loose by assuming a “block-tree” structure in the inducing points, which is equivalent to making sparsity assumptions in their covariance. Their 1D and 2D experiments are not entirely convincing of the effectiveness of their approach to more general problems (see item 5 below).There are also fundamental parts in the inference algorithm that are not obtained from first principles in a single probabilistic framework. For example:(a) The partitioning of the data (and the inducing inputs) into blocks(b) The selection/learning of the inducing points (c) The learning of the tree structure using distances (although the algorithm for learning is sound)All these  are performed in an ad-hoc manner, which is uncharacteristic of a NIPS paper. A fundamental criticism to this paper is that it appears to refer to traditional sparse GPs such as FITC, etc, as if they were state-of-the-art in sparse GP approximations. Clearly this is not the case. For example, (i) The learning of inducing points have done before in sparse GP models and has been shown to be advantageous [Snelson and Ghahramani, 2006; Titsias, 2009]. (ii) The work in [Hensman et al, 2013] has shown that scalability at very large datasets is possible and that these should be the new benchmarks for sparse GPs. In terms of completeness,  although the technique developed is not specific to 1D or 2D, the experiments are only carried out on these types of applications. Hence, the paper will greatly benefit from experiments on higher dimensional spaces. 3. ClarityOverall, the paper is well written and organized. The introduction of sparse GP approximations as a three-step process and the use of the KL divergence in the 3rd step is at best uninformative. For example, there is nothing mysterious (line 110) to sparse GPs understood within a common probabilistic framework  [Quinonero-Candela et al, 2005]. The introduction of the KL divergence is unnecessary and distracting of the main objective of the paper. Approximations such as FITC, PITC, PIC make some assumptions in the prior and of course these can also be written as a KL between the approximate prior and the true prior. Variational methods (such as VFE) and EP are, by definition, optimizing a KL divergence  but such methods are more concerned with dealing with analytically intractable posteriors. In terms of reproducibility, the paper does not provide enough detail for an expert reader to reproduce  the experimental results. There is not detail regarding how the benchmark algorithms were executed (optimization details, etc) and vague sentences are used when referring to the execution of the proposed algorithm (e.g. “until a convergence criteria or a maximum number of function evaluations was reached”).4. OriginalityNeither the problem or the approaches presented in the paper is new. Standard regression problems with Gaussian noise are addressed, and the method proposed builds upon well-know sparse approximation techniques for GPs. The main computational advantage comes from making additional conditional independence assumptions across inducing points, whose impact is not discussed in the paper.  As mentioned in item 2 (Quality) above, the authors seem unaware of the state of the art regarding GP approximations. 5. SignificanceThe results presented in the paper are very limited as they only concern 1D and 2D regression problems. the paper will greatly benefit from experiments on higher dimensional spaces. It is not clear how the paper will advance the state of the art as it does not compare with the most recent advances in GP approximations. 6. References [Hensman et al, 2013]. Gaussian processes for big data. In UAI, 2013.[Quinonero-Candela et al, 2005] A Unifying View of Sparse Approximate Gaussian Process Regression. in JMLR 2005.[Snelson and Ghahramani, 2006] Sparse Gaussian Processes using Pseudo-inputs. In NIPS 2006.[Titsias, 2009] Variational Learning of inducing variables in Sparse Gaussian Processes. In AISTATS 2009.