Incremental Local Gaussian Regression
Locally weighted regression (LWR) was created as a nonparametric method that can approximate a wide range of functions, is computationally efficient, and can learn continually from very large amounts of incrementally collected data. As an interesting feature, LWR can regress on non-stationary functions, a beneficial property, for instance, in control problems. However, it does not provide a proper generative model for function values, and existing algorithms have a variety of manual tuning parameters that strongly influence bias, variance and learning speed of the results. Gaussian (process) regression, on the other hand, does provide a generative model with rather black-box automatic parameter tuning, but it has higher computational cost, especially for big data sets and if a non-stationary model is required. In this paper, we suggest a path from Gaussian (process) regression to locally weighted regression, where we retain the best of both approaches. Using a localizing function basis and approximate inference techniques, we build a Gaussian (process) regression algorithm of increasingly local nature and similar computational complexity to LWR. Empirical evaluations are performed on several synthetic and real robot datasets of increasing complexity and (big) data scale, and demonstrate that we consistently achieve on par or superior performance compared to current state-of-the-art methods while retaining a principled approach to fast incremental regression with minimal manual tuning parameters.
The authors propose a regression scheme based on locally weighted regression (LWR). By making links from LWR to Gaussian process (GP) regression, a probabilistic model is formed. The model initially appears to have scalability like a GP (cubic in N) but a variational scheme rescues the method. The updates in the variational scheme mimic the LWR procedure, recovering scalability of the algorithm. Clarity--The paper is very well written. It's easy to follow, the notation is simple and consistent, and sufficient detail is presented without being overwhelming. The link from LWR to GP regression is especially neat. Although it's clear that any individual model in the LWR scheme comprises a GP (as any linear Gaussian model), it seesm novel to interpret the entire methodology as such. If one section of the paper is unclear, it is section 4. The authors may protest constraints on space, but this section seems somewhat anecdotal in comparison to the thoroughness of the rest of the paper. The presentation of the Algorithm is a help, I suppose. Quality and significance--The paper is technically sound, is of interest to a large section of the NIPS community, and contains solid experiments. The experiments chosen represent interesting challenges and the proposal appears to make a good improvement in terms of speed whilst maintaining accuracy. I would really like to see a mention of the availability of the implementation, which would enhance the paper further. I have one quality related complaint: the probabilistic nature of the algorithm is not explored in the experiments. I would have like to have seen the average log-density of held -out data alongside the MSE. Whilst the LWR method might not provide probabilistic estimates, the proposed method and the SSGPR will. Surely in a robotics environment, where decisions have to be made under uncertainty, log p(y*) is  a more informative measure than MSE? It is widely known that different GP approximations perform very differently in terms of predictive density (e.g. FITC usually provides conservative predictive density): perhaps the authors could provide a supplementary table with the log density scores? Queries. --To make the variational approximation tractable, you have to introduce uncertainties via the parameters beta. In practise, what did these converge to ? Does this slight change of model have a strong impact?The variational updates for the local models are independent, but the beta parameters are global: does this make the mode computationally costly? Do you interlace fewer of these updates with the local updates?Table 2: The SSGPR method was pre-trained with 200 features, but the LWGPR method was allowed to use around 500 local models. Would it be fair to say that both models are of the same complexity? Does the SSGPR method not do better with more that 200 features: I seem to recall that the method scales cubically in the number of features, so could you not have afforded a few more? In table 3 the discrepancy is more severe, I guess due to the offline-online differences in the procedures.Summary--A very well presented paper, enjoyable to read. I have a few technical questions, and my overall score may change depending on the authors rebuttals.