Extended and Unscented Gaussian Processes
We present two new methods for inference in Gaussian process (GP) models with general nonlinear likelihoods. Inference is based on a variational framework where a Gaussian posterior is assumed and the likelihood is linearized about the variational posterior mean using either a Taylor series expansion or statistical linearization. We show that the parameter updates obtained by these algorithms are equivalent to the state update equations in the iterative extended and unscented Kalman filters respectively, hence we refer to our algorithms as extended and unscented GPs. The unscented GP treats the likelihood as a 'black-box' by not requiring its derivative for inference, so it also applies to non-differentiable likelihood models. We evaluate the performance of our algorithms on a number of synthetic inversion problems and a binary classification dataset.
The authors present a simple idea for approximating the variational lower bound on the marginal likelihood for GPs in the case of an intractable likelihood p(y | f) with a particular form ---  the observations y equal an arbitrary nonlinear function g(f) with additive Gaussian noise. The lower bound requires solving an expectation involving the nonlinear function g(f), which can't be done analytically. The authors suggest linearising the function about the current estimate of the posterior mean on f. The paper contains two methods for performing the linearisation: by differentiation of g (as in the extended Kalman filter), and by use of `sigma points' (as in the unscented transform). The authors show how variational inference can then be carried out with their approximations along with learning of the GP hyperparameters, and making predictions at new test points. Empirical evaluation is provided by using their method for regression with 4 different nonlinear functions g(f), and for classification on the USPS dataset using a logistic likelihood.Although the idea is fairly simple it is still a worthy addition to the family of approximate GP inference methods for intractable likelihoods. By approximating the lower bound the method does lose many of the nice statistical properties of variational inference and the paper could benefit from some discussion of this.The paper is mostly well written and provides a clear exposition. I would like the authors to provide some disambiguation around the terms `input' and 'inverse model' however: in section 2 the latent variable f is referred to as the input, whereas in section 3 we now have x which is the input. This is compounded by the discussion of inverse models, by which the authors mean the mapping from the observations y to the latent variable f but which could easily be understood in the GP setting to mean the mapping from the `output' f (or y) to the `input' x. Other than this I find the paper to be eminently understandable.The experiments seem a little suspect to me. There is very little discussion of the results presented in table 1, which the paper would benefit from as very few of the results appear to be significant, given the quoted standard deviations --- for NLL `exp(f)' and `sin(f)' might be significant but the polynomial and tanh functions don't appear to be. Is this also the case for the results in table 2? The error bars are not reported here. The authors should also compare, both theoretically and experimentally, to the variational classification method implemented in the GPML toolbox for the logistic likelihood.