Distributed Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models
Gaussian processes (GPs) are a powerful tool for probabilistic inference over functions. They have been applied to both regression and non-linear dimensionality reduction, and offer desirable properties such as uncertainty estimates, robustness to over-fitting, and principled ways for tuning hyper-parameters. However the scalability of these models to big datasets remains an active topic of research. We introduce a novel re-parametrisation of variational inference for sparse GP regression and latent variable models that allows for an efficient distributed algorithm. This is done by exploiting the decoupling of the data given the inducing points to re-formulate the evidence lower bound in a Map-Reduce setting. We show that the inference scales well with data and computational resources, while preserving a balanced distribution of the load among the nodes. We further demonstrate the utility in scaling Gaussian processes to big data. We show that GP performance improves with increasing amounts of data in regression (on flight data with 2 million records) and latent variable modelling (on MNIST). The results show that GPs perform better than many common models often used for big data.
The authors have come up with a novel way to distribute the optimisation of variational Bayesian sparse GPs (for regression and LVMs). By reformulating Titsias' (2009) variational lower bound, such that the training data become independent given the inducing points, training can be parallelised across nodes on a cluster or network, via a Map-Reduce implementation.There is of course a bit of overhead, because of the optimisation of global parameters, which is negligible considering the overall speed-up and scaling, and the authors demonstrate that with a simple experiment.It is nice that the reformulation of the variational lower bound unifies both cases (regression becomes a special case of the LVM when the inputs are fixed).The paper convincingly demonstrates the benefits of the scheme in terms of balanced computational loads across nodes, and scaling up to millions of training datapoints (a number which they claim have never been attempted in the GP literature).I do not necessarily see a big improvement in terms of prediction accuracy, but this is because it is not clear what the units of the RMSE are. I suspect they are 'minutes of flight delay', in which case I can appreciate that a marginal improvement of a few minutes does matter to airlines.Lastly, the authors focus of scalable GPLVMs, by showing significant improvement in the classification accuracy of MNIST data (3%).Quality and clarity are good. I believe the work is significant to the GP and data mining community.Other comments:line 78: 'the optimal form'. Be more precise.eq.(3.1): Missing an integral symbol.Tables 1,2: How many inducing points are you using in Dist GP? Also, see my comment on units of the RMSE. (If it is 'min of delay' then Disp GP predicts more accurately by 2-3 minutes.)Figure 5: Add a more detailed caption. What are the left and right panels showing?line 395: By 'smaller model' I take it you also mean one using a smaller dataset.line 393: One possible cause for the unexpected result: SCG optimises the GP marginal likelihood (or var. bound) which is shaped by amount of data. The more the data, the sharper the modes are. Using less data, amounts to an annealing-like behaviour of the likelihood surface, thereby giving you better chances of approaching a good mode.