Learning Gaussian Processes by Minimizing PAC-Bayesian Generalization Bounds
Gaussian Processes (GPs) are a generic modelling tool for supervised learning. While they have been successfully applied on large datasets, their use in safetycritical applications is hindered by the lack of good performance guarantees. To this end, we propose a method to learn GPs and their sparse approximations by directly optimizing a PAC-Bayesian bound on their generalization performance, instead of maximizing the marginal likelihood. Besides its theoretical appeal, we find in our evaluation that our learning method is robust and yields significantly better generalization guarantees than other common GP approaches on several regression benchmark datasets.
I  have read the replies from reviewers and I decided to increase the score of this paper from 6 to 7.   -----  By minimizing PAC-Bayesian Generalization Bounds” the authors propose to train Gaussian processes (GPs) using a PAC-Bayesian bound for bounded loss functions in the regression case. The paper first recalls the classic PAC-Bayesian theorem (see, e.g. McAllester (1999), Theorem 1 and Seeger (2003), Theorem 3.1) for [0, 1]-valued loss func- tions. They show, by using a union bound, that the generalization bound holds for all parameters θ in a finite parameter space Θ and they propose to mini- mize this bound to train GP hyper-parameters in case of full GPs and to select hyper-parameters and inducing points for sparse GP models. In order to use the generalization bound the authors restrict their analysis to the 1-0 loss function. Other loss functions are introduced in appendix. They consider a FITC model (Quiñonero-Candela and Rasmussen, 2005) for sparse GP and train by minimizing the generalization bound. They compare the method with standard methods for full GP and sparse GP training on classic data sets and show that their method gives the best generalization guarantees. Quality: The submission is technically sound. The derivations for the training objectives are well explained and an experimental section details the strengths and weaknesses of the method on standard data sets. Clarity: The submission is clearly written and well organized. Below are some points that might need clarification. • The sparse GP section 3.2 could be improved by clarifying the treatment of the inducing points z 1 , . . . , z M during the training. It is not completely clear in line 195, 196 how and if the discretization of Θ influences inducing points selection. • The experimental section details the methods compared, however gives no indication on the programming language and software used. Nonetheless he authors note that the software implementing the experiments will be made available after acceptance. • line 159 it is not clear to me why a coarser rounding should improve the bound or the optimization. • line 200 the FITC model trained with marginal likelihood is known to over-fit, while the kl-PAC-SGP training objective seems to alleviate this behavior it would be interesting to test this on pathological data sets where FITC is known to fail. Originality: The method proposed, to the best of my knowledge, is new for the task of GP regression. However, the use of [0, 1]-valued loss functions renders it a direct extension of GP classification training. Germain et al. (2016) develop a PAC-Bayesian bound for regression with unbounded loss functions which however requires distributional assumptions on the input examples. The authors mention this and note that in order to keep a distribution-free frame- work they use bounded loss functions. The very interesting question raised by this submission is if it is possible to generalize the training method to the unbounded loss functions commonly used when evaluating regression tasks. Significance: The submission introduces a training method for GP regres- sion which, to the best of my knowledge, is a new technique. By shifting the focus on generalization guarantees it provides an interesting point of view in the likelihood dominated landscape of GP training. However, in my opinion, there are several aspects that reduce the significance of this work. • the use of [0, 1]-valued loss functions makes this training method much less appealing for practitioners as it leads to less accurate results in MSE than classic methods. • Germain et al. (2016) show that if we consider as loss function a bounded negative log-likelihood then minimizing the PAC-Bayesian bound is equiv- alent to maximizing the marginal likelihood. I am wondering if by using an appropriately bounded negative log-likelihood we could obtain better results than [0, 1]-valued loss functions in terms of upper bound and Gibbs risk. • The authors show that the discretization of the parameter space Θ does not affect the method drastically on an example. In the same section, Ap- pendix F, they mention that “as long as a minimal discrimination ability is allowed” (line 502) the training and test risks are not affected. However I am not fully convinced that for ARD kernels with high d a minimal dis- crimination ability is within reach computationally because the number of parameter combinations increases drastically.