PAC-Bayes bounds for stable algorithms with instance-dependent priors
PAC-Bayes bounds have been proposed to get risk estimates based on a training sample. In this paper the PAC-Bayes approach is combined with stability of the hypothesis learned by a Hilbert space valued algorithm. The PAC-Bayes setting is used with a Gaussian prior centered at the expected output. Thus a novelty of our paper is using priors defined in terms of the data-generating distribution. Our main result estimates the risk of the randomized algorithm in terms of the hypothesis stability coefficients. We also provide a new bound for the SVM classifier, which is compared to other known bounds experimentally. Ours appears to be the first uniform hypothesis stability-based bound that evaluates to non-trivial values.
Summary: This paper proposes a theoretical analysis of stable algorithms by means of the PAC-Bayesian theory for randomized classifiers. The originality come from the fact that the authors use a stability with respect to a distance (norm) between hypothesis instead of the classic stability with respect to loss functions. The authors propose a new PAC-Bayes bound in the context of Gaussian randomization and exhibits a special case for SVM classifiers. The bound is discussed and compared with other stability based bounds.   Pros/Cons Pros: -New PAC-Bayes bound that appears to be informative and tight -In my point of view, the paper can offer a basis to other developments -As a side contribution, an extension of the McDiarmid inequality to vector-valued functions is proposed. Cons: -A more precise comparison/discussion to related notion of stability should be made -A detailed discussion with other PAC-Bayesian bounds is missing -Results specialized to Gaussian randomization, perspectives of development are not well developed  Overall:   Comments  +The notion of hypothesis sensitivity coefficients (Definition 1) seem very similar to the notion of uniform argument stability of [Liu et al., 2017] (Definition 2). Could the author discuss the difference between the two formulations.  There is also the notion of collective stability which is applied to vector-valued functions: Ben London et al.,PAC-Bayesian Collective Stability. AISTATS 2014 The authors should also discuss this notion and the point is also related to the comparison of PAC-Bayes results (see below)   +One appealing aspect of algorithmic stability is to provide some guarantees for the algorithm used to learn the hypothesis, and more technically with respect to the loss used. By using the hypothesis sensitivity coefficients, you are not directly dependent on the algorithm and the loss used but it seems to me that there is an indirect dependency. It could be interesting to discuss this dependency. It seems natural to think that if we have the hypothesis sensitivity then we have an hypothesis stability for the corresponding loss but what about the other side.  Considering randomized algorithms, this dependency (if confirmed) seems to influence the mean hypothesis of the distribution. Can we say that the learning algorithm can be interpreted as a way to bias the distribution.  +The authors have compared their work with previous bounds using a notion of stability which is good and interesting. But I am a bit disappointed that there is no real discussion with PAC-Bayesian bounds and in particular those using Gaussian distributions (mentioned for example in the paper of Germain, Lacasse, Laviolette, Marchand, Roy; JMLR 2015) It appears that the result seem to have a better convergence than existing bounds.  In general, for having a better rate than O(1/\sqrt(n)) one must rely on some (strong) assumption(s). Since the uniform stability framework and the classic PAC-Bayesian theorems (as used in the paper) with Gaussian distributions do not allow to obtain better rates easily taken separately, I wonder what is the assumption setting that makes this new result possible.  Is is the particular combination of  stability over coefficients with the assumption of Gaussian distributions over them or is there another underlying assumption?  +This is maybe related to the question above, but how the capacity to define good Gaussian  distributions over the Hilbert space of classifiers (as discussed in Appendix E) is key for the result and in particular for developing interesting perspectives?   +The result is based on a Seeger/Langford formulation, is it possible to obtain results based on other PAC-Bayesian theorems such as McAllester1999 or Catoni2007 for example? From new theorems, will it be possible to develop new learning algorithms specifically adapted to the PAC-Bayesian setting ?  +The bound found by the authors is penalized by small variances. This point is interesting because it illustrates that for a randomized algorithm (which can also be interpreted as a majority vote) the classifiers involved must have some diversity. If you have a large variance, the bound will become better and the average hypothesis will have less importance. Do the authors that the presence of the \sigma^2 parameter is reasonable and if there is room for improvement or a better understanding of randomized/majority vote algorithms?   The notion of diversity appears in the C-bound of (Lacasse et al.,NIPS'07), I do not know if some relationships can be made here.  +line 44 notation M_1(.) could be introduced at this line.  +Note that the reference [Liu et al.,2017] has been published at ICML 2017.    #### After Rebuttal ####  Thank you for your answers.  I think that the authors should precise the connections/differences between the works of Liu et al. and London et al. (at least in supplementary for example). The discussion with Germain et al,09 is also necessary.  For the sake of completeness, I would suggest to the authors to add the proof of the  expression of the Kullback-Leibler divergence for Gaussians over Hilbert spaces, line  174 (for example in supplementary).  The discussion about perspectives were rather short in the paper, the authors may try to improve the discussion about possible perspectives.