PAC-Bayesian AUC classification and scoring
We develop a scoring and classification procedure based on the PAC-Bayesian approach and the AUC (Area Under Curve) criterion. We focus initially on the class of linear score functions. We derive PAC-Bayesian non-asymptotic bounds for two types of prior for the score parameters: a Gaussian prior, and a spike-and-slab prior; the latter makes it possible to perform feature selection. One important advantage of our approach is that it is amenable to powerful Bayesian computational tools. We derive in particular a Sequential Monte Carlo algorithm, as an efficient method which may be used as a gold standard, and an Expectation-Propagation algorithm, as a much faster but approximate method. We also extend our method to a class of non-linear score functions, essentially leading to a nonparametric procedure, by considering a Gaussian process prior.
--- Summary of the paperThis paper establishes PAC-Bayesian bounds with AUC risk, correspond to two different prior for linear classifiers.It also offers SMC and EP algorithms (the latter is recommended because of its light computational cost) designed for obtaining the pseudo-posterior distribution of classifier.-- QualityI did not follow all of mathematics in supplemental file, but the conditions, lemmas, and the derived bounds seem legitimate.Claims of the paper are partly supported by experiments, but the presentation of the experimental results should be improved. Description of the datasets and explanation of experimental results are not very precise. Also, somewhat sloppy mathematical expressions slightly impairs the quality of the paper.-- ClarityThe clarity of the paper should be improved. Some mathematical symbols are used without definition, and some notations lack consistency. For example, "S" in theorem 2.3 and 2.4, and "\Xi" in subsection 3.1. The main contributions I think of this paper are establishing PAC-Bayesian bounds with two different priors, and implementing EP algorithm for PAC-Bayesian approach.Since description of experimental setting and discussion on the results are not enough, I think Section 4 can be removed, just mentioning that the results can be extended to non-linear case, to make room for further explanation and discussion on the results.For example, it is better to include the number of samples of each dataset to Table 1. Also, I think the "Glass" dataset has more than two classes. I'd like to know how the authors cast the dataset to binary-labelled dataset.Labels on axes of plots in all of the figures should be larger. They are so small that it is barely readable.From Figure 3(a), I cannot observe the absorbing effects claimed in the paper. Also, there should be more explanation on Figure 3(b).-- Originality and significanceI have never read papers deriving PAC-Bayesian bound for AUC risk, and hence the use of EP for estimating the Gibbs posterior with AUC risk is new to me.The derivation of bounds and algorithm seems sound, and they agree with previously known results.Since the proposed algorithm works well, particularly for highly unbalanced problems, the paper conveys not only theoretically novel finding but also provides a practically useful algorithm.