Efficient and Parsimonious Agnostic Active Learning
We develop a new active learning algorithm for the streaming setting satisfying three important properties: 1) It provably works for any classifier representation and classification problem including those with severe noise. 2) It is efficiently implementable with an ERM oracle. 3) It is more aggressive than all previous approaches satisfying 1 and 2. To do this, we create an algorithm based on a newly defined optimization problem and analyze it. We also conduct the first experimental analysis of all efficient agnostic active learning algorithms, evaluating their strengths and weaknesses in different settings.
The paper presents an active learning algorithm dedicated to the streaming setting: example per example, the algorithm decides whether to ask for a label or not. If good classifiers disagree on the example X, the proposed algorithm queries its label with a given probability given by the level of disagreement. If all good classifiers agree on X, the algorithm automatically labels the example with the predicted value.I confess that I am unfamiliar with most of the current work in active learning, and that I did not verify the validity of the theorems (which proofs are provided in supplementary material). Nevertheless, it appears to be a substantial piece of work, which reunites several previous ideas and push them forward. The good behavior of the proposed algorithm is theoretically justified by generalization guarantees and a bound on the number of queries. Moreover, the empirical experiments corroborate the effectiveness of the approach. The length and content of supplementary material is impressive (it contains 27 pages of non-trivial proofs, details about the experiments and supplementary results). The authors may have faced dilemmas when choosing information to give in the main paper (it is somewhat strange that the final learning algorithm is only explained in an appendix). This may explain the only complain I have about the paper: As a reader, I felt that the ideas were exposed in an expeditious manner. It would be appreciated to introduce some concepts that might be unknown to a reader not familiar with previous works in this area (like your humble reviewer!). Specifically, the introduction should define the exact meaning that the following expressions have in the specific studied framework: "agnostic active learning", "classifier representation", "oracle". Similarly, Section 5 should precise the meaning of "Oracular-CAL". Also, Algorithm 1 and Optimization Problem (OP) rely on many constant values, intervening in relatively complex expressions. To really understand their origin and meaning, the reader must go throughout the long proofs provided as supplementary material. I would appreciate being given an intuition of the meaning of each expression. Moreover, it would do no wrong to sometimes recall where expressions are defined. Indeed, the definitions are scattered. For example, D_j at Line 155 is defined inside Algorithm 1, \mathcal{D} and \mathcal{I}_h^m in Algorithm 2 (page 5) are respectively defined at Line 270 and by Equation (8) (page 6).Minor typo in Appendix F : Theorem 2 => Theorem 3 (two times).