Scalable Semi-Supervised Aggregation of Classifiers
We present and empirically evaluate an efficient algorithm that learns to aggregate the predictions of an ensemble of binary classifiers. The algorithm uses the structure of the ensemble predictions on unlabeled data to yield significant performance improvements. It does this without making assumptions on the structure or origin of the ensemble, without parameters, and as scalably as linear learning. We empirically demonstrate these performance gains with random forests.
This paper focusses on developing an efficient transductive ensemble learning algorithm that follows the theoretical work of [1]. The problem, that is characterized as a zero-sum game between a predictor and nature, is first reviewed. The authors then introduce a generalization to the scenario where the voters of the ensemble are "specialists" : voters that can abstain on a subset of the examples. An instanciation of the method to specialists learned using a Random Forest (named HedgeClipper) is devised from the theoretical framework, and empirically evaluated on real world datasets.This paper is very well written and its original contributions are made very clear by the authors. The introduction and preliminaries are very complete, allowing the reader to understand the work of [1]. While tackling two out of four open problems stated in [1], the presented learning algorithm seems to be efficient, versatile, and fills in the gap between the theoretical framework of [1] and the practice.The generalization to so-called "specialists" is interesting. The resulting learning algorithm based on Random Forests seems to be one of many ways to benefit from the introduced ideas, and open the way the new areas of investigation. To conclude, this is a relevant generalization and instanciation of the novel theoretical work of [1]. I am unsure about the novelty aspects, but the work seem original and significant.  Typos and other minor comments : - Reference [5] appears before Reference [4], which should be correct by simply re-compiling. - Line 32: therefor -> therefore - Figure 1 is a bit blurry (on paper), and the {+1, -1} cells are not distinguishable when printed on paper. - Figure 2 is also a bit blurry. - Footnote numbers in the text have an extra space (before the number). - The source code is conveniently based on Scikit-Learn, and should be made publicly available after publication. - Line 514 of Supplementary Material has an extra space before the reference to Section 3.2.  [1] Akshay Balsubramani and Yoav Freund. Optimally combining classifiers using unlabeled data. In Conference on Learning Theory, 2015. To appear. Update after rebuttal: I kept the same quality score.