Optimal Binary Classifier Aggregation for General Losses
We address the problem of aggregating an ensemble of predictors with known loss bounds in a semi-supervised binary classification setting, to minimize prediction loss incurred on the unlabeled data. We find the minimax optimal predictions for a very general class of loss functions including all convex and many non-convex losses, extending a recent analysis of the problem for misclassification error. The result is a family of semi-supervised ensemble aggregation algorithms which are as efficient as linear learning by convex optimization, but are minimax optimal without any relaxations. Their decision rules take a form familiar in decision theory -applying sigmoid functions to a notion of ensemble margin -without the assumptions typically made in margin-based learning.
In this paper a worst-case optimal aggregation algorithm for binary classifiers under general losses is proposed. This work builds upon a line of work on semi-supervised and transductive learning for classification problems, most notably ref. [1]. The main contribution of the paper is a minimax formulation of the problem exploiting classifier correlation constraints. It is proved that this formulation leads to a convex program, giving a closed-form minimax optimal predictor under rather general conditions on the loss function (and even the type of side constraints used, but this is left in the Appendix). This result is based on a simple application of Sion minimax Theorem, together with a characterization of convexity of the minimax objective, the so-called the Potential Well.  As said before, this paper builds upon existing work, but it extends the applicability of these ideas very broadly. Technically, the paper does not introduce heavy machinery, but the obtained insights lead to new understanding of the problem which seems to be absent in the literature so far. On top of this, it is a very well written paper. For these reasons, I recommend this paper for publication at NIPS.
I believe this paper has great potential impact. First, it tackles a central question in learning theory (semi-supervised classification), it generalizes the literature considerably, and the obtained tools are scalable and practical. The technical content of the paper is good, although not extremely novel; despite this, it is able to provide insights that were not previously obtained in the literature.  A natural high-level question is whether the minimax approach is too conservative in a more benign setting. A possible way to formalize this question is whether the minimax results obtained can be translated to generalization bounds in the distributional setting, possibly by some additional tools such as regularization.  Next I would like to make some minor comments:  1. I couldn't find the justification of the "perfectly tight" minimax bound (Line 97). However, this is evident from Sion minimax Theorem (this is a zero sum game, which is quasiconcave on one variable, and convex --in fact, linear-- in the other), and I suggest to add this as a reference.  2. The discussion on lines 229-233 is confusing. I quickly checked [1], and I couldn't understand where the data distribution is used. Please clarify.  3. Finally, the equation following line 288 needs some further justification.