Large-scale probabilistic predictors with and without guarantees of validity
This paper studies theoretically and empirically a method of turning machinelearning algorithms into probabilistic predictors that automatically enjoys a property of validity (perfect calibration) and is computationally efficient. The price to pay for perfect calibration is that these probabilistic predictors produce imprecise (in practice, almost precise for large data sets) probabilities. When these imprecise probabilities are merged into precise probabilities, the resulting predictors, while losing the theoretical property of perfect calibration, are consistently more accurate than the existing methods in empirical studies.
This paper proposes two techniques for post-processing a score-valued output of a binary classifier by converting it into an estimate of a class probability.The techniques are simple and efficient. It is surely publishable in principle, however, in my perspective the paper in its current form has far too many weak spots to be acceptable. They are listed in the following.Until the conclusion the distinction between contribution and background remains fuzzy.The content of section 2 seems trivial, in particular proposition 2. All it takes is sorting and binary search. However, this seems to be one of the "major" contributions.The hyperparameters of the learning machines were set by minimizing the training error?! E.g., a Gaussian kernel SVMs can easily achieve zero error. Then all we see is over-fitting. This is a clear no-go, it renders all experimental results meaningless. That alone is a reason for rejection.The improvement over the baseline methods is consistent, but the effect size is small. So are the new methods relevant, in particular since they take more space and time than Platt's method? I am missing a discussion of this point. For acceptance I want to be convinced that somebody out there cares.Minor comments:In contrast to what it stated in the paper, Platt's method is not invariant w.r.t. the (sigmoidal?) conversion of the scores to the unit interval. It does not become 100% clear whether the conversion is actually performed with Platt's method or not.At some point the paper mentions that most models output values in the unit interval, which could hence be interpreted as probabilities. This point is not followed upon in the experimental evaluation. Why not?In section 5.2 it is claimed that there is no simple ad-hoc regularization for isotonic regression. I don't see why Platt's regularization technique cannot be applied, which basically amounts to adding two virtual scores of plus and minus infinity.The first word of the title reads "Large-scale". Actually, nothing in this paper is large scale, and I don't see any relation to processing of large-scale data.The plots in figure 1 are far too small to be readable in a standard b/w printout.