Binary Classification from Positive-Confidence Data
Can we learn a binary classifier from only positive data, without any negative data or unlabeled data? We show that if one can equip positive data with confidence (positive-confidence), one can successfully learn a binary classifier, which we name positive-confidence (Pconf) classification. Our work is related to one-class classification which is aimed at "describing" the positive class by clustering-related methods, but one-class classification does not have the ability to tune hyper-parameters and their aim is not on "discriminating" positive and negative classes. For the Pconf classification problem, we provide a simple empirical risk minimization framework that is model-independent and optimization-independent. We theoretically establish the consistency and an estimation error bound, and demonstrate the usefulness of the proposed method for training deep neural networks through experiments.
Overview and Recommendation: This paper presents an algorithm and theoretical analysis for Pconf learning of a binary classifier, when only the positive instances and their conditional class probabilities are available. In contrast to PU learning, which has received attention recently in the ML community, Pconf learning does not require a large amount of unlabeled data or any knowledge of class priors, however it does require confidence weights p(y=+1 | x) for each example x sampled from p(x | y= +1).  The paper is well written and this solution can be useful for many real world applications, therefore, it is a clear accept. Although, the analysis is very similar to the recent analysis for PU learning published by Du Plessis et al. recently at NIPS but the problem setting is novel and interesting.  However, I think that the authors should fix some technical issues with their simulation experiment.  Summary: The main contribution of the paper is the observation that empirical risk minimization for Pconf learning can be done by importance weighting the confidence weighted risk (equation 3) using positive samples. The paper also provides an estimatior error bound for the proposed risk-estimator. The analysis only requires a bounded hypothesis class and a bounded lipschitz loss function on that hypothesis set and shows that estimation error decreases as 1/sqrt(n), which is expected and intuitive. The experimental results also support the theory.  Technical Issues: Loss of accuracy when the separation between clusters increases should simply not happen. The authors justify this by a loss-of-senstivity in the confidence values but in the end this is just a numerical issue. The gradient of the log-linear model used in the experiment can be coded directly, and then an arbitrary-precision floating-point library for python such as mpmath can be used. Since the dimensionality is just 2 and the number of examples is 1000 this experiment should not take up much resources.  ==== After Author Response ===  I am happy to see that the authors made the effort to run the experiment using the arb prec math library which made the experimental results more intuitive. I have increased my rating of the paper, because now the paper does not seem to have any technical issues.