Analysis of Learning from Positive and Unlabeled Data
Learning a classifier from positive and unlabeled data is an important class of classification problems that are conceivable in many practical applications. In this paper, we first show that this problem can be solved by cost-sensitive learning between positive and unlabeled data. We then show that convex surrogate loss functions such as the hinge loss may lead to a wrong classification boundary due to an intrinsic bias, but the problem can be avoided by using non-convex loss functions such as the ramp loss. We next analyze the excess risk when the class prior is estimated from data, and show that the classification accuracy is not sensitive to class prior estimation if the unlabeled data is dominated by the positive data (this is naturally satisfied in inlier-based outlier detection because inliers are dominant in the unlabeled dataset). Finally, we provide generalization error bounds and show that, for an equal number of labeled and unlabeled samples, the generalization error of learning only from positive and unlabeled samples is no worse than 2 √ 2 times the fully supervised case. These theoretical findings are also validated through experiments.
The paper presents an analysis of learning for a binary classification task when only positive and unlabeled data are available. As the authors point out this setting can be found in many applications, in particular in outlier detection. The authors begin by showing that PU classification can be modelled as a cost-sensitive classification problem. It is also argued that in such settings a non-convex loss is necessary, the argument here is based on the shortcomings of the hinge-loss specifically with respects to the ramp loss. In the modelling proposed by the authors the minimized risk depends on an estimate of the class prior. As in practice one rarely has access to the true prior, the authors introduce the notion of effective prior, which is shown to directly influence the risk. They further show that at least in the cases where the true class prior is large the effective prior is not influenced by small variations in the estimated prior. The authors also provide generalization error bounds in the case of kernel machines. They argue that the expected 0-1 loss in the case of PU classification is only 2*sqrt(2) times that of the fully supervised setting. Finally the paper presents empirical evidence in favour of the ramp loss over the hinge loss. The paper is overall well written and the subject is I think quite interesting. I am not personally aware of similar work, though I have not worked on PU classification. Multiple facets of the problem are theoretically analyzed, and I think in this respect the paper is not lacking. However there are a number of issues that prevent me from recommending acceptance. I would be interested on hearing from the authors on these points.- In section 3, I believe the authors overstate their findings. They state (lines 193), that to obtain the correct decision boundary, the loss function should be symmetric and thus the use of a non-convex loss is unavoidable. I don't see how the necessity for symmetry has been proven. In my view all that has been shown is that the ramp loss does not incorporate this "superfluous" term. Furthermore non-convexity does not necessarily follow from symmetry. As an aside, I think it is interesting to note the connection between the hinge loss terms in this settings and the Universum.     - In section 4, line 248 states that the risk when applying \hat{f} is on a line tangent to R* at \pi=\hat{\pi}and that the difference of \hat{\pi} from true \pi results in the excess risk. However in figure 3, the line is tangent at \pi=\tilde{\pi}. Also, the notation here is not helpful as \pi is used to denote the true prior and a variable.- In section 5, the authors state that the generalization error bound is at most 2*sqrt(2) times the error bound in the supervised case. I don't see why this holds, it seems to me that the authors are arguing \frac{2\sqrt{2}}{\sqrt{n + n'}} \geq  \frac{1}{\sqrt{n'}}  + \frac{1}{\sqrt{n}}, but this is not true. In any case, I don't statements of this sort don't make sense when working with big-O classes, as constants are suppressed. Also, in the abstract this relationship is mistakenly stated to relate to the errors, when in fact it relates to the bounds.- The experimental results are not entirely convincing. To be sure the ramp loss dominates the hinge loss as expected from the analysis. However results are shown only on a single dataset and on a small subset of the binary tasks that can be derived (only those involving zero). I understand that the main contribution is theoretical but it does take away a bit. What is truly missing however is how \hat{\pi} was estimated in these experiments. Was the true prior used?- A perhaps minor issue, but the notation in section 2 is not clean. P_1 is used for two different distribution P(X|Y=1) and P(f(X)=1|Y=1).