Theoretical Comparisons of Positive-Unlabeled Learning against Positive-Negative Learning
In PU learning, a binary classifier is trained from positive (P) and unlabeled (U) data without negative (N) data. Although N data is missing, it sometimes outperforms PN learning (i.e., ordinary supervised learning). Hitherto, neither theoretical nor experimental analysis has been given to explain this phenomenon. In this paper, we theoretically compare PU (and NU) learning against PN learning based on the upper bounds on estimation errors. We find simple conditions when PU and NU learning are likely to outperform PN learning, and we prove that, in terms of the upper bounds, either PU or NU learning (depending on the class-prior probability and the sizes of P and N data) given infinite U data will improve on PN learning. Our theoretical findings well agree with the experimental results on artificial and benchmark data even when the experimental setup does not match the theoretical assumptions exactly.
The paper presents a theoretical analysis of when learning from positive and unlabelled data may be superior to learning from fully labelled data. The analysis relies on upper bounds of the estimation errors for various risks. Experiments largely agree with the theoretical analysis.       
   The basic problem studied in the paper concerns learning from data which is only partially labelled, but nonetheless doing better than with fully labelled data. In the particular scenario where one has a pool of unlabelled data in lieu of one of the classes, the paper seeks to quantify the impact this has on the estimation error of the learned classifier. Determining the degradation or lack thereof when learning from unlabelled data is interesting, and thus the paper seems well motivated. The machinery used to illustrate its messages are fairly standard -- the key quantities, namely the estimation errors for each scenario, are derived from a simple Rademacher analysis -- however, the final results appear novel, with implications worked through in various scenarios.    The papers hinges on the simple facts that (a) different risks (for PN/PU/NU learning) may be seen as employing different weightings on individual risks for the positive and negative class, and (b) these ratings are reflected in appropriate terms for the estimation error. This, combined with the fact that in practice one is on the regime where unlabelled data is plentiful, is used to argue for the potential benefit is of learning from positive and unlabelled data. Quantifying the precise benefit that having a large amount of data from one "meta-class" buys you, and how it offsets the fact that this "meta-class" doesn't correspond to the actual underlying label, is nice.   The estimation error analysis is fairly standard, and simply exploits the different weightings on the per-class risks. As a lacuna, the comparison of the various scenarios ends up being based on the upper bounds provided by the Rademacher analysis; this isn't ideal, as it implicitly assumes tightness of the bounds. This aside, the comparison of the of various settings in Thm 6 seems fairly unsurprising: one finds that depending on the relation of the number of observed samples to the base rate, either the positive and unlabelled or the fully labelled scenarios will be preferable. I admittedly did not get a lot out of the specific comparisons provided under various constraints on the number of samples; perhaps it would be better to introduce some practical scenario, and then try to motivate some particular relationship between the number of samples. Additionally, it seems natural to explicitly consider the case where n_- = n_u, where the bound for PN will dominate, to make clear that the benefit of PU data is purely from the ability to collect large numbers of samples from the unlabelled set.   The asymptotic analysis in 3.3 aims to illustrate that with large amounts of unlabelled data, positive and on label learning can dominate the fully laden counterparts. The observation that one of PU or NU will dominate PN is intuitive in hindsight, but perhaps not obvious a-priori. I felt here that one could perhaps start off by considering the case where n_u = infty, before introducing the limit-based analysis. This would perhaps more directly hint at the power of the positive and unlabelled scenario.   One limitation of the present analysis is that it restricts attention to losses satisfying the symmetry condition (2). It wasn't too clear why this was strictly necessary. What about losses satisfying ell(-1, v) - ell(1, v) = v, e.g. those considered in [5]? As I understand, the latter showed a risk equivalence for such losses as well. Is the point that in this case one doesn't have the exact same loss on the PU data, which perhaps complicates analysis somewhat?     Other comments:   - a citation for the empirical observation that learning from positive and unlabelled data performs better than positive and negative data seems prudent.    - for the two sample positive and unlabelled setting, consider also citing  Ward et al. Presence-only data and the EM algorithm. Biometrics, 2009.    - the statement that the two sample scenario is more general than the one sample scenario is a bit unclear. The distributions that one observes in each scenario seem quite different. How exactly does one cast the noisy label scenario in terms of drawing from the positive conditional and the marginal?    - there are a few improper uses of \mapsto instead of \to.    - it wasn't entirely clear why one has the dependence on \pi in \hat{R}_pn. In practice surely one just uses 1/(n+ + n-) as a common scaling factor for the positive and negative risks, with no need to actually know the true value of the base rate. It seems that the paper employs this formulation in order to get an easy comparison to the bounds for the positive and unlabelled case? I note that otherwise Thm 3 for example would be nothing more than a standard Rademacher bound for the empirical risk, e.g. (Boucheron et al.), Thm 4.1.    - it is said that [1] establishes that the 01 loss can be directly estimated in the positive and unlabelled setting; but that paper was only for the OS setting? And what is the specific theorem reference?    - it could be made clearer in the theorem statements that one crucially relies on the loss satisfying the condition (2).    - in the proof of corollary 5, it is unclear why the discussion proceeds onto the case where the function class is of the form (6). Isn't the proof complete at this stage? Is the idea is simply to show a direct proof for a special case?    - Thm 6 could perhaps be demoted to Lemma 6?    - the proof of lemma 8 could surely borrow from existing Rademacher analysis e.g. (Boucheron et al.), rather than invoking McDiamird and Talagrand from scratch.