ON THE MINIMAL SUPERVISION FOR TRAINING ANY BINARY CLASSIFIER FROM ONLY UNLABELED DATA
Empirical risk minimization (ERM), with proper loss function and regularization, is the common practice of supervised classification. In this paper, we study training arbitrary (from linear to deep) binary classifier from only unlabeled (U) data by ERM. We prove that it is impossible to estimate the risk of an arbitrary binary classifier in an unbiased manner given a single set of U data, but it becomes possible given two sets of U data with different class priors. These two facts answer a fundamental question-what the minimal supervision is for training any binary classifier from only U data. Following these findings, we propose an ERM-based learning method from two sets of U data, and then prove it is consistent. Experiments demonstrate the proposed method could train deep models and outperform state-of-the-art methods for learning from two sets of U data.
This paper studies the weak supervision setting of learning a general binary classifier from two unlabeled (U) datasets with known class balances. The authors establish that this is possible by constructing an unbiased estimator, analyze its convergence theoretically, and then run experiments using modern image classification models.
Pros:
- This work demonstrates, theoretically and empirically, a simple way to train generic models using only the known class balances of several sets of unlabeled data (having the same conditional distributions p(x|y))---a very interesting configuration of weak supervision, an increasingly popular and important area
- The treatment is thorough, proceeding from establishing the minimum number of U datasets, constructing the estimator, analyzing convergence, and implementing thorough experiments
Cons:
- This is a crowded area (as covered in their related work section). As they cite, (Quadrianto et al., 2009) proposed this setting and considered linear models for k-wise classification.  Moreover, the two U datasets with known class balances can equivalently be viewed as two weak / noisy label sources with known accuracies.  Thus this work connects to many areas- both in noisy learning, as they cite heavily, but also in methods (in e.g. crowdsourcing and multi-source weak supervision) where several sources label unlabeled datasets with unknown accuracies (which are often estimated in an unsupervised fashion).
- The overall clarity of the paper's writing could be improved. For example, the introduction and related work sections take up a large portion of the paper, but are very dense and heavy with jargon that is not internally defined upfront; for example "risk rewrite" is introduced in paragraph 2 with no internal definition and then used subsequently throughout the paper (this defn would be simple enough to give: in the context of this paper, "risk rewrite" means a linear combination of the class-conditional losses; or more generally, the expected loss w.r.t. distribution over classes...).  Also intuition could be briefly given about the theorem proof strategies.
- The difference between the two class distributions over the U datasets seems like an important quantity (akin, in e.g. weak supervision / crowd source modeling papers, to quantity of how bounded away from random noise the labelers are). This is treated empirically, but would be stronger to have this show up in the theory somewhere.
- Other prior work here has handled k classes with k U sets; could have extended to cover this setting too, since seems natural
Overall take: This learning from label proportions setting has been covered before, but this paper presents it in an overall clean and general way, testing it empirically on modern models and datasets, which is an interesting contribution.
Other minor points:
- The argument for / distinction between using eqns. (3) and (4) seems a bit ad hoc / informal ("we argue that...").  This is an important point...
- Theorem 1 proof seems fine, but some intuition in the main body would be nice.
- What does "classification calibrated" mean?
- Saying that three U sets are needed, where this includes the test set, seems a bit non-standard?  Also I'm confused- isn't a labeled test set used?  So what is this third U set for?
- The labels l_+ and l_- in Defn. 3 seem to imply that the two U sets are positive vs. negative; but this is not the case, correct…?
- Stating both Lemma 5 and Thm 6 seems unnecessary
- In Fig. 2, seems like could have trained for longer and perhaps some of the losses would have continued decreasing?  In particular, small PN?  Also, a table of the final test set accuracies would have been very helpful.
- More detail on experimental protocol would be helpful: what kind of hyperparameter tuning was done? repeated runs averaging?  It seems odd, for example in Fig. 3, that the green lines are so different in (a) vs. (c), and not in the way that one would expect given the decrease in theta