On the Minimal Supervision for Training Any Binary Classifier from Only Unlabeled Data
Empirical risk minimization (ERM), with proper loss function and regularization, is the common practice of supervised classification. In this paper, we study training arbitrary (from linear to deep) binary classifier from only unlabeled (U) data by ERM. We prove that it is impossible to estimate the risk of an arbitrary binary classifier in an unbiased manner given a single set of U data, but it becomes possible given two sets of U data with different class priors. These two facts answer a fundamental question---what the minimal supervision is for training any binary classifier from only U data. Following these findings, we propose an ERM-based learning method from two sets of U data, and then prove it is consistent. Experiments demonstrate the proposed method could train deep models and outperform state-of-the-art methods for learning from two sets of U data.
 The authors propose an unbiased estimator that allows for training models with weak supervision on two unlabeled datasets with known class priors. The theoretical properties of the estimator are discussed and an empirical evaluation shows promising performance. The paper provides a thorough overview of the related work. The experiments compare to the relevant baselines. Minor remarks: The writing seems like it could be improved in multiple places and the main thing that makes some sections of the paper hard to follow is that the concepts often get mentioned and discussed before they are formally defined/introduced. Concepts that are introduced via citations should also be explained even if not in-depth. Figure 2: the curves suggest that the models should have been left to train for a longer time - some of the small PN and small PN prior-shift risks are still decreasing Figure 2: the scaling seems inconsistent - the leftmost subplot in each row doesn’t start at (0,0) in the lower left corner, unlike the other subplots in each row - and it should probably be the same throughout - no need to be showing the negative space. Figure 2: maybe it would be good to plot the different lines in different styles (not just colors) - for BW print and colorblind readers For small PN and small PN prior-shift, the choice of 10% seems arbitrary. At what percentage do the supervised methods start displaying a clear advantage - for the experiments in the paper? When looking into the robustness wrt noise in the training class priors, both are multiplied by the same epsilon coefficient. In a more realistic setting the priors might be perturbed independently, potentially even in a different direction. It would be nice to have a more general experiment here, measuring the robustness of the proposed approach in such a way. 5.2 typo: benchmarksand ; datasetsfor
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
On the Minimal Supervision for Training Any Binary Classifier from Only Unlabeled Data
Empirical risk minimization (ERM), with proper loss function and regularization, is the common practice of supervised classification. In this paper, we study training arbitrary (from linear to deep) binary classifier from only unlabeled (U) data by ERM. We prove that it is impossible to estimate the risk of an arbitrary binary classifier in an unbiased manner given a single set of U data, but it becomes possible given two sets of U data with different class priors. These two facts answer a fundamental question---what the minimal supervision is for training any binary classifier from only U data. Following these findings, we propose an ERM-based learning method from two sets of U data, and then prove it is consistent. Experiments demonstrate the proposed method could train deep models and outperform state-of-the-art methods for learning from two sets of U data.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The authors propose an unbiased estimator that allows for training models with weak supervision on two unlabeled datasets with known class priors. The theoretical properties of the estimator are discussed and an empirical evaluation shows promising performance. The paper provides a thorough overview of the related work. The experiments compare to the relevant baselines. Minor remarks: The writing seems like it could be improved in multiple places and the main thing that makes some sections of the paper hard to follow is that the concepts often get mentioned and discussed before they are formally defined/introduced. Concepts that are introduced via citations should also be explained even if not in-depth. Figure 2: the curves suggest that the models should have been left to train for a longer time - some of the small PN and small PN prior-shift risks are still decreasing Figure 2: the scaling seems inconsistent - the leftmost subplot in each row doesn’t start at (0,0) in the lower left corner, unlike the other subplots in each row - and it should probably be the same throughout - no need to be showing the negative space. Figure 2: maybe it would be good to plot the different lines in different styles (not just colors) - for BW print and colorblind readers For small PN and small PN prior-shift, the choice of 10% seems arbitrary. At what percentage do the supervised methods start displaying a clear advantage - for the experiments in the paper? When looking into the robustness wrt noise in the training class priors, both are multiplied by the same epsilon coefficient. In a more realistic setting the priors might be perturbed independently, potentially even in a different direction. It would be nice to have a more general experiment here, measuring the robustness of the proposed approach in such a way. 5.2 typo: benchmarksand ; datasetsfor
assistant
 The authors propose an unbiased estimator that allows for training models with weak supervision on two unlabeled datasets with known class priors. The theoretical properties of the estimator are discussed and an empirical evaluation shows promising performance. The paper provides a thorough overview of the related work. The experiments compare to the relevant baselines. Minor remarks: The writing seems like it could be improved in multiple places and the main thing that makes some sections of the paper hard to follow is that the concepts often get mentioned and discussed before they are formally defined/introduced. Concepts that are introduced via citations should also be explained even if not in-depth. Figure 2: the curves suggest that the models should have been left to train for a longer time - some of the small PN and small PN prior-shift risks are still decreasing Figure 2: the scaling seems inconsistent - the leftmost subplot in each row doesn’t start at (0,0) in the lower left corner, unlike the other subplots in each row - and it should probably be the same throughout - no need to be showing the negative space. Figure 2: maybe it would be good to plot the different lines in different styles (not just colors) - for BW print and colorblind readers For small PN and small PN prior-shift, the choice of 10% seems arbitrary. At what percentage do the supervised methods start displaying a clear advantage - for the experiments in the paper? When looking into the robustness wrt noise in the training class priors, both are multiplied by the same epsilon coefficient. In a more realistic setting the priors might be perturbed independently, potentially even in a different direction. It would be nice to have a more general experiment here, measuring the robustness of the proposed approach in such a way. 5.2 typo: benchmarksand ; datasetsfor