Robust Classification Under Sample Selection Bias
In many important machine learning applications, the source distribution used to estimate a probabilistic classifier differs from the target distribution on which the classifier will be used to make predictions. Due to its asymptotic properties, sample reweighted empirical loss minimization is a commonly employed technique to deal with this difference. However, given finite amounts of labeled source data, this technique suffers from significant estimation errors in settings with large sample selection bias. We develop a framework for learning a robust bias-aware (RBA) probabilistic classifier that adapts to different sample selection biases using a minimax estimation formulation. Our approach requires only accurate estimates of statistics under the source distribution and is otherwise as robust as possible to unknown properties of the conditional label distribution, except when explicit generalization assumptions are incorporated. We demonstrate the behavior and effectiveness of our approach on binary classification tasks.
Paper ID: 39Title: Robust Classification Under Sample Selection BiasNOTE: Due to the short reviewing time and out of fairness to the other papers I was reviewing, I DID NOT do more than glance over the supplementary material.  In most cases, I therefore did not verify that results claimed by the authors are correct, but instead checked that they are plausible.Summary: This paper is about how to adjust a classifier when the training set is not representative of the test set; a canonical example is active learning, but the problem can also appear in recalibration.  The goal is to find an effective method in the finite-sample regime, since most results are asymptotics.  The author(s) take a minimax approach.Quality, clarity, originality and significance:  This work takes a novel approach to address an important problem in the practical application of classifiers.  The manuscript is generally well-written and clear, but I have some comments about works that seem related to this problem.  Some imprecision in the mathematical description was confusing.Recommendation:  Overall, I believe this is a nice contribution which will be appreciated by the NIPS community, assuming that the manuscript can be clarified appropriately.Comments:  (comments are numbered to allow for easy reference in the rebuttal)1) There are some works on recalibrating classifiers to adjust for different populations (a Google search turned up some papers from IJCAI, CVPR, KDD, and other venues).  Is there a connection between this problem and that literature that may be relevant?2)  The formulation in (1) assumes the alphabets for X and Y are both discrete, but this is not what happens in the experiments.  It may be better to express (1) as an expectation rather than writing it out explicitly.  Otherwise the generality is lost, since one has to replace sums by integrals.3) After Definition 1 it may be appropriate to comment on why matching the constraints *exactly* is even relevant in this setting (this is relaxed later but at the moment Definition 1 seems already like a bad idea).4)  In 2.2, I was not sure what the authors meant by "assuming that source distribution statistics generalize more broadly than the source distribution itself may be justifiable and useful."  What does expert knowledge have to do with this?  I was rather confused by this motivation.5)  By introducing some slack in the constraints the estimator seems somewhat "similar" to things one sees in robust statistics.  Is there a connection here?6)  Somehow the form of the estimator in Theorem 2 comes out of the blue.  Is there a way to explain some of the intuition behind it before introducing it?7)  What does \approx mean in (8)?  That seems very imprecise.8)  In 2.5, this goes back to 2) -- densities have thus far not appeared in the story and suddenly it seems all variables are continuous...9)  In Algorithm 1, how does gamma relate to epsilon in the error bounds in 2.4?10)  In 3.1, are the other methods from other papers on this problem?  If so, please provide citations.11)  In general, in the experiments section, using some notation that had been previously established to connect the plots back to the preceding theory would be extremely helpful.12) Just as a thought, how would increasing the training set size help in Figures 2-3?Small things:- Perhaps before Definition 2 the c' variables could be actually defined, e.g. using \stackrel{\Delta}{=} or some other macro.- check the referenes -- there are some typos (c.f. Bickel's first name inPlease summarize your review in 1-2 sentences:  The authors propose a new method for handling mismatch between the training and test distributions when training a classifier.  Experiments illustrate the benefit of the method over competing approachesQuality Score (1-10):  7Impact Score (1-2):  2Confidence (1-5):  3COMMENTS AFTER AUTHOR FEEDBACK:  The authors have proposed a rather extensive set of revisions to address the feedback from the reviewers. With those changes I think the paper may indeed be in the top 50% of NIPS papers.  I am therefore revising my score upward to an 8.Quality Score (1-10):  8