abstract
faced with distribution shift between training and test set, we wish to detect and quantify
the shift, and to correct our classifiers without test set labels. motivated by medical diagnosis,
where diseases (targets), cause symptoms (observations), we focus on label shift, where the
label marginal p(y) changes but the conditional p(x|y) does not. we propose black box shift
estimation (bbse) to estimate the test distribution p(y). bbse exploits arbitrary black
box predictors to reduce dimensionality prior to shift correction. while better predictors give
tighter estimates, bbse works even when predictors are biased, inaccurate, or uncalibrated, so
long as their confusion matrices are invertible. we prove bbse’s consistency, bound its error,
and introduce a statistical test that uses bbse to detect shift. we also leverage bbse to
correct classifiers. experiments demonstrate accurate estimates and improved prediction, even
on high-dimensional datasets of natural images.