Label shift refers to the phenomenon where the marginal probability p(y) of observing a particular class changes between the training and test distributions, while the conditional probability p(x|y) stays fixed. This is relevant in settings such as medical diagnosis, where a classifier trained to predict disease based on observed symptoms may need to be adapted to a different distribution where the baseline frequency of the disease is higher. Given estimates of p(y|x) from a predictive model, Saerens et al. (2002)  proposed an Expectation Maximization (EM) algorithm to efficiently correct for the difference in class proportions between training and test distributions. A limiting assumption of this EM algorithm is that the probabilities p(y|x) are calibrated, which is not true of modern neural networks. Recently, Black Box Shift Learning (BBSL) (Lipton et al., 2018)  and Regularized Learning under Label Shifts (RLLS) (Azizzadenesheli et al., 2019) have emerged as state-of-the-art techniques to cope with label shift when a classifier does not output calibrated probabilities. In this work, we show that while EM in the absence of calibration achieves weak performance with modern neural networks, the combination of EM with appropriate calibration typically outperforms BBSL and RLLS. In particular, we show that the popular calibration strategy of Temperature Scaling (TS) is often insufficient to completely correct miscalibration; rather, EM achieve state-of-the-art results primarily when used with a calibration approach that includes class-specific bias parameters that TS lacks. We further introduce a theoretically principled strategy for estimating source-domain priors in EM-based domain adaptation that allows the EM algorithm to behave more robustly when the calibrated probabilities contain systematic bias. Colab notebooks reproducing experiments on CIFAR10, CIFAR100, MNIST and diabetic retinopathy detection are available at (anonymized link): https://github.com/blindauth/labelshiftexperiments
This paper builds upon recent work on detecting and correcting for label shift.
They explore both the BBSE algorithm analyzed in Detecting and Correcting for Label Shift (2018)
and another approach based on EM where the predictive posteriors and test set label distributions
are iteratively computed, each an update based on the estimate of the other.
Crucially, while the former method requires only that the confusion matrix be invertible,
the  latter method only appears valid under strong assumptions including the calibration fo the classifier.
Thus the authors propose an approach for “bias-corrected calibration”
and shows that bias-corrected calibration can improve the performance of BBSE and EM.
The method is crucial for EM and with it, the results seem to show that EM,
in the large sample (8000 examples) regime and with good initial classifiers
 (on the relatively easy CIFAR10 task with a strong baseline)
that EM outperforms BBSE.
The paper is easy to follow an the authors should also be credited for releasing 
code anonymously with which we could reproduce their results. 
I have as few specific concerns/questions about the paper that I would like the authors to address: 
 * They consider JS divergence as a metric for evaluation. But they don’t consider other metrics
  like the error in weight estimates which is considered in most of the prior work 
 * They don’t compare their results with regularizations suggested on top of BBSE, particularly Azizzadenesheli et. al. https://arxiv.org/abs/1903.09734. 
 * They compare methods for particularly limited ranges of Dirichlet shift (\alpha=0.1,1.0). 
  * What happens when the \alpha increases to have less severe shifts? 
Optimizing ELBO with EM can lead to local convergence to the likelihood function when the likelihood is not unimodal. 
 *  Is this likelihood function unimodal?  Does the EM approach converges to MLE under some appropriate initialization and assumptions? 
  
A small presentation note: many of the papers are reporting the same metric and ought to be grouped as a large table, not as many tables. Also every table should state clearly what it is reporting in the caption, not just referring to earlier tables.
=======Update
I have read the rebuttal and appreciate that the authors took the time to establish the concavity of the likelihood function for EM. Overall this paper makes an interesting contribution in establishing the usefulness of the likelihood formulation (here optimized by EM) of label shift estimation and its apparent benefits over BBSE in some settings. I am happy to keep my score despite apparent disagreement from the other reviewers. I must say that some other reviews were disappointingly lacking in thoroughness.  
 
The paper still leaves open some serious questions, e.g. --- why is this bias correction heuristic so effective vis-a-vis EM and is this explained by its performance at the calibration task itself? The original temperature scaling paper reported a similar heuristic yet didn't see such a benefit wrt their metrics. Why is it so useful here? 
Still, while this paper can be improved in some key ways, it does make an interesting contribution.