Answers.Body: <p>This sounds not a <a href="https://en.wikipedia.org/wiki/Oversampling_and_undersampling_in_data_analysis" rel="nofollow noreferrer">class imbalance</a> problem which is usually worried about during training stage with very few available data points, it's a <a href="https://towardsdatascience.com/understanding-dataset-shift-f2a5a262a766" rel="nofollow noreferrer">label (prior) shift</a> calibration issue with out-of-distribution (OOD) operational data. Firstly imbalanced training data as a <em>ratio</em> is not a problem in itself, as long as you have enough training data points of the imbalanced class with representative features and not to train a deep learning model with enormous number of parameters.</p>
  <blockquote>
  <p>Poor models... are often a result of fitting deterministic classifiers, using re-sampling or re-weighting methods to balance class frequencies in the training data and evaluating the model with a score such as accuracy... No re-sampling technique will magically generate more information out of the few cases with the rare class... It is well known that these problems typically have multiple incomparable Pareto optimal solutions... A more explicit way than oversampling or downsampling could be... assign explicit costs to misclassified samples and then minimize the total costs via cost-sensitive machine learning.</p>
  </blockquote>
  <p>Now based on your further description of your problem, you <em>already</em> have a good probabilistic binary classifier from balanced training data with a selection bias towards the imbalanced class. Assuming the <em>only</em> thing changed in the target distribution is the label and no new labels at test time as you specified, you can safely apply a MLE based approach called <a href="https://arxiv.org/abs/2003.07554" rel="nofollow noreferrer">MLLS</a>.</p>
  <blockquote>
  <p>Under label shift, the label distribution p(y) might change but the class-conditional distributions <span class="math-container">$p(x|y)$</span> do not... a maximum likelihood estimation approach, which we call MLLS, dominates empirically... This paper provides a unified framework relating techniques that use off-the-shelf predictors for label shift estimation. We argue that these methods all employ calibration, either explicitly or implicitly, differing only in the choice of calibration method and their optimization objective... we believe that this work, which addresses a foundational theoretical problem, does not present a significant societal concern.</p>
  </blockquote>
  <p>It's not a easy solution but a very principled rational approach, where the basic idea is to estimate the OOD label distribution <span class="math-container">$q(y)$</span> then adjust your trained probabilistic classifier <span class="math-container">$p(y|x)$</span> via <span class="math-container">$q(y)$</span> as follows:
  <span class="math-container">$$q(y|x)=\frac{q(x|y)q(y)}{q(x)} \approx \frac{p(x|y)q(y)}{q(x)}=\frac{p(y|x)q(y)}{p(y)} \frac{p(x)}{q(x)}$$</span></p>
  <p>The term <span class="math-container">$\frac{p(x)}{q(x)} \approx 1$</span> with above assumption, so we can plugin our estimate of <span class="math-container">$q(y)$</span> to get calibrated result. Of course you may also train your model from scratch using 'imbalanced' data following the same target distribution with enough data points (perhaps with the same number of positive samples) using my above referenced cost-sensitive ML with greater cost weight on recall for your specific purpose.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-30T05:02:05.777Z
Answers.Text: Thanks for this!! I'll look into MLLS. To your last point, I already tried using SMOTE to oversample the negative data to match the target distribution (~1:1000 pos:neg), though I still wasn't finding the resulting predictions particularly convincing, likely because the resampled negative data still wasn't diverse.
Answers.UserDisplayName: igoogledbing
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-30T05:57:34.267Z
Answers.Text: SMOTE only plays with sample distribution without firm decision-theoretic basis, that's why you'd better employ 'cost sensitive' ML to actually address your concerned precision/recall cost structure aspect in your objective.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-30T22:07:58.010Z
Answers.Text: For my data, it is basically molecules. Positive denotes a certain functional property. The labeled training data I have is a dataset with positive examples and negatives examples that have slight modifications from positives so as to remove functionality. My unlabeled test dataset is a very diverse molecule library. You mentioned MLLS for label shift. I don't think my data would quality as label shift because it is not anti-causal; I think the molecular features X do cause the functional property Y; thus, would label shift be appropriate here? Or would a covariate shift method be better?
Answers.UserDisplayName: igoogledbing
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-30T22:25:33.180Z
Answers.Text: Your described case is mainly label shift (ignoring the slight covariate shift from the selection bias that you didn't include enough negative high-dimensional sample in your balanced training). Usually in practice several shifts would occur, even concept, but for practical purpose the major apparent imbalance issue here is label shift. Note if you have significant label shift, of course the overall covariate would shift, but covariate shift only means shift within each class label! Anti-causal here is not mechanistic but purely Bayesian view. Hope this completely clarifies the main issue here
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-30T22:29:44.017Z
Answers.Text: Thanks! So in terms of practical implementation of MLLS as you have described, I basically multiply the predicted probability by q(y)/p(y)? So, if my labeled training data is 70.6% positive and my unlabeled test data is estimated to be 1% positive, I multiply the predicted probabilities by 1/70.6?
Answers.UserDisplayName: igoogledbing
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-30T22:39:01.703Z
Answers.Text: Thanks! One quick thing: I'm using softmax ouputs as probabilities - do you have recommendations for alternative ways to get probabilities or best practices for calibrating softmax outputs to treat as probabilities? Really appreciate your insight. Thanks
Answers.UserDisplayName: igoogledbing
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-30T22:43:34.473Z
Answers.Text: You only have two fixed classes here without open world it's probably not a big issue, but that's an entirely different question unrelated to this one, you can post another one here if you like. Please also upvote the answer you already accepted.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-30T04:09:24.700Z
Answers.LastActivityDate: 2024-07-30T04:09:24.700Z
Answers.UserDisplayName: cinch
Body: <p>I have a classifier model that has train/val/(initial)test data that is balanced between known (i.e., labeled) 500 negative and 1200 positive samples. The classifier model itself performs well on this labeled data. My concerns arise due to the negative data likely not comprehensively representing the theoretically negative high-dimensional data space. My positive data likely does comprehensively represent the diversity of the theoretically positive space.</p>
  <p>For my unlabeled test data of 4 million datapoints, where likely some ~1% are positive (this condition is rare), I am unsure how to deal with this, as my current classifier is basically classifying this unlabeled data as 50%/50% positive/negative, which seems inaccurate.</p>
  <p>I suppose my problem is basically that I have a Positive Unlabeled learning problem, but I want to make use of the known negatives that I do have. What are strategies for this? Thanks.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-29T21:04:35.943Z
Comments.Text: if it is trained (well) on a balanced dataset and the test data is not out-of-distribution, then what is the problem? you'll have to be careful with your evaluation, of course, but I think that performance should not be affected by imbalance in the test data.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-30T00:13:13.973Z
Comments.Text: This sounds not a class imbalance problem which is usually worried about during training stage with few available data points, it's a label shift calibration issue with OOD operational data since you already have a well trained probabilistic classifier. Is this what you really asking?
Comments.UserDisplayName: cinch
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-30T01:16:55.630Z
Comments.Text: @cinch Thanks. My training data has negative data (500) that occupies what I suspect to be a small part of some high-dimensional space that encapsulates negatives. The positive (1200) training data is fairly comprehensive of its space I believe. For my goal, I have 4,000,000 unlabeled diverse datapoints for which I want to get a probability of being positive. My current classifier is basically classifying these 50/50 pos/neg, which seems inaccurate. Do you have any recommendations for this problem, recognizing the negative training data was likely not comprehensive.
Comments.UserDisplayName: igoogledbing
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-30T02:13:53.873Z
Comments.Text: As an alternative approach, could some combination of classification + anomaly detection (or positive unlabeled) model work? Thanks again
Comments.UserDisplayName: igoogledbing
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-30T02:50:18.500Z
Comments.Text: i chose the title because the classifier model itself performs well on the labeled data. my concerns arise due to the negative data likely not comprehensively representing the theoretically negative space. Thus, for my unlabeled test data of 4 million datapoints, where likely some ~1% are positive (this condition is rare), I am unsure how to deal with this. My problem is basically that I have a Positive Unlabeled learning problem, but want to make use of the negatives that I do have. Thus, i first made a binary classifier. If you'd prefer I make a new thread better phrasing this problem, i can
Comments.UserDisplayName: igoogledbing
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-30T03:04:37.040Z
Comments.Text: Ok, I thought your "*My current classifier is basically classifying these 50/50 pos/neg, which seems inaccurate*" meant your trained ML model is not good for train/val data set. Now I understand they're good to represent your train/val dataset, just 'seems' inaccurate for test data. And you don't have the 4 million test data available yet or never will have, so you're just worried but actually didn't have a test result 50/50 on these 4 million test data which if true it certainly would be bad. Is it correct?
Comments.UserDisplayName: cinch
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-30T03:08:31.890Z
Comments.Text: @cinch yes that is correct. I've modified my initial question to better clarify
Comments.UserDisplayName: igoogledbing
Tags: <classification>
Title: Dealing with likely class imbalance for unlabeled test data
UserDisplayName: igoogledbing
ThreadText: # Dealing with likely class imbalance for unlabeled test data
  
  (Asked by: username_0 on 2024-07-29)
  
  I have a classifier model that has train/val/(initial)test data that is balanced between known (i.e., labeled) 500 negative and 1200 positive samples. The classifier model itself performs well on this labeled data. My concerns arise due to the negative data likely not comprehensively representing the theoretically negative high-dimensional data space. My positive data likely does comprehensively represent the diversity of the theoretically positive space.
  
  For my unlabeled test data of 4 million datapoints, where likely some ~1% are positive (this condition is rare), I am unsure how to deal with this, as my current classifier is basically classifying this unlabeled data as 50%/50% positive/negative, which seems inaccurate.
  
  I suppose my problem is basically that I have a Positive Unlabeled learning problem, but I want to make use of the known negatives that I do have. What are strategies for this? Thanks.
  
  **Tags:** classification
  
  > **Comments:**
  > **username_1**: if it is trained (well) on a balanced dataset and the test data is not out-of-distribution, then what is the problem? you'll have to be careful with your evaluation, of course, but I think that performance should not be affected by imbalance in the test data.  
  > **username_2**: This sounds not a class imbalance problem which is usually worried about during training stage with few available data points, it's a label shift calibration issue with OOD operational data since you already have a well trained probabilistic classifier. Is this what you really asking?  
  > **username_0**: @username_2 Thanks. My training data has negative data (500) that occupies what I suspect to be a small part of some high-dimensional space that encapsulates negatives. The positive (1200) training data is fairly comprehensive of its space I believe. For my goal, I have 4,000,000 unlabeled diverse datapoints for which I want to get a probability of being positive. My current classifier is basically classifying these 50/50 pos/neg, which seems inaccurate. Do you have any recommendations for this problem, recognizing the negative training data was likely not comprehensive.  
  > **username_0**: As an alternative approach, could some combination of classification + anomaly detection (or positive unlabeled) model work? Thanks again  
  > **username_0**: i chose the title because the classifier model itself performs well on the labeled data. my concerns arise due to the negative data likely not comprehensively representing the theoretically negative space. Thus, for my unlabeled test data of 4 million datapoints, where likely some ~1% are positive (this condition is rare), I am unsure how to deal with this. My problem is basically that I have a Positive Unlabeled learning problem, but want to make use of the negatives that I do have. Thus, i first made a binary classifier. If you'd prefer I make a new thread better phrasing this problem, i can  
  > **username_2**: Ok, I thought your "*My current classifier is basically classifying these 50/50 pos/neg, which seems inaccurate*" meant your trained ML model is not good for train/val data set. Now I understand they're good to represent your train/val dataset, just 'seems' inaccurate for test data. And you don't have the 4 million test data available yet or never will have, so you're just worried but actually didn't have a test result 50/50 on these 4 million test data which if true it certainly would be bad. Is it correct?  
  > **username_0**: @username_2 yes that is correct. I've modified my initial question to better clarify  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-07-30. Score: 0 | Accepted answer
  
  This sounds not a [class imbalance](https://en.wikipedia.org/wiki/Oversampling_and_undersampling_in_data_analysis) problem which is usually worried about during training stage with very few available data points, it's a [label (prior) shift](https://towardsdatascience.com/understanding-dataset-shift-f2a5a262a766) calibration issue with out-of-distribution (OOD) operational data. Firstly imbalanced training data as a *ratio* is not a problem in itself, as long as you have enough training data points of the imbalanced class with representative features and not to train a deep learning model with enormous number of parameters.
  
  > Poor models... are often a result of fitting deterministic classifiers, using re-sampling or re-weighting methods to balance class frequencies in the training data and evaluating the model with a score such as accuracy... No re-sampling technique will magically generate more information out of the few cases with the rare class... It is well known that these problems typically have multiple incomparable Pareto optimal solutions... A more explicit way than oversampling or downsampling could be... assign explicit costs to misclassified samples and then minimize the total costs via cost-sensitive machine learning.
  
  Now based on your further description of your problem, you *already* have a good probabilistic binary classifier from balanced training data with a selection bias towards the imbalanced class. Assuming the *only* thing changed in the target distribution is the label and no new labels at test time as you specified, you can safely apply a MLE based approach called [MLLS](https://arxiv.org/abs/2003.07554).
  
  > Under label shift, the label distribution p(y) might change but the class-conditional distributions $p(x|y)$ do not... a maximum likelihood estimation approach, which we call MLLS, dominates empirically... This paper provides a unified framework relating techniques that use off-the-shelf predictors for label shift estimation. We argue that these methods all employ calibration, either explicitly or implicitly, differing only in the choice of calibration method and their optimization objective... we believe that this work, which addresses a foundational theoretical problem, does not present a significant societal concern.
  
  It's not a easy solution but a very principled rational approach, where the basic idea is to estimate the OOD label distribution $q(y)$ then adjust your trained probabilistic classifier $p(y|x)$ via $q(y)$ as follows:
  $$q(y|x)=\frac{q(x|y)q(y)}{q(x)} \approx \frac{p(x|y)q(y)}{q(x)}=\frac{p(y|x)q(y)}{p(y)} \frac{p(x)}{q(x)}$$
  
  The term $\frac{p(x)}{q(x)} \approx 1$ with above assumption, so we can plugin our estimate of $q(y)$ to get calibrated result. Of course you may also train your model from scratch using 'imbalanced' data following the same target distribution with enough data points (perhaps with the same number of positive samples) using my above referenced cost-sensitive ML with greater cost weight on recall for your specific purpose.
  
  > **Comments:**
  > **username_0**: Thanks for this!! I'll look into MLLS. To your last point, I already tried using SMOTE to oversample the negative data to match the target distribution (~1:1000 pos:neg), though I still wasn't finding the resulting predictions particularly convincing, likely because the resampled negative data still wasn't diverse.  
  > **username_2**: SMOTE only plays with sample distribution without firm decision-theoretic basis, that's why you'd better employ 'cost sensitive' ML to actually address your concerned precision/recall cost structure aspect in your objective.  
  > **username_0**: For my data, it is basically molecules. Positive denotes a certain functional property. The labeled training data I have is a dataset with positive examples and negatives examples that have slight modifications from positives so as to remove functionality. My unlabeled test dataset is a very diverse molecule library. You mentioned MLLS for label shift. I don't think my data would quality as label shift because it is not anti-causal; I think the molecular features X do cause the functional property Y; thus, would label shift be appropriate here? Or would a covariate shift method be better?  
  > **username_2**: Your described case is mainly label shift (ignoring the slight covariate shift from the selection bias that you didn't include enough negative high-dimensional sample in your balanced training). Usually in practice several shifts would occur, even concept, but for practical purpose the major apparent imbalance issue here is label shift. Note if you have significant label shift, of course the overall covariate would shift, but covariate shift only means shift within each class label! Anti-causal here is not mechanistic but purely Bayesian view. Hope this completely clarifies the main issue here  
  > **username_0**: Thanks! So in terms of practical implementation of MLLS as you have described, I basically multiply the predicted probability by q(y)/p(y)? So, if my labeled training data is 70.6% positive and my unlabeled test data is estimated to be 1% positive, I multiply the predicted probabilities by 1/70.6?  
  > **username_0**: Thanks! One quick thing: I'm using softmax ouputs as probabilities - do you have recommendations for alternative ways to get probabilities or best practices for calibrating softmax outputs to treat as probabilities? Really appreciate your insight. Thanks  
  > **username_2**: You only have two fixed classes here without open world it's probably not a big issue, but that's an entirely different question unrelated to this one, you can post another one here if you like. Please also upvote the answer you already accepted.  
  
  ---
  
  