Answers.Body: <h2>Sample Reweighting</h2>
  <p>There's actually been a good amount of work on <em>sample reweighting</em>, namely learning a set of weights for each datapoint in your training data. However, the goal of these algorithms seems more to be to correct for dataset <em>bias</em> rather than outliers. For example, if your training data is biased towards a certain class, then you'd want to underweight samples in that class.</p>
  <p><strong>Past work avoids the problems you mentioned by optimizing the weights with respect to a held-out validation set.</strong> Importantly, the weights learned are for the <em>training set</em>, that way, you can't just set all the weights to zero because then your model wouldn't learn anything and your validation set performance would drop.</p>
  <h3>Meta-learning</h3>
  <p><a href="https://arxiv.org/abs/1803.09050" rel="nofollow noreferrer"><em>&quot;Learning to Reweight Examples for Robust Deep Learning&quot;</em></a> is one well-cited paper that does this.</p>
  <p>The intuition behind their algorithm is:</p>
  <blockquote>
  <p>the best example weighting should minimize the loss of a set of unbiased clean validation examples that are consistent with the evaluation procedure</p>
  </blockquote>
  <p>They do this through approximate meta-learning. The weights for each sample in the training set are learnable parameters. Vanilla meta-learning would involve two optimization loops: for each update you make to the weights, you'd need to do a number of iterations for the actual training set. They approximate this by just looking at gradients of the sample weights wrt the validation loss in a <em>single</em> iteration. A large negative gradient would mean that increasing the weight of that sample would lead to a large decrease in the validation loss, so you should weigh that sample more heavily. Note that they <em>do</em> normalize the sample weights here such that weights over each batch sum to one.</p>
  <p>There's a ton of literature on sample weighting, including techniques that don't use meta-learning (e.g., <a href="https://arxiv.org/abs/2107.09044" rel="nofollow noreferrer"><em>Just Train Twice</em>, which weighs points that models misclassify more heavily</a>), or involve <a href="https://arxiv.org/abs/1902.07379" rel="nofollow noreferrer">learning a model to predict weights</a>. I'd recommend you follow the citation trail, or look sample reweighting for more work on this topic.</p>
  <h2>Outlier detection and removal</h2>
  <p>You may also want to consider using a different technique, rather than sample reweighting, especially because the goal of the technique tends to be improving robustness/generalization and reducing the impact of training set bias. In fact, these techniques often weigh difficult samples more <em>highly</em>.</p>
  <p><a href="https://arxiv.org/abs/2210.15183" rel="nofollow noreferrer">This paper</a> looks at this issue with <em>Just Train Twice</em>, where unlearnable or OOD samples end up getting weighed heavily. They tackle this by detecting and removing outliers based on the <a href="https://en.wikipedia.org/wiki/Mahalanobis_distance" rel="nofollow noreferrer">Mahalanobis distance</a> on the hidden representations of the network (based on <a href="https://papers.nips.cc/paper_files/paper/2018/hash/abdeb6f575ac5c6676b747bca8d09cc2-Abstract.html" rel="nofollow noreferrer">this paper</a>). <a href="https://arxiv.org/abs/1610.02136" rel="nofollow noreferrer">Other work involves looking at the softmax probabilities of the predicted class or training a classifier to predict OOD samples.</a></p>
  <p>There's also a ton of work on this, and the papers I mentioned are by no means comprehensive. I'd recommend taking a look at work on <a href="https://paperswithcode.com/task/out-of-distribution-detection" rel="nofollow noreferrer">OOD sample detection</a>, outlier detection, or handling mislabeled data.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-05T21:14:14.203Z
Answers.Text: Thanks for this excellent reply @Alexander. Just wanted to provide one quick clarification (specifically regarding your last section). None of the samples are outliers as such, its more that they all sit on a spectrum of predictability (which is also a function of some variables). E.g. consider image classification, but with a twist where the object to classify can be very close (occupying a large number of pixels),  very far away (small number of pixels) or anywhere in between. Making some images easier/harder to classify then others. (Just an analogy, not actually what I'm trying to do)
Answers.UserDisplayName: Hiho
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-05T07:31:26.117Z
Answers.LastActivityDate: 2023-10-05T07:31:26.117Z
Answers.UserDisplayName: Alexander Wan
Body: <p>I'm trying to train a ML model, however the predictability of the different samples varies, i.e. some samples are inherently much harder to predict/estimate than others. Poorer predictions for these samples is fine and expected. However, after a few iterations these samples dominate the loss, resulting in no further improvement in the prediction of the other samples.</p>
  <p>My plan is to use sample weights to address this, however as I do not have these sample weights and they are a function of another set of features, I was thinking of training another ML model to estimate the sample weights. My thinking is to train these two models together, i.e. using a loss function of the form <span class="math-container">$L = w * MAE(y, \hat{y})$</span>.
  However, this will just result in the weight model predicting <span class="math-container">$w=0$</span> as it minimizes the loss function.</p>
  <p>What are some approaches to prevent the weights from just going to zero during training?
  I'm guessing there's is quite a few, I just can't seem to find them (unsure what to search for).</p>
  <p>The only one that has jumped to mind is to normalize the weights such that the sum to some constant, but I don't think this would work with batch-based gradient descent?</p>
  <p>Thanks!</p>
  
Tags: <machine-learning><training><optimization>
Title: How to train a sample weight model for another ML model?
UserDisplayName: Hiho
ThreadText: # How to train a sample weight model for another ML model?
  
  (Asked by: username_0 on 2023-10-04)
  
  I'm trying to train a ML model, however the predictability of the different samples varies, i.e. some samples are inherently much harder to predict/estimate than others. Poorer predictions for these samples is fine and expected. However, after a few iterations these samples dominate the loss, resulting in no further improvement in the prediction of the other samples.
  
  My plan is to use sample weights to address this, however as I do not have these sample weights and they are a function of another set of features, I was thinking of training another ML model to estimate the sample weights. My thinking is to train these two models together, i.e. using a loss function of the form $L = w \* MAE(y, \hat{y})$.
  However, this will just result in the weight model predicting $w=0$ as it minimizes the loss function.
  
  What are some approaches to prevent the weights from just going to zero during training?
  I'm guessing there's is quite a few, I just can't seem to find them (unsure what to search for).
  
  The only one that has jumped to mind is to normalize the weights such that the sum to some constant, but I don't think this would work with batch-based gradient descent?
  
  Thanks!
  
  **Tags:** machine-learning, training, optimization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-10-05. Score: 3 | Accepted answer
  
  ## Sample Reweighting
  
  There's actually been a good amount of work on *sample reweighting*, namely learning a set of weights for each datapoint in your training data. However, the goal of these algorithms seems more to be to correct for dataset *bias* rather than outliers. For example, if your training data is biased towards a certain class, then you'd want to underweight samples in that class.
  
  **Past work avoids the problems you mentioned by optimizing the weights with respect to a held-out validation set.** Importantly, the weights learned are for the *training set*, that way, you can't just set all the weights to zero because then your model wouldn't learn anything and your validation set performance would drop.
  
  ### Meta-learning
  
  [*"Learning to Reweight Examples for Robust Deep Learning"*](https://arxiv.org/abs/1803.09050) is one well-cited paper that does this.
  
  The intuition behind their algorithm is:
  
  > the best example weighting should minimize the loss of a set of unbiased clean validation examples that are consistent with the evaluation procedure
  
  They do this through approximate meta-learning. The weights for each sample in the training set are learnable parameters. Vanilla meta-learning would involve two optimization loops: for each update you make to the weights, you'd need to do a number of iterations for the actual training set. They approximate this by just looking at gradients of the sample weights wrt the validation loss in a *single* iteration. A large negative gradient would mean that increasing the weight of that sample would lead to a large decrease in the validation loss, so you should weigh that sample more heavily. Note that they *do* normalize the sample weights here such that weights over each batch sum to one.
  
  There's a ton of literature on sample weighting, including techniques that don't use meta-learning (e.g., [*Just Train Twice*, which weighs points that models misclassify more heavily](https://arxiv.org/abs/2107.09044)), or involve [learning a model to predict weights](https://arxiv.org/abs/1902.07379). I'd recommend you follow the citation trail, or look sample reweighting for more work on this topic.
  
  ## Outlier detection and removal
  
  You may also want to consider using a different technique, rather than sample reweighting, especially because the goal of the technique tends to be improving robustness/generalization and reducing the impact of training set bias. In fact, these techniques often weigh difficult samples more *highly*.
  
  [This paper](https://arxiv.org/abs/2210.15183) looks at this issue with *Just Train Twice*, where unlearnable or OOD samples end up getting weighed heavily. They tackle this by detecting and removing outliers based on the [Mahalanobis distance](https://en.wikipedia.org/wiki/Mahalanobis_distance) on the hidden representations of the network (based on [this paper](https://papers.nips.cc/paper_files/paper/2018/hash/abdeb6f575ac5c6676b747bca8d09cc2-Abstract.html)). [Other work involves looking at the softmax probabilities of the predicted class or training a classifier to predict OOD samples.](https://arxiv.org/abs/1610.02136)
  
  There's also a ton of work on this, and the papers I mentioned are by no means comprehensive. I'd recommend taking a look at work on [OOD sample detection](https://paperswithcode.com/task/out-of-distribution-detection), outlier detection, or handling mislabeled data.
  
  > **Comments:**
  > **username_0**: Thanks for this excellent reply @username_1. Just wanted to provide one quick clarification (specifically regarding your last section). None of the samples are outliers as such, its more that they all sit on a spectrum of predictability (which is also a function of some variables). E.g. consider image classification, but with a twist where the object to classify can be very close (occupying a large number of pixels),  very far away (small number of pixels) or anywhere in between. Making some images easier/harder to classify then others. (Just an analogy, not actually what I'm trying to do)  
  
  ---
  
  