Answers.Body: <p>The easiest approach to give more importance to a group of specific training instances is simply using a weight to increase the error loss computed on those specific instances during training. Libraries like sklearn have implemented off the shelf the <a href="https://scikit-learn.org/stable/glossary.html?highlight=sample_weight#glossary-sample-props" rel="nofollow noreferrer">sample_weight</a> parameter to perform precisely this. The obvious downside of this approach is that you need to know before hand which training instances are hard to learn for the model, and you need a metric to translate that difficulty level into a loss weight.</p>
  <p>More fancy approaches try to boost accuracy for instances hard to tackle by trying to better approximate this &quot;hardness level&quot; and then ranking the training instances from the easiest to the hardest, logic being that this order will facilitate gradient descent convergence. The main two approaches are:</p>
  <ul>
  <li><a href="https://papers.nips.cc/paper/2010/file/e57c6b956a6521b28495f2886ca0977a-Paper.pdf" rel="nofollow noreferrer">Self-Paced learning</a>, the core idea is to replace the classic loss with an objective function that can learn both, model weights and a parameter (usually indicated with <span class="math-container">$v$</span>) indicating the difficulty of each training instance.</li>
  <li><a href="https://arxiv.org/pdf/1604.03540.pdf" rel="nofollow noreferrer">Hard Examples Mining</a>, similar to active learning, a model is trained on a set of known easy examples, and then used to harvest miss classified data that are used to perform a focused retraining of the same model.</li>
  </ul>
  <p>Honestly though, these ideas are pretty old, and the fact that they're not widely used today to me suggests that the benefits are way less than the difficulties they introduce. So an in depth analysis of your data to boost standardization in term of quality (brightness/contrast adjustment, histogram equalization and so on) will probably be less of a rabbit hole and most likely still lead to better performance of your model.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-26T16:03:03.320Z
Answers.Text: I don't think I can standardize the quality of individual features in the image since I have both the "easy" labels and the "hard" labels in the same image. I can't alter one without the other and at that point I could just use more data augmentation on top of the basic mirroring I currently do, but I don't think this solves the problem of trying to get the model to focus on the hard labels. Will check out your other suggestions, thanks!
Answers.UserDisplayName: Tetraquark
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-26T16:42:26.363Z
Answers.Text: you could split the task tough. Use normal rgb for body parts easy to detect and train another model on enhanced images to compensate for dark parts to detect limbs. My point is: machine learning is data driven, if the data is crap, the model will be crap, no debate around it, and no matter what fancy approaches you try to compensate for what the data lack. So personally I live by trying to improve the data first, and leave the rest as final resource. But of curse that's just my philosophy.
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-26T15:43:34.880Z
Answers.LastActivityDate: 2022-01-26T15:43:34.880Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>I am training a model to place labels in image data. Some labels are learnt very quickly by the model while others take a long time to perfect. I cannot simply add more labeled data with only the labels I am looking to improve on since most of the image data contains a combination of the easy labels and the more difficult ones.
  Are there any smart ways to get the model to focus on the hard labels? I am just looking for some leads. Of course I can just train for longer but that seems inefficient as half the labels are already predicted almost perfectly.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-26T14:29:43.443Z
Comments.Text: are we talking about low represented classes or balanced classes but some of them are not classified properly for unknown reasons? If it's the latter case it would be helpful to provide more information about the data and some examples.
Comments.UserDisplayName: Edoardo Guerriero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-26T14:38:39.513Z
Comments.Text: The label set is balanced. Some labels are harder to place correctly on the image because they are simply more difficult to identify (since some features I am trying to label have more variance in shape/perspective, lighting conditions and motion).
Comments.UserDisplayName: Tetraquark
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-26T14:47:48.093Z
Comments.Text: if those are the main issues sounds like you need just some extra prerocessing steps to standardize your images and remove that variance. But again, hard to give suggestions without concrete examples.
Comments.UserDisplayName: Edoardo Guerriero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-26T15:43:50.133Z
Comments.Text: Well I am tracking animals frame by frame. Some bodyparts are easy to identify (eg: all the points along spine from head to tail) whereas limbs are harder because they can obscure each other and move quickly. A left front paw and right front paw can look very different depending on where is it seen from whereas the neck or top of the tail is trivial to find.
Comments.UserDisplayName: Tetraquark
Tags: <machine-learning><deep-learning><training><tensorflow><data-labelling>
Title: What techniques exist to increase the learning importance of difficult-to-learn labels over easy ones?
UserDisplayName: Tetraquark
ThreadText: # What techniques exist to increase the learning importance of difficult-to-learn labels over easy ones?
  
  (Asked by: username_0 on 2022-01-26)
  
  I am training a model to place labels in image data. Some labels are learnt very quickly by the model while others take a long time to perfect. I cannot simply add more labeled data with only the labels I am looking to improve on since most of the image data contains a combination of the easy labels and the more difficult ones.
  Are there any smart ways to get the model to focus on the hard labels? I am just looking for some leads. Of course I can just train for longer but that seems inefficient as half the labels are already predicted almost perfectly.
  
  **Tags:** machine-learning, deep-learning, training, tensorflow, data-labelling
  > **Comments:**
  > **username_1**: are we talking about low represented classes or balanced classes but some of them are not classified properly for unknown reasons? If it's the latter case it would be helpful to provide more information about the data and some examples.  
  > **username_0**: The label set is balanced. Some labels are harder to place correctly on the image because they are simply more difficult to identify (since some features I am trying to label have more variance in shape/perspective, lighting conditions and motion).  
  > **username_1**: if those are the main issues sounds like you need just some extra prerocessing steps to standardize your images and remove that variance. But again, hard to give suggestions without concrete examples.  
  > **username_0**: Well I am tracking animals frame by frame. Some bodyparts are easy to identify (eg: all the points along spine from head to tail) whereas limbs are harder because they can obscure each other and move quickly. A left front paw and right front paw can look very different depending on where is it seen from whereas the neck or top of the tail is trivial to find.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-01-26. Score: 1
  
  The easiest approach to give more importance to a group of specific training instances is simply using a weight to increase the error loss computed on those specific instances during training. Libraries like sklearn have implemented off the shelf the [sample\_weight](https://scikit-learn.org/stable/glossary.html?highlight=sample_weight#glossary-sample-props) parameter to perform precisely this. The obvious downside of this approach is that you need to know before hand which training instances are hard to learn for the model, and you need a metric to translate that difficulty level into a loss weight.
  
  More fancy approaches try to boost accuracy for instances hard to tackle by trying to better approximate this "hardness level" and then ranking the training instances from the easiest to the hardest, logic being that this order will facilitate gradient descent convergence. The main two approaches are:
  
  * [Self-Paced learning](https://papers.nips.cc/paper/2010/file/e57c6b956a6521b28495f2886ca0977a-Paper.pdf), the core idea is to replace the classic loss with an objective function that can learn both, model weights and a parameter (usually indicated with $v$) indicating the difficulty of each training instance.
  * [Hard Examples Mining](https://arxiv.org/pdf/1604.03540.pdf), similar to active learning, a model is trained on a set of known easy examples, and then used to harvest miss classified data that are used to perform a focused retraining of the same model.
  
  Honestly though, these ideas are pretty old, and the fact that they're not widely used today to me suggests that the benefits are way less than the difficulties they introduce. So an in depth analysis of your data to boost standardization in term of quality (brightness/contrast adjustment, histogram equalization and so on) will probably be less of a rabbit hole and most likely still lead to better performance of your model.
  
  > **Comments:**
  > **username_0**: I don't think I can standardize the quality of individual features in the image since I have both the "easy" labels and the "hard" labels in the same image. I can't alter one without the other and at that point I could just use more data augmentation on top of the basic mirroring I currently do, but I don't think this solves the problem of trying to get the model to focus on the hard labels. Will check out your other suggestions, thanks!  
  > **username_1**: you could split the task tough. Use normal rgb for body parts easy to detect and train another model on enhanced images to compensate for dark parts to detect limbs. My point is: machine learning is data driven, if the data is crap, the model will be crap, no debate around it, and no matter what fancy approaches you try to compensate for what the data lack. So personally I live by trying to improve the data first, and leave the rest as final resource. But of curse that's just my philosophy.  
  
  ---
  
  