Body: <p>I've come across a few binary classification problems lately where the labelling was challenging even for an expert. I'm wondering what I should do with this. Here are some of my suggestions to get the ball rolling:</p>
  <ol>
  <li>Make a third category called &quot;unsure&quot; then make it a three-class classification problem instead.</li>
  <li>Make a third category called &quot;unsure&quot; and just remove these from your training set.</li>
  <li>Make a third category called &quot;unsure&quot; and during training model this as a 0.5 such that the binary cross entropy loss looks like <span class="math-container">$-0.5\log(\hat{y})-0.5\log(1-\hat{y})$</span></li>
  <li>Allow the labeller to pick a percentage on a sliding scale (or maybe multiple choice: (0%, 25%, 50%, 75%, 100%), and take that into account when calculating cross entropy (as in my point above).</li>
  </ol>
  <p>I recently saw a <a href="https://arxiv.org/abs/1804.02555" rel="nofollow noreferrer">paper</a> which goes for option 2, although that's not enough to convince me. Here's the relevant quote:</p>
  <blockquote>
  <p>In case of a high-level risk, collision is imminent and
  the driver must react in less than 0.5 s (TTC &lt; 0.5s). For
  low-level risk, the TTC is more than 2.0 s (TTC &gt; 2.0s).
  Videos that show intermediate-level risk (0.5 s ≤ TTC
  ≤ 2.0 s), which is a mixture of high- and low-level risks,
  were not included in the NIDB because when training
  a convnet, it must be possible to make a clear visual
  distinction of risk.</p>
  </blockquote>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-23T23:34:35.210Z
Comments.Text: Your problem seems to be related to [weakly supervised learning](https://academic.oup.com/nsr/article/5/1/44/4093912) (WSL). Maybe you get an answer to your question by reading about WSL.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-02T18:31:58.637Z
Comments.Text: Duplicates:   https://stats.stackexchange.com/questions/74042/classifier-for-uncertain-class-labels and  https://stats.stackexchange.com/questions/218656/classification-with-noisy-labels
Comments.UserDisplayName: kjetil b halvorsen
Tags: <neural-networks><classification><data-labelling>
Title: For binary classification learning problems, how should I label instances where I'm only 60% sure?
UserDisplayName: Alexander Soare
ThreadText: # For binary classification learning problems, how should I label instances where I'm only 60% sure?
  
  (Asked by: username_0 on 2021-01-22)
  
  I've come across a few binary classification problems lately where the labelling was challenging even for an expert. I'm wondering what I should do with this. Here are some of my suggestions to get the ball rolling:
  
  1. Make a third category called "unsure" then make it a three-class classification problem instead.
  2. Make a third category called "unsure" and just remove these from your training set.
  3. Make a third category called "unsure" and during training model this as a 0.5 such that the binary cross entropy loss looks like $-0.5\log(\hat{y})-0.5\log(1-\hat{y})$
  4. Allow the labeller to pick a percentage on a sliding scale (or maybe multiple choice: (0%, 25%, 50%, 75%, 100%), and take that into account when calculating cross entropy (as in my point above).
  
  I recently saw a [paper](https://arxiv.org/abs/1804.02555) which goes for option 2, although that's not enough to convince me. Here's the relevant quote:
  
  > In case of a high-level risk, collision is imminent and
  > the driver must react in less than 0.5 s (TTC < 0.5s). For
  > low-level risk, the TTC is more than 2.0 s (TTC > 2.0s).
  > Videos that show intermediate-level risk (0.5 s ≤ TTC
  > ≤ 2.0 s), which is a mixture of high- and low-level risks,
  > were not included in the NIDB because when training
  > a convnet, it must be possible to make a clear visual
  > distinction of risk.
  
  **Tags:** neural-networks, classification, data-labelling
  > **Comments:**
  > **username_1**: Your problem seems to be related to [weakly supervised learning](https://academic.oup.com/nsr/article/5/1/44/4093912) (WSL). Maybe you get an answer to your question by reading about WSL.  
  > **username_2**: Duplicates:   https://stats.stackexchange.com/questions/74042/classifier-for-uncertain-class-labels and  https://stats.stackexchange.com/questions/218656/classification-with-noisy-labels  
  
  ---
  
  ## 0 answers
  
  