Answers.Body: <p>This depends on the behaviour you want. If the ambiguous sample's ground truth is classified by a range of people, your network will get an average* based on that group. If it's only by one person, your network will be biased to how that one person classifies these samples.</p>
  <p>Alternatively, depending on your loss function, you could train the network to classify ambiguous samples with an ambiguous label. For example, if 40 people label the digit as 1, and 60 as 7, you could have the desired output for the network be 0.4 for 1 and 0.6 for 7 (assuming it's a probability).</p>
  <p>This doesn't have an exact answer, it's whatever behaviour you deem best for your scenario. If you want to keep it simple, you can remove the samples and then see how the network performs on these ambiguous samples at testing time. Assuming your dataset is good, you'll probably find the network performs about the same on these samples anyway.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-28T03:54:24.427Z
Answers.LastActivityDate: 2021-09-28T03:54:24.427Z
Answers.UserDisplayName: Recessive
Body: <p>Imagine that I am training a model to classify handwritten digits. Suppose there are some bad quality images that could be classified by a human as either 0 or 8, 1 or 7 or other commonly misclassified pair of digits.
  My question is, should I simply remove such ambiguous samples? Should I annotate it as the most similar digit, even though there are other similar answers? Should I present it repeat the sample, presenting it once per each 'acceptable' answer?</p>
  
Tags: <neural-networks><classification><data-labelling>
Title: Should I train my network for classification on samples whose ground truth label is ambiguous?
UserDisplayName: Manveru
ThreadText: # Should I train my network for classification on samples whose ground truth label is ambiguous?
  
  (Asked by: username_0 on 2021-09-28)
  
  Imagine that I am training a model to classify handwritten digits. Suppose there are some bad quality images that could be classified by a human as either 0 or 8, 1 or 7 or other commonly misclassified pair of digits.
  My question is, should I simply remove such ambiguous samples? Should I annotate it as the most similar digit, even though there are other similar answers? Should I present it repeat the sample, presenting it once per each 'acceptable' answer?
  
  **Tags:** neural-networks, classification, data-labelling
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-09-28. Score: 1
  
  This depends on the behaviour you want. If the ambiguous sample's ground truth is classified by a range of people, your network will get an average\* based on that group. If it's only by one person, your network will be biased to how that one person classifies these samples.
  
  Alternatively, depending on your loss function, you could train the network to classify ambiguous samples with an ambiguous label. For example, if 40 people label the digit as 1, and 60 as 7, you could have the desired output for the network be 0.4 for 1 and 0.6 for 7 (assuming it's a probability).
  
  This doesn't have an exact answer, it's whatever behaviour you deem best for your scenario. If you want to keep it simple, you can remove the samples and then see how the network performs on these ambiguous samples at testing time. Assuming your dataset is good, you'll probably find the network performs about the same on these samples anyway.
  
  
  ---
  
  