Answers.Body: <p>I will break it down for you in very simple words. The accuracy will drop down as you label them wrong. In simpler words- accuracy is directly proportional on how perfect the data is labelled. If you think about it, suppose you have 2 categories-cats and dogs, and you have a dataset of 10,000 pictures. Out of which 50 are wrongly labelled. The accuracy will less than the perfectly labelled but not that less since the neural network built will not be that bad. But suppose now you have 1000 wrongly labelled which is 1/10 of the dataset, then the NN will have more abrupt outcomes.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-03T19:03:27.807Z
Answers.LastActivityDate: 2020-12-18T17:02:44.233Z
Answers.LastEditDate: 2020-12-18T17:02:44.233Z
Answers.UserDisplayName: Swapnil Tiwari
Answers.Body: <p>I think the crucial point here is what you precisely mean by <em>mislabelled</em>. Google's image classifier will likely do a 'pretty good' job of retrieving images with the given subject included, but how strict or lenient your class requisites are is quite important. For example, if one of your classes is 'dog' there may be hundreds of images procured from scraping that could display (examples off the top of my head, but you can get even more creative): </p>
  
  <ul>
  <li>Ancient canine fossils</li>
  <li>Wolves</li>
  <li>Stuffed animal dogs</li>
  <li>Hard mode dogs (i.e. partially occluded, variable lighting schemas, background color variation, intraclass/species variation)</li>
  </ul>
  
  <p>Additionally, your computational tool will impact this. If you're using a neural network, the above issues can to some extent be accounted for, but a linear classifier would likely have difficulty adopting a broader/flexible view of your class. </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-05T05:06:09.620Z
Answers.LastActivityDate: 2017-12-18T20:38:17.267Z
Answers.LastEditDate: 2017-12-18T20:38:17.267Z
Answers.UserDisplayName: Ubermensch_010
Body: <p>Collecting and labeling training data for supervised learning tasks is incredibly time-consuming and costly.</p>
  
  <p>For instance, let's say you wrote a script that went on Google images and got you 5000 pictures for each of 10 classes. You then use an unsupervised algorithm to cluster them. Then, you train a supervised algorithm using the labels from the scraper as ground truth. Obviously, your network will perform more poorly than one with perfectly labeled data, but is there a way to guesstimate how much?</p>
  
  <p>Perhaps there are 50 mislabeled images in each class. That would most likely be better than 500 mislabeled images, but I'm wondering if there is a way to predict how much (even if it is by someone's rules of thumb or something like that).</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-03T20:16:18.497Z
Comments.Text: There exists a multitude of work on dealing with label noisy data if you look into it. One approach that’s super easy that I find useful was relating to a paper that came out last year in creating a super small gold label classifier and then use that as a prior for the larger model
Comments.UserDisplayName: mshlis
Tags: <neural-networks><machine-learning><datasets><labeled-datasets><data-labelling>
Title: What is the effect of mislabeled training data?
UserDisplayName: pshlady
ThreadText: # What is the effect of mislabeled training data?
  
  (Asked by: username_0 on 2017-12-05)
  
  Collecting and labeling training data for supervised learning tasks is incredibly time-consuming and costly.
  
  For instance, let's say you wrote a script that went on Google images and got you 5000 pictures for each of 10 classes. You then use an unsupervised algorithm to cluster them. Then, you train a supervised algorithm using the labels from the scraper as ground truth. Obviously, your network will perform more poorly than one with perfectly labeled data, but is there a way to guesstimate how much?
  
  Perhaps there are 50 mislabeled images in each class. That would most likely be better than 500 mislabeled images, but I'm wondering if there is a way to predict how much (even if it is by someone's rules of thumb or something like that).
  
  **Tags:** neural-networks, machine-learning, datasets, labeled-datasets, data-labelling
  
  > **Comments:**
  > **username_1**: There exists a multitude of work on dealing with label noisy data if you look into it. One approach that’s super easy that I find useful was relating to a paper that came out last year in creating a super small gold label classifier and then use that as a prior for the larger model  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2017-12-05. Score: 3 | Accepted answer
  
  I think the crucial point here is what you precisely mean by *mislabelled*. Google's image classifier will likely do a 'pretty good' job of retrieving images with the given subject included, but how strict or lenient your class requisites are is quite important. For example, if one of your classes is 'dog' there may be hundreds of images procured from scraping that could display (examples off the top of my head, but you can get even more creative):
  
  * Ancient canine fossils
  * Wolves
  * Stuffed animal dogs
  * Hard mode dogs (i.e. partially occluded, variable lighting schemas, background color variation, intraclass/species variation)
  
  Additionally, your computational tool will impact this. If you're using a neural network, the above issues can to some extent be accounted for, but a linear classifier would likely have difficulty adopting a broader/flexible view of your class.
  
  
  ---
  
  ### Answer by username_2 on 2019-08-03. Score: 1
  
  I will break it down for you in very simple words. The accuracy will drop down as you label them wrong. In simpler words- accuracy is directly proportional on how perfect the data is labelled. If you think about it, suppose you have 2 categories-cats and dogs, and you have a dataset of 10,000 pictures. Out of which 50 are wrongly labelled. The accuracy will less than the perfectly labelled but not that less since the neural network built will not be that bad. But suppose now you have 1000 wrongly labelled which is 1/10 of the dataset, then the NN will have more abrupt outcomes.
  
  
  ---
  
  