Answers.Body: <p>This is referred to as <em><strong>label noise</strong></em> (or <em>noisy labels</em>), and it is indeed different from just &quot;noisy data&quot;; quoting from the 2014 paper <a href="https://www.researchgate.net/publication/261601383_Classification_in_the_Presence_of_Label_Noise_A_Survey" rel="nofollow noreferrer">Classiﬁcation in the Presence of Label Noise: a Survey</a>:</p>
  <blockquote>
  <p>In this survey,
  it is assumed that each training sample is associated with an
  observed label. This label often corresponds to the true class of
  the sample, but it may be <em>subjected to a noise process before
  being presented to the learning algorithm</em>. It is therefore
  important to distinguish the true class of an instance from its
  observed label. The process which pollutes labels is called
  label noise and must be separated from feature (or attribute)
  noise which affects the value of features.</p>
  </blockquote>
  <p>You may also find the (preprint) paper <a href="https://arxiv.org/abs/2003.10471" rel="nofollow noreferrer">Label Noise Types and Their Effects on Deep Learning</a> useful; there, the authors propose a taxonomy of different types of label noise:</p>
  <blockquote>
  <p>Label noise types can be subdivided into
  three main groups as follows.</p>
  <p>• <strong>Uniform noise</strong>: Flipping probability of label from its
  true class to any other class is equally distributed. Many
  works in literature use synthetic uniform label noise by
  just flipping labels randomly for a given percentage of
  data instances [21]–[24].</p>
  <p>• <strong>Class-dependent noise</strong>: Flipping probability of label
  depends on the true class of the data instance. This
  is mostly represented by a confusion matrix and can
  be designed in different ways. The easiest way is to
  attain inter-class transition probabilities just random [25],
  so that there is still class dependence since transition
  probabilities are given according to classes but without
  any correlation to class similarities. In a more structured
  way, noise transition matrix can be designed in a way that
  similar classes have a bigger probability to be flipped to
  each other [26]–[30]. Some works use pairwise noise,
  in which transition from one class can only be defined
  to one another class [31]–[35]. Work of [36] checks the
  popularity of classes and constructs transition matrix so
  that mislabeling happens from popular class to unpopular
  class or vice versa.</p>
  <p>• <strong>Feature-dependent noise</strong>: The probability of mislabeling
  depends on features of instances. In order to generate
  feature-dependent noise, features of each instance should
  be extracted, and their similarities to other instances from
  different classes should be evaluated. Unlike uniform and
  class-dependent noise, there are much fewer implemen-
  tations of synthetic feature-dependent label noise. One
  particular work in this field is [37], where data is clustered
  with the kNN algorithm, and labels are flipped randomly
  for clusters of data. This method provides concentrated
  noise in the feature space. But, this type of synthetic noise
  doesn’t evaluate the instance similarities and therefore
  different from our proposed approach. Alternatively, in
  case there is a surrounding text for each image in the
  dataset, some works create noisy labels from the interpretations of these texts [38]–[41], assuming surrounding
  texts are related to features of data. But this approach is
  restricted to datasets with surrounding user-defined texts,
  which is not the case for most of the time.</p>
  </blockquote>
  <p>Other possibly useful resources on the subject:</p>
  <ul>
  <li><a href="https://www.cs.bham.ac.uk/%7Eaxk/ESANNtut.pdf" rel="nofollow noreferrer">A Comprehensive Introduction to Label Noise</a> (conference paper, 2014)</li>
  <li><a href="http://knowdive.disi.unitn.it/2018/09/a-little-talk-on-label-noise/" rel="nofollow noreferrer">A Little Talk on Label Noise</a> (blog post, 2018)</li>
  <li><a href="https://towardsdatascience.com/data-noise-and-label-noise-in-machine-learning-98c8a3c8322e" rel="nofollow noreferrer">Data Noise and Label Noise in Machine Learning</a> (blog post, 2021)</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-30T14:48:23.113Z
Answers.LastActivityDate: 2022-01-30T14:59:11.173Z
Answers.LastEditDate: 2022-01-30T14:59:11.173Z
Answers.UserDisplayName: desertnaut
Body: <p>I wonder if following characteristic of data has some standard &quot;professional&quot; or scientific term associated with it.</p>
  <p>Let's assume that I have a set of dog/cat images labeled 0 for a cat and 1 for a dog. My purpose is to extract (classify) as many true dog images as possible, with the smallest amount of non-dog images. But the problem with data is that:</p>
  <ul>
  <li>some dogs images are labeled as 0</li>
  <li>some cats images are labeled as 1</li>
  <li>some images are of other animals or non-animal objects and can have both labels 1 and 0</li>
  </ul>
  <p>Is there any specific term describing such cases, or they are  just &quot;noisy&quot; data?</p>
  
Tags: <classification><terminology><datasets><data-labelling><training-datasets>
Title: Is there a standard term for the following flaw in the data?
UserDisplayName: GKozinski
ThreadText: # Is there a standard term for the following flaw in the data?
  
  (Asked by: username_0 on 2022-01-29)
  
  I wonder if following characteristic of data has some standard "professional" or scientific term associated with it.
  
  Let's assume that I have a set of dog/cat images labeled 0 for a cat and 1 for a dog. My purpose is to extract (classify) as many true dog images as possible, with the smallest amount of non-dog images. But the problem with data is that:
  
  * some dogs images are labeled as 0
  * some cats images are labeled as 1
  * some images are of other animals or non-animal objects and can have both labels 1 and 0
  
  Is there any specific term describing such cases, or they are just "noisy" data?
  
  **Tags:** classification, terminology, datasets, data-labelling, training-datasets
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-01-30. Score: 0 | Accepted answer
  
  This is referred to as ***label noise*** (or *noisy labels*), and it is indeed different from just "noisy data"; quoting from the 2014 paper [Classiﬁcation in the Presence of Label Noise: a Survey](https://www.researchgate.net/publication/261601383_Classification_in_the_Presence_of_Label_Noise_A_Survey):
  
  > In this survey,
  > it is assumed that each training sample is associated with an
  > observed label. This label often corresponds to the true class of
  > the sample, but it may be *subjected to a noise process before
  > being presented to the learning algorithm*. It is therefore
  > important to distinguish the true class of an instance from its
  > observed label. The process which pollutes labels is called
  > label noise and must be separated from feature (or attribute)
  > noise which affects the value of features.
  
  You may also find the (preprint) paper [Label Noise Types and Their Effects on Deep Learning](https://arxiv.org/abs/2003.10471) useful; there, the authors propose a taxonomy of different types of label noise:
  
  > Label noise types can be subdivided into
  > three main groups as follows.
  >
  > • **Uniform noise**: Flipping probability of label from its
  > true class to any other class is equally distributed. Many
  > works in literature use synthetic uniform label noise by
  > just flipping labels randomly for a given percentage of
  > data instances [21]–[24].
  >
  > • **Class-dependent noise**: Flipping probability of label
  > depends on the true class of the data instance. This
  > is mostly represented by a confusion matrix and can
  > be designed in different ways. The easiest way is to
  > attain inter-class transition probabilities just random [25],
  > so that there is still class dependence since transition
  > probabilities are given according to classes but without
  > any correlation to class similarities. In a more structured
  > way, noise transition matrix can be designed in a way that
  > similar classes have a bigger probability to be flipped to
  > each other [26]–[30]. Some works use pairwise noise,
  > in which transition from one class can only be defined
  > to one another class [31]–[35]. Work of [36] checks the
  > popularity of classes and constructs transition matrix so
  > that mislabeling happens from popular class to unpopular
  > class or vice versa.
  >
  > • **Feature-dependent noise**: The probability of mislabeling
  > depends on features of instances. In order to generate
  > feature-dependent noise, features of each instance should
  > be extracted, and their similarities to other instances from
  > different classes should be evaluated. Unlike uniform and
  > class-dependent noise, there are much fewer implemen-
  > tations of synthetic feature-dependent label noise. One
  > particular work in this field is [37], where data is clustered
  > with the kNN algorithm, and labels are flipped randomly
  > for clusters of data. This method provides concentrated
  > noise in the feature space. But, this type of synthetic noise
  > doesn’t evaluate the instance similarities and therefore
  > different from our proposed approach. Alternatively, in
  > case there is a surrounding text for each image in the
  > dataset, some works create noisy labels from the interpretations of these texts [38]–[41], assuming surrounding
  > texts are related to features of data. But this approach is
  > restricted to datasets with surrounding user-defined texts,
  > which is not the case for most of the time.
  
  Other possibly useful resources on the subject:
  
  * [A Comprehensive Introduction to Label Noise](https://www.cs.bham.ac.uk/%7Eaxk/ESANNtut.pdf) (conference paper, 2014)
  * [A Little Talk on Label Noise](http://knowdive.disi.unitn.it/2018/09/a-little-talk-on-label-noise/) (blog post, 2018)
  * [Data Noise and Label Noise in Machine Learning](https://towardsdatascience.com/data-noise-and-label-noise-in-machine-learning-98c8a3c8322e) (blog post, 2021)
  
  
  ---
  
  