Answers.Body: <p>We typically think of machine learning models as modeling two different parts of the training data--the underlying generalizable truth (the signal), and the randomness specific to that dataset (the noise).</p>
  
  <p>Fitting both of those parts increases training set accuracy, but fitting the signal also increases test set accuracy (and real-world performance) while fitting the noise decreases both. So we use things like regularization and dropout and similar techniques in order to make it harder to fit the noise, and so more likely to fit the signal.</p>
  
  <p>Just increasing the amount of noise in the training data is one such approach, but seems unlikely to be as useful. Compare random jitter to adversarial boosting, for example; the first will slowly and indirectly improve robustness whereas the latter will dramatically and directly improve it.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-02T15:48:56.970Z
Answers.LastActivityDate: 2016-08-02T15:48:56.970Z
Answers.UserDisplayName: Matthew Gray
Answers.Body: <p>PS: <em>There is already some very good answers provided here, I will merely add to this answers in the hope that someone will find this useful:</em></p>
  
  <p>Introducing noise to a dataset can indeed have a positive influence on a model. In fact this can be seen as doing the same thing that you would normally do with <a href="https://en.wikipedia.org/wiki/Regularization_(mathematics)" rel="nofollow noreferrer">regularizers</a> like <a href="https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pd" rel="nofollow noreferrer">dropout</a>. Some of the example of doing this are <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2771718/" rel="nofollow noreferrer">Zur at.al</a>, <a href="https://arxiv.org/pdf/1003.0358.pdf" rel="nofollow noreferrer">Cires¸at.al</a> where the authors successfully introduced noise into the dataset to reduce over-fitting. </p>
  
  <p>The catch is in knowing how much noise is too much. If you add too much noise, this might render your dataset useless in that the resulting dataset may no longer contain sufficient resemblance to the original dataset, so you might as well be training on a completely different dataset. Thus too much noise could be seen to cause under-fitting, just like extremely high dropout rates. </p>
  
  <p><em>As the saying goes;  <strike>change</strike> balance is the spice of life :).</em></p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-07-27T10:17:17.960Z
Answers.LastActivityDate: 2017-07-27T10:26:19.407Z
Answers.LastEditDate: 2017-07-27T10:26:19.407Z
Answers.UserDisplayName: Tshilidzi Mudau
Answers.Body: <p>Noise in the data, to a reasonable amount, may help the network to generalize better. Sometimes, it has the opposite effect. It partly depends on the kind of noise ("true" vs. artificial).</p>
  
  <p>The <a href="ftp://ftp.sas.com/pub/neural/FAQ3.html#A_noise" rel="nofollow noreferrer">AI FAQ on ANN</a> gives a good overview. Excerpt:</p>
  
  <blockquote>
    <p>Noise in the actual data is never a good thing, since it limits the accuracy of generalization that can be achieved no matter how extensive the training set is. On the other hand, injecting artificial noise (jitter) into the inputs during training is one of several ways to improve generalization for smooth functions when you have a small training set.</p>
  </blockquote>
  
  <p>In some field, such as computer vision, it's common to increase the size of the training set by copying some samples and adding some noises or other transformation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2016-08-02T15:47:02.993Z
Answers.LastActivityDate: 2019-02-23T22:36:37.133Z
Answers.LastEditDate: 2019-02-23T22:36:37.133Z
Answers.UserDisplayName: Franck Dernoncourt
Body: <p>Does increasing the noise in data help to improve the learning ability of a network? Does it make any difference or does it depend on the problem being solved? How is it affect the generalization process overall?</p>
  
Tags: <neural-networks><machine-learning><statistical-ai><generalization>
Title: How does noise affect generalization?
UserDisplayName: kenorb
ThreadText: # How does noise affect generalization?
  
  (Asked by: username_0 on 2016-08-02)
  
  Does increasing the noise in data help to improve the learning ability of a network? Does it make any difference or does it depend on the problem being solved? How is it affect the generalization process overall?
  
  **Tags:** neural-networks, machine-learning, statistical-ai, generalization
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2016-08-02. Score: 9 | Accepted answer
  
  Noise in the data, to a reasonable amount, may help the network to generalize better. Sometimes, it has the opposite effect. It partly depends on the kind of noise ("true" vs. artificial).
  
  The [AI FAQ on ANN](ftp://ftp.sas.com/pub/neural/FAQ3.html#A_noise) gives a good overview. Excerpt:
  
  > Noise in the actual data is never a good thing, since it limits the accuracy of generalization that can be achieved no matter how extensive the training set is. On the other hand, injecting artificial noise (jitter) into the inputs during training is one of several ways to improve generalization for smooth functions when you have a small training set.
  
  In some field, such as computer vision, it's common to increase the size of the training set by copying some samples and adding some noises or other transformation.
  
  
  ---
  
  ### Answer by username_1 on 2016-08-02. Score: 9
  
  We typically think of machine learning models as modeling two different parts of the training data--the underlying generalizable truth (the signal), and the randomness specific to that dataset (the noise).
  
  Fitting both of those parts increases training set accuracy, but fitting the signal also increases test set accuracy (and real-world performance) while fitting the noise decreases both. So we use things like regularization and dropout and similar techniques in order to make it harder to fit the noise, and so more likely to fit the signal.
  
  Just increasing the amount of noise in the training data is one such approach, but seems unlikely to be as useful. Compare random jitter to adversarial boosting, for example; the first will slowly and indirectly improve robustness whereas the latter will dramatically and directly improve it.
  
  
  ---
  
  ### Answer by username_2 on 2017-07-27. Score: 1
  
  PS: *There is already some very good answers provided here, I will merely add to this answers in the hope that someone will find this useful:*
  
  Introducing noise to a dataset can indeed have a positive influence on a model. In fact this can be seen as doing the same thing that you would normally do with [regularizers](https://en.wikipedia.org/wiki/Regularization_(mathematics)) like [dropout](https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pd). Some of the example of doing this are [Zur at.al](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2771718/), [Cires¸at.al](https://arxiv.org/pdf/1003.0358.pdf) where the authors successfully introduced noise into the dataset to reduce over-fitting.
  
  The catch is in knowing how much noise is too much. If you add too much noise, this might render your dataset useless in that the resulting dataset may no longer contain sufficient resemblance to the original dataset, so you might as well be training on a completely different dataset. Thus too much noise could be seen to cause under-fitting, just like extremely high dropout rates.
  
  *As the saying goes; change balance is the spice of life :).*
  
  
  ---
  
  