Answers.Body: <blockquote>
  <p>Successively, we get a new training picture, which we want to use to train the FFNN even further. However, wouldn't this new training picture destroy the previously learned weights, which have been calibrated to recognize the former training pictures?</p>
  </blockquote>
  <p>This can happen, and happen to various degrees depending on how the neural network is set up, but it is usually something you want to avoid.</p>
  <p>Provided a neural network has enough <em>capacity</em> (in terms of number of free parameters), then the function it generates can be flexible enough to approximate the &quot;true&quot; function of the problem you are trying to match to data. Then as long as the learning steps are small enough, adjustments to learn one point on the function will take a larger step towards correct answer on <em>that</em> point than they adjust away from correct answer on <em>other</em> points (intuitively this is about the mutually good solutions being in part orthogonal and/or correlated, so a step towards A does not necessarily move the same distance away from B). Repeating the points again and again will allow the NN to match <em>all</em> the points over time.</p>
  <p>Often this is possible to do almost exactly for a large amount of data at the same time. However, this is the opposite problem to your worries in the question, and is considered a bad thing because it can cause <a href="https://en.wikipedia.org/wiki/Overfitting" rel="nofollow noreferrer">overfitting</a>. Overfitting is very easy to achieve with neural networks because they are so flexible and can have large numbers of parameters.</p>
  <p>Most often, the goal of training a supervised learner, such as a neural network, is not to exactly match the training data, but to <em>generalise</em> to new data from the same population distribution. You will typically spend far more time and effort combatting overfitting when training a NN, than worrying about lack of capacity to handle multiple data points. Methods to combat overfitting are:</p>
  <ul>
  <li>Regularisation. Techniques that adjust a learning algorithm, typically by adding some constraint.</li>
  <li>Cross validation. Measuring results of training against unseen data, and picking the best generalised model.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-04-29T10:40:37.917Z
Answers.LastActivityDate: 2020-12-20T12:27:48.880Z
Answers.LastEditDate: 2020-12-20T12:27:48.880Z
Answers.UserDisplayName: Neil Slater
Body: <p>Let's say an image has 28*28 pixels, which leads to 784 input nodes in a feed-forward neural network. If an image can be classified into 1 of 10 numbers (e.g. <a href="https://en.wikipedia.org/wiki/MNIST_database" rel="nofollow noreferrer">MNIST</a>), there are 10 output nodes.</p>
  <p>We train (with gradient descent and back-propagation) the FFNN with a set of known pictures until we get a good accuracy.</p>
  <p>Successively, we get a new training picture, which we want to use to train the FFNN even further. However, wouldn't this new training picture destroy the previously learned weights, which have been calibrated to recognize the former training pictures?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-20T12:26:01.923Z
Comments.Text: This question seems to be a duplicate of [this](https://ai.stackexchange.com/q/13289/2444), which is a newer question/post, but has received a lot more attention.
Comments.UserDisplayName: nbro
Tags: <neural-networks><training><image-recognition><feedforward-neural-networks><catastrophic-forgetting>
Title: Could new training pictures destroy the trained weights of the neural network?
UserDisplayName: David
ThreadText: # Could new training pictures destroy the trained weights of the neural network?
  
  (Asked by: username_0 on 2018-04-29)
  
  Let's say an image has 28\*28 pixels, which leads to 784 input nodes in a feed-forward neural network. If an image can be classified into 1 of 10 numbers (e.g. [MNIST](https://en.wikipedia.org/wiki/MNIST_database)), there are 10 output nodes.
  
  We train (with gradient descent and back-propagation) the FFNN with a set of known pictures until we get a good accuracy.
  
  Successively, we get a new training picture, which we want to use to train the FFNN even further. However, wouldn't this new training picture destroy the previously learned weights, which have been calibrated to recognize the former training pictures?
  
  **Tags:** neural-networks, training, image-recognition, feedforward-neural-networks, catastrophic-forgetting
  
  > **Comments:**
  > **username_1**: This question seems to be a duplicate of [this](https://ai.stackexchange.com/q/13289/2444), which is a newer question/post, but has received a lot more attention.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2018-04-29. Score: 1
  
  > Successively, we get a new training picture, which we want to use to train the FFNN even further. However, wouldn't this new training picture destroy the previously learned weights, which have been calibrated to recognize the former training pictures?
  
  This can happen, and happen to various degrees depending on how the neural network is set up, but it is usually something you want to avoid.
  
  Provided a neural network has enough *capacity* (in terms of number of free parameters), then the function it generates can be flexible enough to approximate the "true" function of the problem you are trying to match to data. Then as long as the learning steps are small enough, adjustments to learn one point on the function will take a larger step towards correct answer on *that* point than they adjust away from correct answer on *other* points (intuitively this is about the mutually good solutions being in part orthogonal and/or correlated, so a step towards A does not necessarily move the same distance away from B). Repeating the points again and again will allow the NN to match *all* the points over time.
  
  Often this is possible to do almost exactly for a large amount of data at the same time. However, this is the opposite problem to your worries in the question, and is considered a bad thing because it can cause [overfitting](https://en.wikipedia.org/wiki/Overfitting). Overfitting is very easy to achieve with neural networks because they are so flexible and can have large numbers of parameters.
  
  Most often, the goal of training a supervised learner, such as a neural network, is not to exactly match the training data, but to *generalise* to new data from the same population distribution. You will typically spend far more time and effort combatting overfitting when training a NN, than worrying about lack of capacity to handle multiple data points. Methods to combat overfitting are:
  
  * Regularisation. Techniques that adjust a learning algorithm, typically by adding some constraint.
  * Cross validation. Measuring results of training against unseen data, and picking the best generalised model.
  
  
  ---
  
  