Answers.Body: <p>Yes, the problem of <em>forgetting</em> older training examples is a characteristic of Neural Networks. I wouldn't call it a "flaw" though because it helps them be more adaptive and allows for interesting applications such as transfer learning (if a network remembered old training too well, fine tuning it to new data would be meaningless).</p>
  
  <p>In practice what you want to do is to mix the training examples for <em>dangerous</em> and <em>not dangerous</em> so that it doesn't see one category in the beginning and one at the end.</p>
  
  <p>A standard training procedure would work like this:</p>
  
  
  
  <pre><code>for e in epochs:
      shuffle dataset
      for x_batch, y_batch in dataset:
          train neural_network on x_batxh, y_batch
  </code></pre>
  
  <p>Note that the shuffle at every epoch guarantees that the network won't see the same training examples in the same order every epoch and that the classes will be mixed</p>
  
  <p>Now to answer your question, yes decreasing the learning rate would make the network less prone to <em>forgetting</em> its previous training, but how would this work in a non-online setting? In order for a network to converge it needs multiple epochs of training (i.e. seeing each sample in the dataset many times).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-10T08:29:04.967Z
Answers.LastActivityDate: 2019-07-10T08:29:04.967Z
Answers.UserDisplayName: Djib2011
Answers.Body: <p>What you are describing sounds like it could be a deliberate case of <a href="http://wiki.fast.ai/index.php/Fine_tuning" rel="noreferrer">fine-tuning</a>.</p>
  
  <p>There is a fundamental assumption that makes <a href="https://adventuresinmachinelearning.com/stochastic-gradient-descent/" rel="noreferrer">minibatch gradient descent</a> work for learning problems: It is assumed that any batch or temporal window of consecutive batches forms a decent approximation of the true <em>global</em> gradient of the error function with respect to any parameterization of the model. If the error surface itself is moving in a big way, that would thwart the purposes of gradient descent--since gradient descent is a local refinement algorithm, all bets are off when you suddenly change the underlying distribution.
  In the example you cited, catastrophic forgetting seems like it would be an after-effect of having "forgotten" data points previously seen, and is either a symptom of the distribution having changed, or of under-representation in the data of some important phenomenon, such that it is rarely seen relative to its importance.</p>
  
  <p>Experience replay from reinforcement learning is a relevant concept that transfers well to this domain. Here is <a href="https://arxiv.org/abs/1811.11682" rel="noreferrer">a paper</a> that explores this concept with respect to catastrophic forgetting. As long as sampling represents the true gradients sufficiently well (look at <a href="https://towardsdatascience.com/handling-imbalanced-datasets-in-deep-learning-f48407a0e758" rel="noreferrer">training sample balancing</a> for this) and the model has enough parameters, the catastrophic forgetting problem is unlikely to occur. In randomly shuffled datasets with replacement, it is most likely to occur where datapoints of a particular class are so rare that they are unlikely to be included for a long time during training, effectively fine-tuning the model to a different problem until a matching sample is seen again.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-11T01:21:03.087Z
Answers.LastActivityDate: 2019-07-11T01:21:03.087Z
Answers.UserDisplayName: pygosceles
Answers.Body: <p>Maybe in theory, but not in practice. The thing is you seem to consider only chronological/sequential training. </p>
  
  <p>And there are two ways to view this issue:</p>
  
  <ol>
  <li>online learning -> then it is a feature of the method</li>
  <li>offline learning -> it does not happen thanks to several order randomizations</li>
  </ol>
  
  <p><hr>
  1. Online-Training or <a href="https://en.wikipedia.org/wiki/Online_machine_learning" rel="nofollow noreferrer">Online Machine Learning</a>.</p>
  
  <p>Using the <a href="https://en.wikipedia.org/wiki/Vowpal_Wabbit" rel="nofollow noreferrer">woppal wabbit library</a>. It is a <strong>feature</strong> (not an issue like you consider) of this library to adapt chronologically to the input it is fed with. </p>
  
  <p>I insist: it is a feature to adapt chronologically. It is wanted that when you start only telling him that lions are dangerous, then that it adapts consequently.</p>
  
  <p><hr>
  2. Offline-Training</p>
  
  <p>In my personal experience, I have used only <strong>randomized</strong> subsets of my input data as training set. And this randomization is <strong>crucial</strong>.</p>
  
  <p>Randomizations happens namely:</p>
  
  <ul>
  <li>during the training of the neural network, each epochs generally randomize the dataset order</li>
  <li>during cross-validation, randomization is used as a way to evaluate a robust model that generalises well and does not overfit</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-29T08:28:05.917Z
Answers.Text: This does not answer the OP's question ("Are neural networks prone to catastrophic forgetting?") at all. Except perhaps for the first line: "Maybe in theory, but not in practice." - which is incorrect. Neural networks are indeed prone to catastrophic forgetting. Please back up your claims with sources.
Answers.UserDisplayName: Mathias Müller
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-29T09:04:20.523Z
Answers.Text: @MathiasMüller I probably misunderstand your comment. Please could you explain how "Online Machine Learning" and "Cross Validation" are unrelated: 1/ to my statement 2/ to the original question . Thanks for detailing your contradiction just sufficiently.
Answers.UserDisplayName: Stephane Rolland
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-29T09:52:17.697Z
Answers.Text: My apologies if I there is a misunderstanding on my part, but cross validation is not a solution to catastrophic forgetting. Cross validation is a method for robust model selection. Regardless of whether cross-validation is used or not, catastrophic forgetting is not an issue in offline training settings because 1) the training set is fixed and 2) randomly shuffled so that for each epoch, examples are presented in a different order.
Answers.UserDisplayName: Mathias Müller
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-30T17:10:59.220Z
Answers.Text: I have looked at my notes from the Machine Learning course by Andrew Ng , and it is not clear from my notes that Epoching is aimed at shuffling the training set as you suggest. Do you have sources for this claim ? The definition I found is: one epoch is when the entire dataset has been fed into the backpropagation algorithm. It may very well be shuffled in modern implementations (like you suggest), I just don't have something backing this up.
Answers.UserDisplayName: Stephane Rolland
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-30T17:20:50.920Z
Answers.Text: I'll edit my answer including your point about the shuffling of each epoch, if you provide me info about it.
Answers.UserDisplayName: Stephane Rolland
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-30T17:36:25.447Z
Answers.Text: Yes, one epoch is defined as having presented the entire training set once to the model that is being trained. "Epoching is aimed at shuffling the training set " - I don't understand sorry!
Answers.UserDisplayName: Mathias Müller
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-30T17:36:51.917Z
Answers.Text: "Do you have sources for this claim?" yes, of course. Here is a good answer: https://stats.stackexchange.com/a/272412/228948
Answers.UserDisplayName: Mathias Müller
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-30T17:44:51.993Z
Answers.Text: @MathiasMüller Ok. Shuffling at every or each X epochs is something greatly recommended. I will revise the answer and include your point, thanks for the link.
Answers.UserDisplayName: Stephane Rolland
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-03T10:11:40.813Z
Answers.Text: @MathiasMüller I have edited the answer according to your remark.
Answers.UserDisplayName: Stephane Rolland
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-11T11:23:56.817Z
Answers.LastActivityDate: 2020-02-03T10:08:57.373Z
Answers.LastEditDate: 2020-02-03T10:08:57.373Z
Answers.UserDisplayName: Stephane Rolland
Answers.Body: <p>Yes, indeed, neural networks are very prone to <a href="https://www.sciencedirect.com/science/article/pii/S0079742108605368?via%3Dihub" rel="noreferrer">catastrophic forgetting (or interference)</a>. Currently, this problem is often ignored because neural networks are mainly trained offline (sometimes called <em>batch training</em>), where this problem does not often arise, and not <a href="https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2016-19.pdf" rel="noreferrer">online or incrementally</a>, which is <a href="https://nips.cc/Conferences/2018/Schedule?showEvent=10910" rel="noreferrer">fundamental to the development of <em>artificial general intelligence</em></a>.</p>
  <p>There are some people that work on <em>continual lifelong learning</em> in neural networks, which attempts to adapt neural networks to continual lifelong learning, which is the ability of a model to learn from a stream of data continually, so that they do not completely forget previously acquired knowledge while learning new information. See, for example, the paper <a href="https://www.sciencedirect.com/science/article/pii/S0893608019300231" rel="noreferrer">Continual lifelong learning with neural networks: A review</a> (2019), by German I. Parisi et al., which summarises the problems and existing solutions related to catastrophic forgetting of neural networks.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-06T01:06:46.640Z
Answers.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/105234/discussion-on-answer-by-nbro-are-neural-networks-prone-to-catastrophic-forgettin).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-10T10:14:52.953Z
Answers.LastActivityDate: 2020-06-27T00:59:38.967Z
Answers.LastEditDate: 2020-06-27T00:59:38.967Z
Answers.UserDisplayName: nbro
Body: <p>Imagine you show a neural network a picture of a lion 100 times and label it with &quot;dangerous&quot;, so it learns that lions are dangerous.</p>
  <p>Now imagine that previously you have shown it millions of images of lions and alternatively labeled it as &quot;dangerous&quot; and &quot;not dangerous&quot;, such that the probability of a lion being dangerous is 50%.</p>
  <p>But those last 100 times have pushed the neural network into being very positive about regarding the lion as &quot;dangerous&quot;, thus ignoring the last million lessons.</p>
  <p>Therefore, it seems there is a flaw in neural networks, in that they can change their mind too quickly based on recent evidence. Especially if that previous evidence was in the middle.</p>
  <p>Is there a neural network model that keeps track of how much evidence it has seen? (Or would this be equivalent to letting the learning rate decrease by <span class="math-container">$1/T$</span> where <span class="math-container">$T$</span> is the number of trials?)</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-06T01:08:06.657Z
Comments.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/105235/discussion-on-question-by-zooby-are-neural-networks-prone-to-catastrophic-forget).
Comments.UserDisplayName: nbro
Tags: <neural-networks><machine-learning><incremental-learning><catastrophic-forgetting>
Title: Are neural networks prone to catastrophic forgetting?
UserDisplayName: zooby
ThreadText: # Are neural networks prone to catastrophic forgetting?
  
  (Asked by: username_0 on 2019-07-10)
  
  Imagine you show a neural network a picture of a lion 100 times and label it with "dangerous", so it learns that lions are dangerous.
  
  Now imagine that previously you have shown it millions of images of lions and alternatively labeled it as "dangerous" and "not dangerous", such that the probability of a lion being dangerous is 50%.
  
  But those last 100 times have pushed the neural network into being very positive about regarding the lion as "dangerous", thus ignoring the last million lessons.
  
  Therefore, it seems there is a flaw in neural networks, in that they can change their mind too quickly based on recent evidence. Especially if that previous evidence was in the middle.
  
  Is there a neural network model that keeps track of how much evidence it has seen? (Or would this be equivalent to letting the learning rate decrease by $1/T$ where $T$ is the number of trials?)
  
  **Tags:** neural-networks, machine-learning, incremental-learning, catastrophic-forgetting
  > **Comments:**
  > **username_1**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/105235/discussion-on-question-by-zooby-are-neural-networks-prone-to-catastrophic-forget).  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_1 on 2019-07-10. Score: 72 | Accepted answer
  
  Yes, indeed, neural networks are very prone to [catastrophic forgetting (or interference)](https://www.sciencedirect.com/science/article/pii/S0079742108605368?via%3Dihub). Currently, this problem is often ignored because neural networks are mainly trained offline (sometimes called *batch training*), where this problem does not often arise, and not [online or incrementally](https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2016-19.pdf), which is [fundamental to the development of *artificial general intelligence*](https://nips.cc/Conferences/2018/Schedule?showEvent=10910).
  
  There are some people that work on *continual lifelong learning* in neural networks, which attempts to adapt neural networks to continual lifelong learning, which is the ability of a model to learn from a stream of data continually, so that they do not completely forget previously acquired knowledge while learning new information. See, for example, the paper [Continual lifelong learning with neural networks: A review](https://www.sciencedirect.com/science/article/pii/S0893608019300231) (2019), by German I. Parisi et al., which summarises the problems and existing solutions related to catastrophic forgetting of neural networks.
  
  > **Comments:**
  > **username_1**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/105234/discussion-on-answer-by-nbro-are-neural-networks-prone-to-catastrophic-forgettin).  
  
  ---
  
  ### Answer by username_2 on 2019-07-10. Score: 23
  
  Yes, the problem of forgetting older training examples is a characteristic of Neural Networks. I wouldn't call it a "flaw" though because it helps them be more adaptive and allows for interesting applications such as transfer learning (if a network remembered old training too well, fine tuning it to new data would be meaningless).
  In practice what you want to do is to mix the training examples for dangerous and not dangerous so that it doesn't see one category in the beginning and one at the end.
  A standard training procedure would work like this:
  for e in epochs:
      shuffle dataset
      for x_batch, y_batch in dataset:
          train neural_network on x_batxh, y_batch
  
  Note that the shuffle at every epoch guarantees that the network won't see the same training examples in the same order every epoch and that the classes will be mixed
  Now to answer your question, yes decreasing the learning rate would make the network less prone to forgetting its previous training, but how would this work in a non-online setting? In order for a network to converge it needs multiple epochs of training (i.e. seeing each sample in the dataset many times).
  
  
  ---
  
  ### Answer by username_3 on 2019-07-11. Score: 6
  
  What you are describing sounds like it could be a deliberate case of [fine-tuning](http://wiki.fast.ai/index.php/Fine_tuning).
  
  There is a fundamental assumption that makes [minibatch gradient descent](https://adventuresinmachinelearning.com/stochastic-gradient-descent/) work for learning problems: It is assumed that any batch or temporal window of consecutive batches forms a decent approximation of the true *global* gradient of the error function with respect to any parameterization of the model. If the error surface itself is moving in a big way, that would thwart the purposes of gradient descent--since gradient descent is a local refinement algorithm, all bets are off when you suddenly change the underlying distribution.
  In the example you cited, catastrophic forgetting seems like it would be an after-effect of having "forgotten" data points previously seen, and is either a symptom of the distribution having changed, or of under-representation in the data of some important phenomenon, such that it is rarely seen relative to its importance.
  
  Experience replay from reinforcement learning is a relevant concept that transfers well to this domain. Here is [a paper](https://arxiv.org/abs/1811.11682) that explores this concept with respect to catastrophic forgetting. As long as sampling represents the true gradients sufficiently well (look at [training sample balancing](https://towardsdatascience.com/handling-imbalanced-datasets-in-deep-learning-f48407a0e758) for this) and the model has enough parameters, the catastrophic forgetting problem is unlikely to occur. In randomly shuffled datasets with replacement, it is most likely to occur where datapoints of a particular class are so rare that they are unlikely to be included for a long time during training, effectively fine-tuning the model to a different problem until a matching sample is seen again.
  
  
  ---
  
  ### Answer by username_4 on 2019-07-11. Score: 2
  
  Maybe in theory, but not in practice. The thing is you seem to consider only chronological/sequential training.
  
  And there are two ways to view this issue:
  
  1. online learning -> then it is a feature of the method
  2. offline learning -> it does not happen thanks to several order randomizations
  
  ---
  
  1. Online-Training or [Online Machine Learning](https://en.wikipedia.org/wiki/Online_machine_learning).
  
  Using the [woppal wabbit library](https://en.wikipedia.org/wiki/Vowpal_Wabbit). It is a **feature** (not an issue like you consider) of this library to adapt chronologically to the input it is fed with.
  
  I insist: it is a feature to adapt chronologically. It is wanted that when you start only telling him that lions are dangerous, then that it adapts consequently.
  
  ---
  
  2. Offline-Training
  
  In my personal experience, I have used only **randomized** subsets of my input data as training set. And this randomization is **crucial**.
  
  Randomizations happens namely:
  
  * during the training of the neural network, each epochs generally randomize the dataset order
  * during cross-validation, randomization is used as a way to evaluate a robust model that generalises well and does not overfit
  
  > **Comments:**
  > **username_5**: This does not answer the OP's question ("Are neural networks prone to catastrophic forgetting?") at all. Except perhaps for the first line: "Maybe in theory, but not in practice." - which is incorrect. Neural networks are indeed prone to catastrophic forgetting. Please back up your claims with sources.  
  > **username_4**: @MathiasMüller I probably misunderstand your comment. Please could you explain how "Online Machine Learning" and "Cross Validation" are unrelated: 1/ to my statement 2/ to the original question . Thanks for detailing your contradiction just sufficiently.  
  > **username_5**: My apologies if I there is a misunderstanding on my part, but cross validation is not a solution to catastrophic forgetting. Cross validation is a method for robust model selection. Regardless of whether cross-validation is used or not, catastrophic forgetting is not an issue in offline training settings because 1) the training set is fixed and 2) randomly shuffled so that for each epoch, examples are presented in a different order.  
  > **username_4**: I have looked at my notes from the Machine Learning course by Andrew Ng , and it is not clear from my notes that Epoching is aimed at shuffling the training set as you suggest. Do you have sources for this claim ? The definition I found is: one epoch is when the entire dataset has been fed into the backpropagation algorithm. It may very well be shuffled in modern implementations (like you suggest), I just don't have something backing this up.  
  > **username_4**: I'll edit my answer including your point about the shuffling of each epoch, if you provide me info about it.  
  > **username_5**: Yes, one epoch is defined as having presented the entire training set once to the model that is being trained. "Epoching is aimed at shuffling the training set " - I don't understand sorry!  
  > **username_5**: "Do you have sources for this claim?" yes, of course. Here is a good answer: https://stats.stackexchange.com/a/272412/228948  
  > **username_4**: @MathiasMüller Ok. Shuffling at every or each X epochs is something greatly recommended. I will revise the answer and include your point, thanks for the link.  
  > **username_4**: @MathiasMüller I have edited the answer according to your remark.  
  
  ---
  
  