Answers.Body: <p>To enlighten this, you need to understand the cause of the catastrophic forgetting. Fundamentally, the cause is an overlap in representations of different aspects of data in the learning model [<a href="https://www.aaai.org/Papers/Symposia/Spring/1993/SS-93-06/SS93-06-007.pdf" rel="nofollow noreferrer">Using Semi-Distributed Representations to Overcome
  Catastrophic Forgetting in Connectionlst Networks</a>]. This can be explained easily in neural networks based on the representations of the data in hidden layers. However, in learning models such as SGD Classifier, we cannot see any explicit representation for input data, and we merely have a decision function to classify it. Hence, we cannot see any serious discussion about catastrophic forgetting in machine learning literature around other learning methods than neural networks.</p>
  <p>By the way, this forgetting can happen in another form of the problem called &quot;concept drift&quot; [<a href="https://arxiv.org/pdf/1704.00362.pdf" rel="nofollow noreferrer">Understanding Concept Drifts</a>]. If the data distribution will change during training a model (any learning model such as SGD Classifier), the model is likely to forget the first part of the data as it is fitted to the new distribution. The cause of such an issue is the capacity of the model for training different distributions of data. So, you can find many discussions in the literature on &quot;Concept Drift&quot; for all types of learning methods.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-12T21:29:38.340Z
Answers.Text: thank you for the reply. Just to make sure, is it that catastrophic forgetting can take place even when there is no concept drift? It would greatly help if you can illustrate an example at a very broad level for the case where catastrophic forgetting can take place but there is no concept drift.
Answers.UserDisplayName: user3631804
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-12T16:39:00.450Z
Answers.LastActivityDate: 2021-08-12T16:45:20.793Z
Answers.LastEditDate: 2021-08-12T16:45:20.793Z
Answers.UserDisplayName: OmG
Body: <p>All the literature I read seems to indicate catastrophic forgetting affects only neural networks. Do other online/incremental algorithms not suffer from catastrophic forgetting (for example, <a href="https://scikit-learn.org/0.15/modules/generated/sklearn.linear_model.SGDClassifier.html#sklearn.linear_model.SGDClassifier" rel="nofollow noreferrer">SGDClassifier</a>)? Why would that be the case?</p>
  
Tags: <machine-learning><incremental-learning><online-learning><catastrophic-forgetting>
Title: Do other online/incremental algorithms not suffer from catastrophic forgetting?
UserDisplayName: user3631804
ThreadText: # Do other online/incremental algorithms not suffer from catastrophic forgetting?
  
  (Asked by: username_0 on 2021-08-12)
  
  All the literature I read seems to indicate catastrophic forgetting affects only neural networks. Do other online/incremental algorithms not suffer from catastrophic forgetting (for example, [SGDClassifier](https://scikit-learn.org/0.15/modules/generated/sklearn.linear_model.SGDClassifier.html#sklearn.linear_model.SGDClassifier))? Why would that be the case?
  
  **Tags:** machine-learning, incremental-learning, online-learning, catastrophic-forgetting
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-08-12. Score: 0 | Accepted answer
  
  To enlighten this, you need to understand the cause of the catastrophic forgetting. Fundamentally, the cause is an overlap in representations of different aspects of data in the learning model [[Using Semi-Distributed Representations to Overcome
  Catastrophic Forgetting in Connectionlst Networks](https://www.aaai.org/Papers/Symposia/Spring/1993/SS-93-06/SS93-06-007.pdf)]. This can be explained easily in neural networks based on the representations of the data in hidden layers. However, in learning models such as SGD Classifier, we cannot see any explicit representation for input data, and we merely have a decision function to classify it. Hence, we cannot see any serious discussion about catastrophic forgetting in machine learning literature around other learning methods than neural networks.
  
  By the way, this forgetting can happen in another form of the problem called "concept drift" [[Understanding Concept Drifts](https://arxiv.org/pdf/1704.00362.pdf)]. If the data distribution will change during training a model (any learning model such as SGD Classifier), the model is likely to forget the first part of the data as it is fitted to the new distribution. The cause of such an issue is the capacity of the model for training different distributions of data. So, you can find many discussions in the literature on "Concept Drift" for all types of learning methods.
  
  > **Comments:**
  > **username_0**: thank you for the reply. Just to make sure, is it that catastrophic forgetting can take place even when there is no concept drift? It would greatly help if you can illustrate an example at a very broad level for the case where catastrophic forgetting can take place but there is no concept drift.  
  
  ---
  
  