Answers.Body: <blockquote>
  <p>Do you know which are the state-of-the-art approaches on this topic, and could you point me to some literature on them?</p>
  </blockquote>
  <p><a href="https://ai.stackexchange.com/a/14306/2444">This answer</a> already mentions some of the approaches. More concretely, currently, the most common approaches to continual learning (i.e. learning with progressively more data while attempting to address the <a href="https://ai.stackexchange.com/a/13293/2444"><em>catastrophic forgetting</em> problem</a>) are</p>
  <ul>
  <li><strong>dynamic/changing topologies</strong> approaches</li>
  <li><strong>regularization</strong> approaches</li>
  <li><a href="https://ai.stackexchange.com/a/23832/2444"><strong>rehearsal</strong></a> (or <strong>pseudo-rehearsal</strong>) approaches</li>
  <li><strong>ensemble</strong> approaches</li>
  <li><strong>hybrid</strong> approaches</li>
  </ul>
  <p>You can also take a look at <a href="https://ai.stackexchange.com/a/24529/2444">this answer</a>. If you are interested in an exhaustive overview of the <em>state-of-the-art</em> (at least, until 2019), you should read the paper <a href="https://www.sciencedirect.com/science/article/pii/S0893608019300231" rel="nofollow noreferrer">Continual lifelong learning with neural networks: A review</a> (2019, by Parisi et al.).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-10T11:45:40.920Z
Answers.LastActivityDate: 2020-11-10T11:53:07.663Z
Answers.LastEditDate: 2020-11-10T11:53:07.663Z
Answers.UserDisplayName: nbro
Answers.Body: <p>There are lots of different approaches that try to avoid catastrophic forgetting in neural networks. It is impossible to summarize all contributions here.</p>
  <p>However, in addition to the already mentioned techniques, there are <strong>sparsity</strong> approaches that try to disentangle internal representations of the network on different tasks or learning steps. Sparsity usually helps, but the network has to learn to use it, imposing a structural sparsity by construction is not enough. Also, you can leverage <strong>bayesian</strong> approaches, through which you can associate a confidence measure to each of your weights and use this measure to mitigate forgetting. Also, <strong>meta-learning</strong> can be employed to meta-learn a model which is robust to forgetting on different sequences of tasks.</p>
  <p>What I can suggest you in addition, is to take a look at <a href="https://wiki.continualai.org/research.html" rel="nofollow noreferrer">ContinualAI wiki</a>, which maintains a list of updated publications classified by the type of Continual Learning strategy and tagged with additional information. (<strong>Disclaimer</strong>: I am a member of <a href="https://www.continualai.org/" rel="nofollow noreferrer">ContinualAI association</a>).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-12T12:07:02.447Z
Answers.LastActivityDate: 2020-12-12T12:02:12.357Z
Answers.LastEditDate: 2020-12-12T12:02:12.357Z
Answers.UserDisplayName: andcos
Answers.Body: <p>What I understand from your questions is that you are trying to avoid catastrophic forgetting while applying online learning. </p>
  
  <p>This problem should be addressed by implementing methods that reduce catastrophic forgetting for different tasks. At first glance it might seem that they don't apply because it's data that change and not a particular task but changing data result in a change of the task. Say your goal is to classify different breeds of dogs. Your online data-set morphs into excluding "Great Danes". Your neural network after enough epochs would forget about "Great Danes". The task is still serving its purpose by classifying different breeds but the task still changed. It changed from recognizing "Great Danes" as a dog breed to not recognizing "Great Danes" as a dog breed. The weights changed to exclude them but the methods I linked tries and keep weights intact even though it was not intended for the purpose of online learning. Just set the hyper parameters to include these techniques to low as I believe data won't have an instant change but would change over time, and you should be fine. </p>
  
  <p>The most obvious technique being storing information as you train. This is called <a href="https://www.semanticscholar.org/paper/A-Strategy-for-an-Uncompromising-Incremental-Venkatesan-Venkateswara/ac7ea1a3050fe81a565b22b4bafdd5baa465be07" rel="nofollow noreferrer">pseudo-rehearsal</a>. With this at least you would be able to use stochastic gradient decent but you need memory and resources as the data set grows. </p>
  
  <p>Then there was an attempt to reduce impacts of weights on old tasks to keep some relevancy to them. <a href="https://www.pnas.org/content/pnas/114/13/3521.full.pdf" rel="nofollow noreferrer">Structural Regularization</a>. </p>
  
  <p>Later these guys implemented <a href="https://arxiv.org/pdf/1801.01423.pdf" rel="nofollow noreferrer">HAT</a> which seems to keep some weights static while others adapt to new tasks. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-06T09:43:08.030Z
Answers.LastActivityDate: 2019-09-06T09:43:08.030Z
Answers.UserDisplayName: SandMan
Body: <p>There seems to be a lot of literature and research on the problems of stochastic gradient descent and catastrophic forgetting, but I can't find much on solutions to perform continual learning with neural network architectures.</p>
  <p>By continual learning, I mean improving a model (while using it) with a stream of data coming in (maybe after a partial initial training with ordinary batches and epochs).</p>
  <p>A lot of real-world distributions are likely to gradually change with time, so I believe that we should be able to train NNs in an online fashion.</p>
  <p>Do you know which are the state-of-the-art approaches on this topic, and could you point me to some literature on them?</p>
  
Tags: <neural-networks><reference-request><incremental-learning><online-learning><catastrophic-forgetting>
Title: What are the state-of-the-art approaches for continual learning with neural networks?
UserDisplayName: gcorso
ThreadText: # What are the state-of-the-art approaches for continual learning with neural networks?
  
  (Asked by: username_0 on 2019-08-19)
  
  There seems to be a lot of literature and research on the problems of stochastic gradient descent and catastrophic forgetting, but I can't find much on solutions to perform continual learning with neural network architectures.
  
  By continual learning, I mean improving a model (while using it) with a stream of data coming in (maybe after a partial initial training with ordinary batches and epochs).
  
  A lot of real-world distributions are likely to gradually change with time, so I believe that we should be able to train NNs in an online fashion.
  
  Do you know which are the state-of-the-art approaches on this topic, and could you point me to some literature on them?
  
  **Tags:** neural-networks, reference-request, incremental-learning, online-learning, catastrophic-forgetting
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2019-09-06. Score: 4 | Accepted answer
  
  What I understand from your questions is that you are trying to avoid catastrophic forgetting while applying online learning.
  
  This problem should be addressed by implementing methods that reduce catastrophic forgetting for different tasks. At first glance it might seem that they don't apply because it's data that change and not a particular task but changing data result in a change of the task. Say your goal is to classify different breeds of dogs. Your online data-set morphs into excluding "Great Danes". Your neural network after enough epochs would forget about "Great Danes". The task is still serving its purpose by classifying different breeds but the task still changed. It changed from recognizing "Great Danes" as a dog breed to not recognizing "Great Danes" as a dog breed. The weights changed to exclude them but the methods I linked tries and keep weights intact even though it was not intended for the purpose of online learning. Just set the hyper parameters to include these techniques to low as I believe data won't have an instant change but would change over time, and you should be fine.
  
  The most obvious technique being storing information as you train. This is called [pseudo-rehearsal](https://www.semanticscholar.org/paper/A-Strategy-for-an-Uncompromising-Incremental-Venkatesan-Venkateswara/ac7ea1a3050fe81a565b22b4bafdd5baa465be07). With this at least you would be able to use stochastic gradient decent but you need memory and resources as the data set grows.
  
  Then there was an attempt to reduce impacts of weights on old tasks to keep some relevancy to them. [Structural Regularization](https://www.pnas.org/content/pnas/114/13/3521.full.pdf).
  
  Later these guys implemented [HAT](https://arxiv.org/pdf/1801.01423.pdf) which seems to keep some weights static while others adapt to new tasks.
  
  
  ---
  
  ### Answer by username_1 on 2020-11-10. Score: 3
  
  > Do you know which are the state-of-the-art approaches on this topic, and could you point me to some literature on them?
  
  [This answer](https://ai.stackexchange.com/a/14306/2444) already mentions some of the approaches. More concretely, currently, the most common approaches to continual learning (i.e. learning with progressively more data while attempting to address the [*catastrophic forgetting* problem](https://ai.stackexchange.com/a/13293/2444)) are
  
  * **dynamic/changing topologies** approaches
  * **regularization** approaches
  * [**rehearsal**](https://ai.stackexchange.com/a/23832/2444) (or **pseudo-rehearsal**) approaches
  * **ensemble** approaches
  * **hybrid** approaches
  
  You can also take a look at [this answer](https://ai.stackexchange.com/a/24529/2444). If you are interested in an exhaustive overview of the *state-of-the-art* (at least, until 2019), you should read the paper [Continual lifelong learning with neural networks: A review](https://www.sciencedirect.com/science/article/pii/S0893608019300231) (2019, by Parisi et al.).
  
  
  ---
  
  ### Answer by username_2 on 2020-11-12. Score: 2
  
  There are lots of different approaches that try to avoid catastrophic forgetting in neural networks. It is impossible to summarize all contributions here.
  
  However, in addition to the already mentioned techniques, there are **sparsity** approaches that try to disentangle internal representations of the network on different tasks or learning steps. Sparsity usually helps, but the network has to learn to use it, imposing a structural sparsity by construction is not enough. Also, you can leverage **bayesian** approaches, through which you can associate a confidence measure to each of your weights and use this measure to mitigate forgetting. Also, **meta-learning** can be employed to meta-learn a model which is robust to forgetting on different sequences of tasks.
  
  What I can suggest you in addition, is to take a look at [ContinualAI wiki](https://wiki.continualai.org/research.html), which maintains a list of updated publications classified by the type of Continual Learning strategy and tagged with additional information. (**Disclaimer**: I am a member of [ContinualAI association](https://www.continualai.org/)).
  
  
  ---
  
  