Answers.Body: <blockquote>
  <p>In general, is continuous learning possible with a deep convolutional neural network, without changing its topology?</p>
  </blockquote>
  <p>Your intuition that it is possible to perform <em>incremental</em> (aka <em>continual</em>, <em>continuous</em> or <em>lifelong</em>) <em>learning</em> by changing the NN's topology is correct. However, <strong>dynamically adapting the NN's topology</strong> is <em>just</em> one approach to continual learning (a specific example of this approach is <a href="https://arxiv.org/pdf/1708.01547.pdf" rel="nofollow noreferrer">DEN</a>). So, there are other approaches, such as</p>
  <ul>
  <li><p><strong>regularization approaches</strong> (examples are <a href="https://arxiv.org/pdf/1606.09282.pdf" rel="nofollow noreferrer">LwF</a>, <a href="https://arxiv.org/pdf/1612.00796.pdf" rel="nofollow noreferrer">EWC</a> and <a href="https://arxiv.org/pdf/1703.04200.pdf" rel="nofollow noreferrer">SI</a>)</p>
  </li>
  <li><p><strong>ensemble approaches</strong> (examples are <a href="https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.16.1297&amp;rep=rep1&amp;type=pdf" rel="nofollow noreferrer">Learn++</a>, <a href="https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.409.2942&amp;rep=rep1&amp;type=pdf" rel="nofollow noreferrer">FEL</a> and <a href="https://arxiv.org/pdf/1701.08734.pdf" rel="nofollow noreferrer">PathNet</a>)</p>
  </li>
  <li><p><strong><a href="https://ai.stackexchange.com/a/23832/2444">rehearsal</a> approaches</strong> (an example is <a href="https://arxiv.org/pdf/1611.07725.pdf" rel="nofollow noreferrer">iCaRL</a>)</p>
  </li>
  </ul>
  <p>For more details about these and other approaches (and problems related to continual learning and catastrophic forgetting in neural networks), take a look at <a href="https://www.sciencedirect.com/science/article/pii/S0893608019300231" rel="nofollow noreferrer">this very nice review of continual learning approaches in neural networks</a>. You should also check <a href="https://ai.stackexchange.com/a/13293/2444">this answer</a>.</p>
  <blockquote>
  <p>Are there ways to implement continuous learning in a deep neural network for image recognition?</p>
  </blockquote>
  <p>Yes. Many of the approaches focus on image recognition and classification, and often the experiments are performed on MNIST or similar datasets (e.g. see <a href="https://arxiv.org/pdf/1902.06494.pdf" rel="nofollow noreferrer">this paper</a>).</p>
  <blockquote>
  <p>Does such an implementation make sense if the labels have to be specially prepared in advance?</p>
  </blockquote>
  <p>Yes, you can prepare your dataset in advance, and then later train incrementally (in fact, in the experiments I have seen in some of these papers, they usually do this to simulate the continual learning scenario), but I am not sure about the optimality of this approach. Maybe with batch learning (i.e. the usual offline learning where you train on all data), you would achieve higher performance.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-10T11:11:51.543Z
Answers.LastActivityDate: 2020-11-10T12:48:38.913Z
Answers.LastEditDate: 2020-11-10T12:48:38.913Z
Answers.UserDisplayName: nbro
Body: <p>In general, is continuous learning possible with a deep convolutional neural network, without changing its topology?</p>
  <p>In my case, I want to use a convolutional neural network as a classifier of heartbeat types. The ECG signal is split, and a color image is created using feature extraction. These photos (the inputs) are fed into a deep CNN, but they must be labeled by someone first.</p>
  <p>Are there ways to implement continuous learning in a deep neural network for image recognition? Does such an implementation make sense if the labels have to be specially prepared in advance?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-14T12:29:23.380Z
Comments.Text: Your question is very similar to [this](https://ai.stackexchange.com/q/14047/2444) (which is asking about the state-of-the-art approaches, which is not your question), [this](https://ai.stackexchange.com/q/13194/2444) (which does not seem to focus on CNNs and image recognition), [this](https://ai.stackexchange.com/q/3981/2444) (which focuses on the case where new classes are incrementally given) or [this](https://ai.stackexchange.com/q/12619/2444) (which asks if a CNN can be trained incrementally, without putting any restriction on the topology) questions.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-10T11:32:30.860Z
Comments.Text: Here, you are specifically asking about "incremental/continual learning for image recognition without changing the NN's topology", so I will not close this post as a duplicate of those.
Comments.UserDisplayName: nbro
Tags: <deep-learning><convolutional-neural-networks><image-recognition><incremental-learning><online-learning>
Title: Is continuous learning possible with a deep convolutional neural network, without changing its topology?
UserDisplayName: Dominiksr
ThreadText: # Is continuous learning possible with a deep convolutional neural network, without changing its topology?
  
  (Asked by: username_0 on 2020-09-13)
  
  In general, is continuous learning possible with a deep convolutional neural network, without changing its topology?
  
  In my case, I want to use a convolutional neural network as a classifier of heartbeat types. The ECG signal is split, and a color image is created using feature extraction. These photos (the inputs) are fed into a deep CNN, but they must be labeled by someone first.
  
  Are there ways to implement continuous learning in a deep neural network for image recognition? Does such an implementation make sense if the labels have to be specially prepared in advance?
  
  **Tags:** deep-learning, convolutional-neural-networks, image-recognition, incremental-learning, online-learning
  
  > **Comments:**
  > **username_1**: Your question is very similar to [this](https://ai.stackexchange.com/q/14047/2444) (which is asking about the state-of-the-art approaches, which is not your question), [this](https://ai.stackexchange.com/q/13194/2444) (which does not seem to focus on CNNs and image recognition), [this](https://ai.stackexchange.com/q/3981/2444) (which focuses on the case where new classes are incrementally given) or [this](https://ai.stackexchange.com/q/12619/2444) (which asks if a CNN can be trained incrementally, without putting any restriction on the topology) questions.  
  > **username_1**: Here, you are specifically asking about "incremental/continual learning for image recognition without changing the NN's topology", so I will not close this post as a duplicate of those.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-11-10. Score: 3 | Accepted answer
  
  > In general, is continuous learning possible with a deep convolutional neural network, without changing its topology?
  
  Your intuition that it is possible to perform *incremental* (aka *continual*, *continuous* or *lifelong*) *learning* by changing the NN's topology is correct. However, **dynamically adapting the NN's topology** is *just* one approach to continual learning (a specific example of this approach is [DEN](https://arxiv.org/pdf/1708.01547.pdf)). So, there are other approaches, such as
  
  * **regularization approaches** (examples are [LwF](https://arxiv.org/pdf/1606.09282.pdf), [EWC](https://arxiv.org/pdf/1612.00796.pdf) and [SI](https://arxiv.org/pdf/1703.04200.pdf))
  * **ensemble approaches** (examples are [Learn++](https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.16.1297&rep=rep1&type=pdf), [FEL](https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.409.2942&rep=rep1&type=pdf) and [PathNet](https://arxiv.org/pdf/1701.08734.pdf))
  * **[rehearsal](https://ai.stackexchange.com/a/23832/2444) approaches** (an example is [iCaRL](https://arxiv.org/pdf/1611.07725.pdf))
  
  For more details about these and other approaches (and problems related to continual learning and catastrophic forgetting in neural networks), take a look at [this very nice review of continual learning approaches in neural networks](https://www.sciencedirect.com/science/article/pii/S0893608019300231). You should also check [this answer](https://ai.stackexchange.com/a/13293/2444).
  
  > Are there ways to implement continuous learning in a deep neural network for image recognition?
  
  Yes. Many of the approaches focus on image recognition and classification, and often the experiments are performed on MNIST or similar datasets (e.g. see [this paper](https://arxiv.org/pdf/1902.06494.pdf)).
  
  > Does such an implementation make sense if the labels have to be specially prepared in advance?
  
  Yes, you can prepare your dataset in advance, and then later train incrementally (in fact, in the experiments I have seen in some of these papers, they usually do this to simulate the continual learning scenario), but I am not sure about the optimality of this approach. Maybe with batch learning (i.e. the usual offline learning where you train on all data), you would achieve higher performance.
  
  
  ---
  
  