Answers.Body: <p>There are several ways to add new classes to the trained model, which require just training for the new classes.</p>
  <ul>
  <li>Incremental training (<a href="https://github.com/khurramjaved96/incremental-learning" rel="nofollow noreferrer">GitHub</a>)</li>
  <li>continuously learn a stream of data (<a href="https://github.com/creme-ml/creme" rel="nofollow noreferrer">GitHub</a>)</li>
  <li>online machine learning (<a href="https://github.com/GMvandeVen/continual-learning" rel="nofollow noreferrer">GitHub</a>)</li>
  <li>Transfer Learning Twice</li>
  <li>Continual learning approaches (Regularization, Expansion, Rehearsal) (<a href="https://github.com/facebookresearch/Adversarial-Continual-Learning" rel="nofollow noreferrer">GitHub</a>)</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-10T10:00:39.813Z
Answers.Text: What do you mean by "transfer learning twice"? Apart from that, your suggestions are more or less appropriate.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-17T18:35:02.587Z
Answers.Text: used the previous datasets of training and add new class to them and re train the model.
Answers.UserDisplayName: Farshid PirahanSiah
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-17T18:37:52.350Z
Answers.Text: Given that you provided links for all approaches other than "transfer learning twice", you may also add a link to some implementation/paper that shows that approach.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-19T09:19:26.043Z
Answers.LastActivityDate: 2020-11-17T18:37:19.453Z
Answers.LastEditDate: 2020-11-17T18:37:19.453Z
Answers.UserDisplayName: Farshid PirahanSiah
Answers.Body: <p>Here is one way you could do that.</p>
  
  <p>After training your network, you can save its weights to disk. This allows you to load this weights when new data becomes available and continue training pretty much from where your last training left off. However, since this new data might come with additional classes, you now do <a href="https://stats.stackexchange.com/q/193082/82135">pre-training or fine-tuning</a> on the network with weights previously saved. The only thing you have to do, at this point, is make the last layer(s) accommodate the new classes that have now been introduced with the arrival of your new dataset, most importantly include the extra classes (e.g., if your last layer initially had 10 classes, and now you have found 2 more classes, as part of your pre-training/fine-tuning, you replace it with 12 classes). In short, repeat this circle : </p>
  
  <p><a href="https://i.sstatic.net/8HHw7.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/8HHw7.png" alt="Repeat"></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-20T11:55:14.933Z
Answers.Text: if you only accommodate the new classes in the last layer ( training classes + new classes) the model cannot be fit because we want to train with the new classes (only) and the model expect an array with the shape of (training + the new classes, ).
Answers.UserDisplayName: Joel Carneiro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-09-06T13:59:03.557Z
Answers.LastActivityDate: 2019-05-02T15:50:34.873Z
Answers.LastEditDate: 2019-05-02T15:50:34.873Z
Answers.UserDisplayName: Tshilidzi Mudau
Answers.Body: <p>You <em>could</em> use <em>transfer learning</em> (i.e. use a pre-trained model, then change its last layer to accommodate the new classes, and re-train this slightly modified model, maybe with a lower learning rate) to achieve that, but transfer learning does <strong>not</strong> necessarily attempt to retain any of the previously acquired information (especially if you don't use very small learning rates, you keep on training and you do not freeze the weights of the convolutional layers), but only to speed up training or when your new dataset is not big enough, by starting from a model that has already learned general features that are supposedly similar to the features needed for your specific task. There is also the related <em>domain adaptation</em> problem.</p>
  <p>There are more suitable approaches to perform <strong>incremental class learning</strong> (which is what you are asking for!), which directly address the <a href="https://ai.stackexchange.com/a/13293/2444"><strong>catastrophic forgetting</strong> problem</a>. For instance, you can take a look at this paper <a href="https://arxiv.org/pdf/1903.07864.pdf" rel="nofollow noreferrer">Class-incremental Learning via Deep Model Consolidation</a>, which proposes the <strong>Deep Model Consolidation (DMC)</strong> approach. There are other <strong>continual/incremental learning</strong> approaches, many of them are described <a href="https://ai.stackexchange.com/a/24529/2444">here</a> or in more detail <a href="https://reader.elsevier.com/reader/sd/pii/S0893608019300231" rel="nofollow noreferrer">here</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-10T10:16:01.320Z
Answers.LastActivityDate: 2020-11-10T12:25:16.283Z
Answers.LastEditDate: 2020-11-10T12:25:16.283Z
Answers.UserDisplayName: nbro
Answers.Body: <p>What you are after is called &quot;Class-incremental learning&quot; (IL).</p>
  <p>In <a href="https://arxiv.org/pdf/2010.15277.pdf" rel="nofollow noreferrer">this</a> study they consider three classes of solutions:</p>
  <ul>
  <li>regularization-based solutions that aim to
  minimize the impact of learning new tasks on the weights
  that are important for previous tasks;</li>
  <li>exemplar-based solutions
  that store a limited set of exemplars to prevent forgetting of
  previous tasks;</li>
  <li>solutions that directly address the problem
  of the bias towards recently-learned tasks.</li>
  </ul>
  <p>Their main findings are:</p>
  <ul>
  <li>For exemplar-free class-IL, data regularization methods outperform weight regularization methods.</li>
  <li>Finetuning with exemplars (FT-E) yields a good baseline that
  outperforms more complex methods on several experimental
  settings.</li>
  <li>Weight regularization combines better with exemplars than data
  regularization for some scenarios.</li>
  <li>Methods that explicitly address task-recency bias outperform those that
  do not.</li>
  <li>Network architecture greatly influences the performance of class-IL
  methods, in particular the presence or absence of skip connections
  has a significant impact.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-09T14:49:30.143Z
Answers.LastActivityDate: 2021-12-09T14:49:30.143Z
Answers.UserDisplayName: Rexcirus
Answers.Body: <p>I'd like to add to what's been said already that your question touches upon an important notion in machine learning called <em>transfer learning</em>. In practice, very few people train an entire convolutional network from scratch (with random initialization), because it is time consuming and relatively rare to have a dataset of sufficient size. </p>
  
  <p>Modern ConvNets take 2-3 weeks to train across multiple GPUs on ImageNet. So it is common to see people release their final ConvNet checkpoints for the benefit of others who can use the networks for fine-tuning. For example, the Caffe library has a <a href="https://github.com/BVLC/caffe/wiki/Model-Zoo" rel="noreferrer">Model Zoo</a> where people share their network weights.</p>
  
  <p>When you need a ConvNet for image recognition, no matter what your application domain is, you should consider taking an existing network, for example <a href="http://www.robots.ox.ac.uk/~vgg/research/very_deep/" rel="noreferrer">VGGNet</a> is a common choice.</p>
  
  <p>There are a few things to keep in mind when performing <em>transfer learning</em>:</p>
  
  <ul>
  <li><p>Constraints from pretrained models. Note that if you wish to use a pretrained network, you may be slightly constrained in terms of the architecture you can use for your new dataset. For example, you can’t arbitrarily take out Conv layers from the pretrained network. However, some changes are straight-forward: due to parameter sharing, you can easily run a pretrained network on images of different spatial size. This is clearly evident in the case of Conv/Pool layers because their forward function is independent of the input volume spatial size (as long as the strides “fit”).</p></li>
  <li><p>Learning rates. It’s common to use a smaller learning rate for ConvNet weights that are being fine-tuned, in comparison to the (randomly-initialized) weights for the new linear classifier that computes the class scores of your new dataset. This is because we expect that the ConvNet weights are relatively good, so we don’t wish to distort them too quickly and too much (especially while the new Linear Classifier above them is being trained from random initialization).</p></li>
  </ul>
  
  <p>Additional reference if you are interested in this topic: <a href="https://arxiv.org/abs/1411.1792" rel="noreferrer">How transferable are features in deep neural networks?</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-02T15:44:57.027Z
Answers.Text: Transfer learning does **not** necessarily address the _catastrophic forgetting_ problem, though with a small learning rate (and by freezing some weights) you may not forget previously learned knowledge. Transfer learning is good for speeding up training or when you have a small number of samples. See [my answer](https://ai.stackexchange.com/a/24527/2444). Your answer can be misleading, and people continue to upvote it (probably because "transfer learning" is just popular, when it does not emphasize certain important aspects).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-09-18T16:37:27.630Z
Answers.LastActivityDate: 2017-09-18T16:37:27.630Z
Answers.UserDisplayName: Maxim
Body: <p>I would like to train a neural network (NN) where the output classes are not (all) defined from the start. More and more classes will be introduced later based on incoming data. This means that, every time I introduce a new class, I would need to retrain the NN.</p>
  <p>How can I train an NN incrementally, that is, without forgetting the previously acquired information during the previous training phases?</p>
  
Tags: <neural-networks><deep-learning><transfer-learning><incremental-learning><catastrophic-forgetting>
Title: Is it possible to train a neural network as new classes are given?
UserDisplayName: Th3Nic3Guy
ThreadText: # Is it possible to train a neural network as new classes are given?
  
  (Asked by: username_0 on 2017-09-06)
  
  I would like to train a neural network (NN) where the output classes are not (all) defined from the start. More and more classes will be introduced later based on incoming data. This means that, every time I introduce a new class, I would need to retrain the NN.
  
  How can I train an NN incrementally, that is, without forgetting the previously acquired information during the previous training phases?
  
  **Tags:** neural-networks, deep-learning, transfer-learning, incremental-learning, catastrophic-forgetting
  
  ---
  
  ## 5 answers
  
  ### Answer by username_6 on 2017-09-18. Score: 14 | Accepted answer
  
  I'd like to add to what's been said already that your question touches upon an important notion in machine learning called *transfer learning*. In practice, very few people train an entire convolutional network from scratch (with random initialization), because it is time consuming and relatively rare to have a dataset of sufficient size.
  
  Modern ConvNets take 2-3 weeks to train across multiple GPUs on ImageNet. So it is common to see people release their final ConvNet checkpoints for the benefit of others who can use the networks for fine-tuning. For example, the Caffe library has a [Model Zoo](https://github.com/BVLC/caffe/wiki/Model-Zoo) where people share their network weights.
  
  When you need a ConvNet for image recognition, no matter what your application domain is, you should consider taking an existing network, for example [VGGNet](http://www.robots.ox.ac.uk/~vgg/research/very_deep/) is a common choice.
  
  There are a few things to keep in mind when performing *transfer learning*:
  
  * Constraints from pretrained models. Note that if you wish to use a pretrained network, you may be slightly constrained in terms of the architecture you can use for your new dataset. For example, you can’t arbitrarily take out Conv layers from the pretrained network. However, some changes are straight-forward: due to parameter sharing, you can easily run a pretrained network on images of different spatial size. This is clearly evident in the case of Conv/Pool layers because their forward function is independent of the input volume spatial size (as long as the strides “fit”).
  * Learning rates. It’s common to use a smaller learning rate for ConvNet weights that are being fine-tuned, in comparison to the (randomly-initialized) weights for the new linear classifier that computes the class scores of your new dataset. This is because we expect that the ConvNet weights are relatively good, so we don’t wish to distort them too quickly and too much (especially while the new Linear Classifier above them is being trained from random initialization).
  
  Additional reference if you are interested in this topic: [How transferable are features in deep neural networks?](https://arxiv.org/abs/1411.1792)
  
  > **Comments:**
  > **username_2**: Transfer learning does **not** necessarily address the _catastrophic forgetting_ problem, though with a small learning rate (and by freezing some weights) you may not forget previously learned knowledge. Transfer learning is good for speeding up training or when you have a small number of samples. See [my answer](https://ai.stackexchange.com/a/24527/2444). Your answer can be misleading, and people continue to upvote it (probably because "transfer learning" is just popular, when it does not emphasize certain important aspects).  
  
  ---
  
  ### Answer by username_3 on 2017-09-06. Score: 7
  
  Here is one way you could do that.
  
  After training your network, you can save its weights to disk. This allows you to load this weights when new data becomes available and continue training pretty much from where your last training left off. However, since this new data might come with additional classes, you now do [pre-training or fine-tuning](https://stats.stackexchange.com/q/193082/82135) on the network with weights previously saved. The only thing you have to do, at this point, is make the last layer(s) accommodate the new classes that have now been introduced with the arrival of your new dataset, most importantly include the extra classes (e.g., if your last layer initially had 10 classes, and now you have found 2 more classes, as part of your pre-training/fine-tuning, you replace it with 12 classes). In short, repeat this circle :
  
  [![Repeat](https://i.sstatic.net/8HHw7.png)](https://i.sstatic.net/8HHw7.png)
  
  > **Comments:**
  > **username_4**: if you only accommodate the new classes in the last layer ( training classes + new classes) the model cannot be fit because we want to train with the new classes (only) and the model expect an array with the shape of (training + the new classes, ).  
  
  ---
  
  ### Answer by username_1 on 2020-10-19. Score: 3
  
  There are several ways to add new classes to the trained model, which require just training for the new classes.
  
  * Incremental training ([GitHub](https://github.com/khurramjaved96/incremental-learning))
  * continuously learn a stream of data ([GitHub](https://github.com/creme-ml/creme))
  * online machine learning ([GitHub](https://github.com/GMvandeVen/continual-learning))
  * Transfer Learning Twice
  * Continual learning approaches (Regularization, Expansion, Rehearsal) ([GitHub](https://github.com/facebookresearch/Adversarial-Continual-Learning))
  
  > **Comments:**
  > **username_2**: What do you mean by "transfer learning twice"? Apart from that, your suggestions are more or less appropriate.  
  > **username_1**: used the previous datasets of training and add new class to them and re train the model.  
  > **username_2**: Given that you provided links for all approaches other than "transfer learning twice", you may also add a link to some implementation/paper that shows that approach.  
  
  ---
  
  ### Answer by username_2 on 2020-11-10. Score: 1
  
  You *could* use *transfer learning* (i.e. use a pre-trained model, then change its last layer to accommodate the new classes, and re-train this slightly modified model, maybe with a lower learning rate) to achieve that, but transfer learning does **not** necessarily attempt to retain any of the previously acquired information (especially if you don't use very small learning rates, you keep on training and you do not freeze the weights of the convolutional layers), but only to speed up training or when your new dataset is not big enough, by starting from a model that has already learned general features that are supposedly similar to the features needed for your specific task. There is also the related *domain adaptation* problem.
  
  There are more suitable approaches to perform **incremental class learning** (which is what you are asking for!), which directly address the [**catastrophic forgetting** problem](https://ai.stackexchange.com/a/13293/2444). For instance, you can take a look at this paper [Class-incremental Learning via Deep Model Consolidation](https://arxiv.org/pdf/1903.07864.pdf), which proposes the **Deep Model Consolidation (DMC)** approach. There are other **continual/incremental learning** approaches, many of them are described [here](https://ai.stackexchange.com/a/24529/2444) or in more detail [here](https://reader.elsevier.com/reader/sd/pii/S0893608019300231).
  
  
  ---
  
  ### Answer by username_5 on 2021-12-09. Score: 0
  
  What you are after is called "Class-incremental learning" (IL).
  
  In [this](https://arxiv.org/pdf/2010.15277.pdf) study they consider three classes of solutions:
  
  * regularization-based solutions that aim to
    minimize the impact of learning new tasks on the weights
    that are important for previous tasks;
  * exemplar-based solutions
    that store a limited set of exemplars to prevent forgetting of
    previous tasks;
  * solutions that directly address the problem
    of the bias towards recently-learned tasks.
  
  Their main findings are:
  
  * For exemplar-free class-IL, data regularization methods outperform weight regularization methods.
  * Finetuning with exemplars (FT-E) yields a good baseline that
    outperforms more complex methods on several experimental
    settings.
  * Weight regularization combines better with exemplars than data
    regularization for some scenarios.
  * Methods that explicitly address task-recency bias outperform those that
    do not.
  * Network architecture greatly influences the performance of class-IL
    methods, in particular the presence or absence of skip connections
    has a significant impact.
  
  
  ---
  
  