Answers.Body: <p>Learning without Forgetting (LwF) is an incremental learning (sometimes also called continual or lifelong learning) technique for neural networks, which is a machine learning technique that attempts to avoid <a href="https://ai.stackexchange.com/a/13293/2444">catastrophic forgetting</a>. There are several incremental learning approaches. LwF is an incremental learning approach based on the concept of <a href="https://arxiv.org/pdf/1710.10686.pdf" rel="noreferrer"><em>regularization</em></a>. In section 3.2 of the paper <a href="https://www.sciencedirect.com/science/article/pii/S0893608019300231" rel="noreferrer">Continual lifelong learning with neural networks: A review</a> (2019), by Parisi et al., other regularisation-based continual learning techniques are described. </p>
  
  <p>LwF could be seen as a combination of <a href="https://arxiv.org/abs/1503.02531" rel="noreferrer">distillation networks</a> and <a href="https://arxiv.org/abs/1311.2524" rel="noreferrer">fine-tuning</a>, which refers to the re-training with a low learning rate (which is a very rudimentary technique to avoid catastrophically forgetting the previously learned knowledge) an already trained model <span class="math-container">$\mathcal{M}$</span> with new and (usually) more specific dataset, <span class="math-container">$\mathcal{D}_{\text{new}}$</span>, with respect to the dataset, <span class="math-container">$\mathcal{D}_{\text{old}}$</span>, with which you originally trained the given model <span class="math-container">$\mathcal{M}$</span>. </p>
  
  <p>LwF, as opposed to other continual learning techniques, only uses the new data, so it assumes that past data (used to pre-train the network) is unavailable. The paper <a href="https://arxiv.org/abs/1606.09282" rel="noreferrer">Learning without Forgetting</a> goes into the details of the technique and it also describes the concepts of <em>feature extraction</em>, <em>fine tuning</em> and <em>multitask learning</em>, which are related to incremental learning techniques.</p>
  
  <p><em>What is the difference between LwF and transfer learning?</em> LwF is a combination of distillation networks and fine-tuning, which is a <a href="https://www.cse.ust.hk/~qyang/Docs/2009/tkde_transfer_learning.pdf" rel="noreferrer">transfer learning</a> technique, which is a special case of incremental learning, where the old and new tasks are different, while, in general, in incremental learning, the old and new tasks can also be the same (which is called <a href="https://arxiv.org/abs/1606.09282" rel="noreferrer">domain adaptation</a>).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-10T13:07:46.593Z
Answers.Text: TODO: This answer needs to be improved because the concept of "task" is not well defined here. Saying that the old and new tasks are the same in domain adaptation can be confusing, without a definition of "task".
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-29T14:33:14.560Z
Answers.LastActivityDate: 2019-10-29T16:11:20.250Z
Answers.LastEditDate: 2019-10-29T16:11:20.250Z
Answers.UserDisplayName: nbro
Answers.Body: <blockquote>
    <p>What I want to achieve is incremental training. So, as soon as I get new data, I can further train my already trained model and I don't have to retrain everything.</p>
  </blockquote>
  
  <p>Learning without forgetting is one of the methods to solve multitask learning. If your model trained to solve problem A and then after sometimes you need your model to solve new problem B without forgetting the problem A (the model still good to solve the problem A), then you need this.</p>
  
  <p>Transfer learning is a method to use a trained model to solve another task (and may forget the original task). For example, you use a model that originally trained to classify cat or dog to a new task that trying to classify goat or cow. You use this in hopes of speeding up your training process.</p>
  
  <p>If your new data has the same task with the old data, you don't need to use multitask learning method. For example:</p>
  
  <ul>
  <li>if your model trained with 50 images to detect an apple in the image, and then you get new 100 images to detect an apple then you just need to continue your train (incremental learning). In this case, you need (to save) the latest parameter of your model after trained (latest learning rate value, epoch, etc.), if you have it then you just need to run your training again (continue the epoch).</li>
  <li>if your model trained with 100 images to detect an apple in the image, and then you get new 100 images to train your model to detect an orange and <strong>you don't care if your model will give a bad result to detect an apple</strong>, then you can use transfer learning. You may freeze a few first layers as "extractor" and initialize a new layer at the end.</li>
  <li>if your model trained with 100 images to detect an apple in the image, and then you get new 100 images to detect an orange and your model must good to detect both apple and orange in an image, then you use multitask learning. The easiest method is to train your model with the apple+orange image, but you can also use another approach like proposed in Learning without Forgetting paper.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-06T23:10:17.067Z
Answers.Text: Hi could you take a look here? https://ai.stackexchange.com/questions/27687/visualisation-for-mobilenet-feature-extraction
Answers.UserDisplayName: x89
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-02T09:26:33.913Z
Answers.LastActivityDate: 2019-08-05T04:02:21.260Z
Answers.LastEditDate: 2019-08-05T04:02:21.260Z
Answers.UserDisplayName: malioboro
Body: <p>I would like to incrementally train my model with my current dataset and <a href="https://github.com/tensorflow/models/issues/7200#issuecomment-510850230" rel="nofollow noreferrer">I asked this question on Github</a>, which is what I'm using SSD MobileNet v1.</p>
  <p>Someone there told me about <a href="https://arxiv.org/abs/1606.09282" rel="nofollow noreferrer"><strong>learning without forgetting</strong></a>. I'm now confused between <em>learning without forgetting</em> and <em>transfer learning</em>. How they differ from each other?</p>
  <p>My initial problem, what I'm trying to achieve (mentioned in Github issue) is the following.</p>
  <p>I have trained my dataset on <code>ssd_mobilenet_v1_coco</code> model. I'm getting continuous incremental data. Right now, my dataset is very limited.</p>
  <p>What I want to achieve is <em>incremental training</em>, i.e. as soon as I get new data, I can further train my already trained model and I don't have to retrain everything:</p>
  <ol>
  <li>Save trained model <span class="math-container">$M_t$</span></li>
  <li>Get new data <span class="math-container">$D_{t+1}$</span></li>
  <li>Train <span class="math-container">$M_t$</span> on <span class="math-container">$D_{t+1}$</span> to produce <span class="math-container">$M_{t+1}$</span></li>
  <li>Let <span class="math-container">$t = t+1$</span>, then go back to <span class="math-container">$1$</span></li>
  </ol>
  <p>How do I perform this incremental training/learning? Should I use LwF or transfer learning?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-06T23:09:51.013Z
Comments.Text: Hi could you take a look here? https://ai.stackexchange.com/questions/27687/visualisation-for-mobilenet-feature-extraction
Comments.UserDisplayName: x89
Tags: <comparison><transfer-learning><incremental-learning><domain-adaptation><learning-without-forgetting>
Title: What is the difference between learning without forgetting and transfer learning?
UserDisplayName: abhimanyuaryan
ThreadText: # What is the difference between learning without forgetting and transfer learning?
  
  (Asked by: username_0 on 2019-07-29)
  
  I would like to incrementally train my model with my current dataset and [I asked this question on Github](https://github.com/tensorflow/models/issues/7200#issuecomment-510850230), which is what I'm using SSD MobileNet v1.
  
  Someone there told me about [**learning without forgetting**](https://arxiv.org/abs/1606.09282). I'm now confused between *learning without forgetting* and *transfer learning*. How they differ from each other?
  
  My initial problem, what I'm trying to achieve (mentioned in Github issue) is the following.
  
  I have trained my dataset on `ssd_mobilenet_v1_coco` model. I'm getting continuous incremental data. Right now, my dataset is very limited.
  
  What I want to achieve is *incremental training*, i.e. as soon as I get new data, I can further train my already trained model and I don't have to retrain everything:
  
  1. Save trained model $M\_t$
  2. Get new data $D\_{t+1}$
  3. Train $M\_t$ on $D\_{t+1}$ to produce $M\_{t+1}$
  4. Let $t = t+1$, then go back to $1$
  
  How do I perform this incremental training/learning? Should I use LwF or transfer learning?
  
  **Tags:** comparison, transfer-learning, incremental-learning, domain-adaptation, learning-without-forgetting
  > **Comments:**
  > **username_1**: Hi could you take a look here? https://ai.stackexchange.com/questions/27687/visualisation-for-mobilenet-feature-extraction  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2019-10-29. Score: 7
  
  Learning without Forgetting (LwF) is an incremental learning (sometimes also called continual or lifelong learning) technique for neural networks, which is a machine learning technique that attempts to avoid [catastrophic forgetting](https://ai.stackexchange.com/a/13293/2444). There are several incremental learning approaches. LwF is an incremental learning approach based on the concept of [*regularization*](https://arxiv.org/pdf/1710.10686.pdf). In section 3.2 of the paper [Continual lifelong learning with neural networks: A review](https://www.sciencedirect.com/science/article/pii/S0893608019300231) (2019), by Parisi et al., other regularisation-based continual learning techniques are described.
  
  LwF could be seen as a combination of [distillation networks](https://arxiv.org/abs/1503.02531) and [fine-tuning](https://arxiv.org/abs/1311.2524), which refers to the re-training with a low learning rate (which is a very rudimentary technique to avoid catastrophically forgetting the previously learned knowledge) an already trained model $\mathcal{M}$ with new and (usually) more specific dataset, $\mathcal{D}\_{\text{new}}$, with respect to the dataset, $\mathcal{D}\_{\text{old}}$, with which you originally trained the given model $\mathcal{M}$.
  
  LwF, as opposed to other continual learning techniques, only uses the new data, so it assumes that past data (used to pre-train the network) is unavailable. The paper [Learning without Forgetting](https://arxiv.org/abs/1606.09282) goes into the details of the technique and it also describes the concepts of *feature extraction*, *fine tuning* and *multitask learning*, which are related to incremental learning techniques.
  
  *What is the difference between LwF and transfer learning?* LwF is a combination of distillation networks and fine-tuning, which is a [transfer learning](https://www.cse.ust.hk/~qyang/Docs/2009/tkde_transfer_learning.pdf) technique, which is a special case of incremental learning, where the old and new tasks are different, while, in general, in incremental learning, the old and new tasks can also be the same (which is called [domain adaptation](https://arxiv.org/abs/1606.09282)).
  
  > **Comments:**
  > **username_2**: TODO: This answer needs to be improved because the concept of "task" is not well defined here. Saying that the old and new tasks are the same in domain adaptation can be confusing, without a definition of "task".  
  
  ---
  
  ### Answer by username_3 on 2019-08-02. Score: 2
  
  > What I want to achieve is incremental training. So, as soon as I get new data, I can further train my already trained model and I don't have to retrain everything.
  
  Learning without forgetting is one of the methods to solve multitask learning. If your model trained to solve problem A and then after sometimes you need your model to solve new problem B without forgetting the problem A (the model still good to solve the problem A), then you need this.
  
  Transfer learning is a method to use a trained model to solve another task (and may forget the original task). For example, you use a model that originally trained to classify cat or dog to a new task that trying to classify goat or cow. You use this in hopes of speeding up your training process.
  
  If your new data has the same task with the old data, you don't need to use multitask learning method. For example:
  
  * if your model trained with 50 images to detect an apple in the image, and then you get new 100 images to detect an apple then you just need to continue your train (incremental learning). In this case, you need (to save) the latest parameter of your model after trained (latest learning rate value, epoch, etc.), if you have it then you just need to run your training again (continue the epoch).
  * if your model trained with 100 images to detect an apple in the image, and then you get new 100 images to train your model to detect an orange and **you don't care if your model will give a bad result to detect an apple**, then you can use transfer learning. You may freeze a few first layers as "extractor" and initialize a new layer at the end.
  * if your model trained with 100 images to detect an apple in the image, and then you get new 100 images to detect an orange and your model must good to detect both apple and orange in an image, then you use multitask learning. The easiest method is to train your model with the apple+orange image, but you can also use another approach like proposed in Learning without Forgetting paper.
  
  > **Comments:**
  > **username_1**: Hi could you take a look here? https://ai.stackexchange.com/questions/27687/visualisation-for-mobilenet-feature-extraction  
  
  ---
  
  