Answers.Body: <p>It has been a while since the question was asked, but I came up with this <a href="https://developer.baidu.com/article/details/2727266" rel="nofollow noreferrer">article</a>. It helped me to understand the topic. From the article:</p>
  <p><strong>Feature-based</strong> methods involve using the intermediate representations or features from a pre-trained model as additional inputs to a task-specific model.</p>
  <p><strong>Fine-tuning</strong>, on the other hand, involves modifying and retraining a pre-trained model to adapt it to a specific task.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-20T14:23:51.013Z
Answers.LastActivityDate: 2024-03-20T14:23:51.013Z
Answers.UserDisplayName: cancan
Answers.Body: <p>The difference between the two approaches (<strong>feature extraction</strong> vs <strong>fine-tuning</strong>) is well explained here:
  <a href="https://stats.stackexchange.com/questions/255364/fine-tuning-vs-joint-training-vs-feature-extraction">Fine Tuning vs Joint Training vs Feature Extraction</a></p>
  <p>Also, this paper evaluate the performance one can hope to achieve with 2 sequence models (ELMo and BERT) with each approach:
  <a href="https://arxiv.org/abs/1903.05987" rel="nofollow noreferrer">To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-12T11:47:54.993Z
Answers.Text: Hello. Welcome to our community! Could you please provide more details and not just provide a link to an external source? We usually expect answers to be self-contained and links should only be used to provide additional details, unless the person was specifically asking for a reference/link.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-12T17:33:20.443Z
Answers.Text: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).
Answers.UserDisplayName: Community
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-12T11:41:52.127Z
Answers.LastActivityDate: 2022-01-12T11:41:52.127Z
Answers.UserDisplayName: couturierc
Answers.Body: <p>Typically, in transfer learning, you have 2-3 stages</p>
  <ol>
  <li><p><strong>Pre-training</strong>: pre-train some base model <span class="math-container">$M_\text{base}$</span> on some &quot;general&quot; dataset <span class="math-container">$A$</span>; note that you may not necessarily need to train <span class="math-container">$M_\text{base}$</span>, but it may already be available e.g. on the web. During this phase, we <strong>extract</strong> (general) <strong>features</strong> or <a href="https://ai.stackexchange.com/a/28001/2444">learn representations of the data</a>, which can &quot;bootstrap&quot; the learning task with your specific dataset</p>
  </li>
  <li><p><strong>Training</strong>:  You replace the last layers of <span class="math-container">$M_\text{base}$</span> (i.e. the classifier/regression part) with new layers to solve your task, then you <strong>might</strong> freeze the initial layers (e.g. the convolutional layers) that are assumed to contain the general extracted features that can also be useful for your task: let's call this model <span class="math-container">$M_\text{main}$</span>; at this point, you train this <strong>partially</strong> frozen model <span class="math-container">$M_\text{main}$</span> with your dataset <span class="math-container">$B$</span>.</p>
  </li>
  <li><p><strong>Fine-tuning</strong>: after training, you could unfreeze some of the frozen layers in <span class="math-container">$M_\text{main}$</span>, especially the ones closest to your new classifier, then train again</p>
  </li>
  </ol>
  <p>In all 3 stages, one could say that we're extracting features (because we're learning weights), but some people, I guess, will refer to the pre-training phase as the feature extraction phase. I think I've seen people call the training stage also the fine-tuning stage (and the previous version of this answer actually was referring to the training phase as the fine-tuning phase), but, in the end, these terms could be used inconsistently anyway, so the important thing is that you understand what's going on and keep context into account.</p>
  <p>You can find more information about this topic <a href="https://www.tensorflow.org/tutorials/images/transfer_learning" rel="nofollow noreferrer">here</a>. Note that there may be other more sophisticated or simply different approaches to transfer learning.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-12T14:42:33.900Z
Answers.Text: Fine-tuning most often does not include freezing the previous trained layers. Actually it is more likely related to training the whole model for a few more iterations on the task specific dataset after pretraining on the task-unrelated dataset.
Answers.UserDisplayName: Marcel_marcel1991
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-12T14:56:03.987Z
Answers.Text: @Marcel_marcel1991 Well, during fine-tuning, you don't need to unfreeze all layers, but maybe you're right that some people may use fine-tuning to mean specifically when you unfreeze some of the layers. The TF tutorial says, for instance, "Also, you should try to fine-tune a small number of top layers rather than the whole MobileNet model." I will edit my answer to point that out.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-12T15:24:15.720Z
Answers.Text: @Marcel_marcel1991 If you look at the answer on Stats that the other answer links to, they are using the word _fine-tuning_ to refer to something slightly different (i.e. learning new tasks or classes continuously by slightly changing the output layer), and it's in the context of _continual learning_, rather than transfer learning (as described in TF tutorial and in other papers). So, as I say now in my answer, it's possible people may be using these terms to refer to slightly different things.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-12T15:27:56.463Z
Answers.Text: I agree with you that there is no clear definition. I was only relating to that your answer assumes that you would always freeze the previous layers in fine-tuning, which is not the case. The edit now is fine to me.
Answers.UserDisplayName: Marcel_marcel1991
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-07T23:34:37.283Z
Answers.LastActivityDate: 2022-01-12T15:26:39.683Z
Answers.LastEditDate: 2022-01-12T15:26:39.683Z
Answers.UserDisplayName: nbro
Body: <p>I'm building a model for <strong>facial expression recognition</strong>, and I want to use <em><strong>transfer learning</strong></em>. From what I understand, there are different steps to do it. The first is the <strong>feature extraction</strong> and the second is <strong>fine-tuning</strong>. I want to understand more about these two stages, and the difference between them. Must we use them simultaneously in the same training?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-07T22:39:13.810Z
Comments.Text: Hello. Welcome to AI SE. Could you please provide the link the resource where you were reading that claims that "feature extraction" and "fine-tuning" are two different approaches to transfer learning? Because I don't think that's the case. I think these are just 2 stages of transfer learning, and maybe this would be the start of an answer to your question in the title, but I could be wrong.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-07T22:53:48.053Z
Comments.Text: Hello. Yeah I just realized that I was totaly wrong. Yes there are 2 stages in transfer leraning but should we use them in the same training or there is a difference ? Thank you for your reply.
Comments.UserDisplayName: Speedskillsx
Tags: <deep-learning><transfer-learning><feature-extraction><fine-tuning><emotion-recognition>
Title: What is the difference between feature extraction and fine-tuning in transfer learning?
UserDisplayName: Speedskillsx
ThreadText: # What is the difference between feature extraction and fine-tuning in transfer learning?
  
  (Asked by: username_0 on 2021-06-07)
  
  I'm building a model for **facial expression recognition**, and I want to use ***transfer learning***. From what I understand, there are different steps to do it. The first is the **feature extraction** and the second is **fine-tuning**. I want to understand more about these two stages, and the difference between them. Must we use them simultaneously in the same training?
  
  **Tags:** deep-learning, transfer-learning, feature-extraction, fine-tuning, emotion-recognition
  
  > **Comments:**
  > **username_1**: Hello. Welcome to AI SE. Could you please provide the link the resource where you were reading that claims that "feature extraction" and "fine-tuning" are two different approaches to transfer learning? Because I don't think that's the case. I think these are just 2 stages of transfer learning, and maybe this would be the start of an answer to your question in the title, but I could be wrong.  
  > **username_0**: Hello. Yeah I just realized that I was totaly wrong. Yes there are 2 stages in transfer leraning but should we use them in the same training or there is a difference ? Thank you for your reply.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2024-03-20. Score: 0 | Accepted answer
  
  It has been a while since the question was asked, but I came up with this [article](https://developer.baidu.com/article/details/2727266). It helped me to understand the topic. From the article:
  
  **Feature-based** methods involve using the intermediate representations or features from a pre-trained model as additional inputs to a task-specific model.
  
  **Fine-tuning**, on the other hand, involves modifying and retraining a pre-trained model to adapt it to a specific task.
  
  
  ---
  
  ### Answer by username_3 on 2022-01-12. Score: 1
  
  The difference between the two approaches (**feature extraction** vs **fine-tuning**) is well explained here:
  [Fine Tuning vs Joint Training vs Feature Extraction](https://stats.stackexchange.com/questions/255364/fine-tuning-vs-joint-training-vs-feature-extraction)
  
  Also, this paper evaluate the performance one can hope to achieve with 2 sequence models (ELMo and BERT) with each approach:
  [To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks](https://arxiv.org/abs/1903.05987)
  
  > **Comments:**
  > **username_1**: Hello. Welcome to our community! Could you please provide more details and not just provide a link to an external source? We usually expect answers to be self-contained and links should only be used to provide additional details, unless the person was specifically asking for a reference/link.  
  > **username_4**: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).  
  
  ---
  
  ### Answer by username_1 on 2021-06-07. Score: 0
  
  Typically, in transfer learning, you have 2-3 stages
  
  1. **Pre-training**: pre-train some base model $M\_\text{base}$ on some "general" dataset $A$; note that you may not necessarily need to train $M\_\text{base}$, but it may already be available e.g. on the web. During this phase, we **extract** (general) **features** or [learn representations of the data](https://ai.stackexchange.com/a/28001/2444), which can "bootstrap" the learning task with your specific dataset
  2. **Training**: You replace the last layers of $M\_\text{base}$ (i.e. the classifier/regression part) with new layers to solve your task, then you **might** freeze the initial layers (e.g. the convolutional layers) that are assumed to contain the general extracted features that can also be useful for your task: let's call this model $M\_\text{main}$; at this point, you train this **partially** frozen model $M\_\text{main}$ with your dataset $B$.
  3. **Fine-tuning**: after training, you could unfreeze some of the frozen layers in $M\_\text{main}$, especially the ones closest to your new classifier, then train again
  
  In all 3 stages, one could say that we're extracting features (because we're learning weights), but some people, I guess, will refer to the pre-training phase as the feature extraction phase. I think I've seen people call the training stage also the fine-tuning stage (and the previous version of this answer actually was referring to the training phase as the fine-tuning phase), but, in the end, these terms could be used inconsistently anyway, so the important thing is that you understand what's going on and keep context into account.
  
  You can find more information about this topic [here](https://www.tensorflow.org/tutorials/images/transfer_learning). Note that there may be other more sophisticated or simply different approaches to transfer learning.
  
  > **Comments:**
  > **username_5**: Fine-tuning most often does not include freezing the previous trained layers. Actually it is more likely related to training the whole model for a few more iterations on the task specific dataset after pretraining on the task-unrelated dataset.  
  > **username_1**: @username_5 Well, during fine-tuning, you don't need to unfreeze all layers, but maybe you're right that some people may use fine-tuning to mean specifically when you unfreeze some of the layers. The TF tutorial says, for instance, "Also, you should try to fine-tune a small number of top layers rather than the whole MobileNet model." I will edit my answer to point that out.  
  > **username_1**: @username_5 If you look at the answer on Stats that the other answer links to, they are using the word _fine-tuning_ to refer to something slightly different (i.e. learning new tasks or classes continuously by slightly changing the output layer), and it's in the context of _continual learning_, rather than transfer learning (as described in TF tutorial and in other papers). So, as I say now in my answer, it's possible people may be using these terms to refer to slightly different things.  
  > **username_5**: I agree with you that there is no clear definition. I was only relating to that your answer assumes that you would always freeze the previous layers in fine-tuning, which is not the case. The edit now is fine to me.  
  
  ---
  
  