Answers.Body: <p><strong>Wikipedia</strong> defines &quot;<a href="https://en.wikipedia.org/wiki/Fine-tuning_(machine_learning)" rel="nofollow noreferrer">fine-tuning</a>&quot; as:</p>
  <blockquote>
  <p>an approach to transfer learning in which the weights of a pre-trained model are trained on new data. Fine-tuning can be done on the entire neural network, or on only a subset of its layers, in which case the layers that are not being fine-tuned are &quot;frozen&quot;</p>
  </blockquote>
  <p>You didn't update any weights of the original model, so, at least according to Wikipedia, that wasn't &quot;fine-tuning&quot;.</p>
  <p><strong>On the other hand</strong>, one can argue that nowadays the common use of &quot;fine-tuning&quot; covers much more than just <em>directly</em> updating the weights of the original model.</p>
  <p>A popular <a href="https://github.com/huggingface/peft" rel="nofollow noreferrer">Parameter-Efficient Fine-Tuning</a> package by huggingface lists (soft)prompt tuning and Low-Rank Adaptation (LoRA) as its &quot;fine-tuning&quot; methods. It is possible to argue that these approaches do not <em>completely</em> satisfy the Wikipedia definition. Still, both approaches modify the original model weights, albeit indirectly: soft prompts modify the embedding vectors at the model inputs, and LoRA adds the low-rank restored matrices to the original model weights.</p>
  <p><strong>Finally,</strong> if somebody told me that he &quot;fine-tuned BERT&quot; and later I've discovered that he did what you've described -- I would think that I was misled. To me, the ultimate criterion than disqualifies something from being &quot;fine-tuning&quot; is the fact that for a fixed input you can just <em>save</em> the pre-trained BERT outputs and never return to running BERT inference again.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-02T11:33:35.317Z
Answers.LastActivityDate: 2023-06-02T11:33:35.317Z
Answers.UserDisplayName: Kostya
Answers.Body: <p>Yes, that would still be counted as a fine-tuning model. Unless untouching the pre-trained model, any modification that helps for your downstream task or to suit your data more IS an act of fine-tuning your model.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-31T05:10:43.070Z
Answers.LastActivityDate: 2023-05-31T05:10:43.070Z
Answers.UserDisplayName: GunFire
Answers.Body: <p>I'd like to add another angle to @kostya's excellent answer.</p>
  <p>I see the lack of use of additional input as a key distinction for not calling this adaptation fine-tuning. Moreover, as in today's realities, fine-tuning is still an art, so we might consider the amount of professional skill and effort required. The amount of input, skill and effort are all continuous quantities, and there is no clear threshold at which a process of adapting of the trained model is no longer “fine-tuning”. However, when the above qualities are absent or minimal, the use of the term &quot;fine-tuning&quot;  tends to devalue the term and blur its meaning.</p>
  <p>It is likely that the term “fine-tuning” and a few others will continue their transformation and become a bit of a catch-all phrase, and being used rather loosely in different contexts. But in a professional setting, it's better to avoid any misunderstanding about the complexity of the work involved.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-25T09:38:30.740Z
Answers.LastActivityDate: 2024-08-25T09:40:55.687Z
Answers.LastEditDate: 2024-08-25T09:40:55.687Z
Answers.UserDisplayName: David Khosid
Body: <p>In the context of machine learning.  If I freeze pre-trained model weights (for example, BERT) and then train a classifier on top of its embeddings, does that called fine-tuning?</p>
  
Tags: <neural-networks><machine-learning><deep-learning><natural-language-processing><computer-vision>
Title: If I freeze pre-trained model weights and than train a classifier on top of its embeddings does that called fine-tunning?
UserDisplayName: Lampent96
ThreadText: # If I freeze pre-trained model weights and than train a classifier on top of its embeddings does that called fine-tunning?
  
  (Asked by: username_0 on 2023-05-30)
  
  In the context of machine learning. If I freeze pre-trained model weights (for example, BERT) and then train a classifier on top of its embeddings, does that called fine-tuning?
  
  **Tags:** neural-networks, machine-learning, deep-learning, natural-language-processing, computer-vision
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2023-06-02. Score: 2
  
  **Wikipedia** defines "[fine-tuning](https://en.wikipedia.org/wiki/Fine-tuning_(machine_learning))" as:
  
  > an approach to transfer learning in which the weights of a pre-trained model are trained on new data. Fine-tuning can be done on the entire neural network, or on only a subset of its layers, in which case the layers that are not being fine-tuned are "frozen"
  
  You didn't update any weights of the original model, so, at least according to Wikipedia, that wasn't "fine-tuning".
  
  **On the other hand**, one can argue that nowadays the common use of "fine-tuning" covers much more than just *directly* updating the weights of the original model.
  
  A popular [Parameter-Efficient Fine-Tuning](https://github.com/huggingface/peft) package by huggingface lists (soft)prompt tuning and Low-Rank Adaptation (LoRA) as its "fine-tuning" methods. It is possible to argue that these approaches do not *completely* satisfy the Wikipedia definition. Still, both approaches modify the original model weights, albeit indirectly: soft prompts modify the embedding vectors at the model inputs, and LoRA adds the low-rank restored matrices to the original model weights.
  
  **Finally,** if somebody told me that he "fine-tuned BERT" and later I've discovered that he did what you've described -- I would think that I was misled. To me, the ultimate criterion than disqualifies something from being "fine-tuning" is the fact that for a fixed input you can just *save* the pre-trained BERT outputs and never return to running BERT inference again.
  
  
  ---
  
  ### Answer by username_2 on 2023-05-31. Score: 0
  
  Yes, that would still be counted as a fine-tuning model. Unless untouching the pre-trained model, any modification that helps for your downstream task or to suit your data more IS an act of fine-tuning your model.
  
  
  ---
  
  ### Answer by username_3 on 2024-08-25. Score: 0
  
  I'd like to add another angle to @username_1's excellent answer.
  
  I see the lack of use of additional input as a key distinction for not calling this adaptation fine-tuning. Moreover, as in today's realities, fine-tuning is still an art, so we might consider the amount of professional skill and effort required. The amount of input, skill and effort are all continuous quantities, and there is no clear threshold at which a process of adapting of the trained model is no longer “fine-tuning”. However, when the above qualities are absent or minimal, the use of the term "fine-tuning" tends to devalue the term and blur its meaning.
  
  It is likely that the term “fine-tuning” and a few others will continue their transformation and become a bit of a catch-all phrase, and being used rather loosely in different contexts. But in a professional setting, it's better to avoid any misunderstanding about the complexity of the work involved.
  
  
  ---
  
  