Answers.Body: <p>Corrections and other answers are welcome, but here are a few thoughts:</p>
  <p>There are several approaches in terms of which weights get frozen (and also other considerations, see for example Fig. 5 in <a href="https://arxiv.org/abs/2211.09085v1" rel="nofollow noreferrer">&quot;Galactica: A Large Language Model for Science&quot;</a>).</p>
  <p>Which of the approaches yields higher-quality results <strong>depends on the architecture (and hyperparameters) and dataset</strong>.</p>
  <p>There can be rules of thumb, for example <a href="https://web.archive.org/web/20211220033410/https://huggingface.co/docs/transformers/training" rel="nofollow noreferrer">this old snapshot of a &quot;Documentation&quot; of Transformer architectures at Hugging Face</a> said:</p>
  <blockquote>
  <p>we are directly fine-tuning the whole model without taking any precaution. It actually works better this way for Transformers model</p>
  </blockquote>
  <p>but this explanation apparently was removed from <a href="https://huggingface.co/docs/transformers/training" rel="nofollow noreferrer">the new vesion of this page</a>. Maybe it turned out that such <strong>rules of thumb aren't right in general</strong>.</p>
  <p><strong>Quality of results</strong> is also not the only thing being optimized. Some choices are made due to <strong>memory or compute considerations</strong>. For example, when freezing the first layers, their output features can be computed only once for all samples, saved, and used thereafter; moreover, computing the gradient of the loss with respect to the weights of the first frozen network block is not necessary.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-07T15:50:21.727Z
Answers.LastActivityDate: 2023-01-07T15:50:21.727Z
Answers.UserDisplayName: root
Body: <p>During transfer learning in computer vision, I've seen that the layers of the base model are frozen if the images aren't too different from the model on which the base model is trained on.</p>
  <p>However, on the NLP side, I see that the layers of the BERT model aren't ever frozen. What is the reason for this?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-03T14:38:34.180Z
Comments.Text: Maybe you should provide the link to 2-3 examples (of implementations or papers/models) where the layers are not frozen (just to give more context). In any case, freezing the layers may not be necessarily required if you use a very low learning rate (but I am not an expert on this topic).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-05T17:20:47.663Z
Comments.Text: I wouldn't say they are never frozen, just rarely. Generally, you fine-tune the entire model if you care more about accuracy (or whatever metric) while you freeze layers and fine-tune only the later layers if you care more about fine-tuning/prediction speed and memory usage. For example, fine-tuning the entire BERT on a GPU can take about 8GB of VRAM in my experience, which can be reduced by a gb or two by freezing layers.
Comments.UserDisplayName: primussucks
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-16T14:01:55.827Z
Comments.Text: @primussucks if we don't freeze bert new layers will mess up trained weights. isn't it so?
Comments.UserDisplayName: dato nefaridze
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-18T07:53:10.183Z
Comments.Text: @datonefaridze It will change the weights yes, but "mess up" suggests that the model will perform poorly if they aren't frozen, which just isn't true.
Comments.UserDisplayName: primussucks
Tags: <natural-language-processing><computer-vision><bert><transfer-learning><fine-tuning>
Title: Why aren't the BERT layers frozen during fine-tuning tasks?
UserDisplayName: Bunny Rabbit
ThreadText: # Why aren't the BERT layers frozen during fine-tuning tasks?
  
  (Asked by: username_0 on 2020-10-03)
  
  During transfer learning in computer vision, I've seen that the layers of the base model are frozen if the images aren't too different from the model on which the base model is trained on.
  
  However, on the NLP side, I see that the layers of the BERT model aren't ever frozen. What is the reason for this?
  
  **Tags:** natural-language-processing, computer-vision, bert, transfer-learning, fine-tuning
  
  > **Comments:**
  > **username_1**: Maybe you should provide the link to 2-3 examples (of implementations or papers/models) where the layers are not frozen (just to give more context). In any case, freezing the layers may not be necessarily required if you use a very low learning rate (but I am not an expert on this topic).  
  > **username_2**: I wouldn't say they are never frozen, just rarely. Generally, you fine-tune the entire model if you care more about accuracy (or whatever metric) while you freeze layers and fine-tune only the later layers if you care more about fine-tuning/prediction speed and memory usage. For example, fine-tuning the entire BERT on a GPU can take about 8GB of VRAM in my experience, which can be reduced by a gb or two by freezing layers.  
  > **username_3**: @username_2 if we don't freeze bert new layers will mess up trained weights. isn't it so?  
  > **username_2**: @username_3 It will change the weights yes, but "mess up" suggests that the model will perform poorly if they aren't frozen, which just isn't true.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_4 on 2023-01-07. Score: 0
  
  Corrections and other answers are welcome, but here are a few thoughts:
  
  There are several approaches in terms of which weights get frozen (and also other considerations, see for example Fig. 5 in ["Galactica: A Large Language Model for Science"](https://arxiv.org/abs/2211.09085v1)).
  
  Which of the approaches yields higher-quality results **depends on the architecture (and hyperparameters) and dataset**.
  
  There can be rules of thumb, for example [this old snapshot of a "Documentation" of Transformer architectures at Hugging Face](https://web.archive.org/web/20211220033410/https://huggingface.co/docs/transformers/training) said:
  
  > we are directly fine-tuning the whole model without taking any precaution. It actually works better this way for Transformers model
  
  but this explanation apparently was removed from [the new vesion of this page](https://huggingface.co/docs/transformers/training). Maybe it turned out that such **rules of thumb aren't right in general**.
  
  **Quality of results** is also not the only thing being optimized. Some choices are made due to **memory or compute considerations**. For example, when freezing the first layers, their output features can be computed only once for all samples, saved, and used thereafter; moreover, computing the gradient of the loss with respect to the weights of the first frozen network block is not necessary.
  
  
  ---
  
  