Answers.Body: <p>Taken directly from <a href="https://huggingface.co/transformers/training.html" rel="nofollow noreferrer">HuggingFace</a></p>
  <blockquote>
  <p>Note that if you are used to freezing the body of your pretrained model (like in computer vision) the above may seem a bit strange, as we are directly fine-tuning the whole model without taking any precaution. It actually works better this way for Transformers model (so this is not an oversight on our side). If you’re not familiar with what “freezing the body” of the model means, forget you read this paragraph.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-15T03:40:49.720Z
Answers.LastActivityDate: 2021-07-15T12:29:19.293Z
Answers.LastEditDate: 2021-07-15T12:29:19.293Z
Answers.UserDisplayName: Joon
Body: <p>Recently, I came across the BERT model. I did some research and tried some implementations.</p>
  <p>I wanted to tackle a NER task, so I chose the BertForSequenceClassifications provided by HuggingFace.</p>
  <pre><code>for epoch in range(1, args.epochs + 1):
      total_loss = 0
      model.train()
      for step, batch in enumerate(train_loader):
          b_input_ids = batch[0].to(device)
          b_input_mask = batch[1].to(device)
          b_labels = batch[2].to(device)
          model.zero_grad()
  
          outputs = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)
          loss = outputs[0]
  
          total_loss += loss.item()
          loss.backward()
          torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
          # modified based on their gradients, the learning rate, etc.
          optimizer.step()
  </code></pre>
  <p>The main part of my fine-tuning follows as above.</p>
  <p>I am curious about to what extent the fine-tuning alters the model. Does it freeze the weights that have been provided by the pre-trained model and only alter the top classification layer, or does it change the hidden layers that are contained in the already pre-trained BERT model?</p>
  
Tags: <bert><pretrained-models><fine-tuning><named-entity-recognition>
Title: Does BERT freeze the entire model body when it does fine-tuning?
UserDisplayName: Joon
ThreadText: # Does BERT freeze the entire model body when it does fine-tuning?
  
  (Asked by: username_0 on 2021-07-07)
  
  Recently, I came across the BERT model. I did some research and tried some implementations.
  
  I wanted to tackle a NER task, so I chose the BertForSequenceClassifications provided by HuggingFace.
  
  ```
  for epoch in range(1, args.epochs + 1):
      total_loss = 0
      model.train()
      for step, batch in enumerate(train_loader):
          b_input_ids = batch[0].to(device)
          b_input_mask = batch[1].to(device)
          b_labels = batch[2].to(device)
          model.zero_grad()
  
          outputs = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)
          loss = outputs[0]
  
          total_loss += loss.item()
          loss.backward()
          torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
          # modified based on their gradients, the learning rate, etc.
          optimizer.step()
  
  ```
  
  The main part of my fine-tuning follows as above.
  
  I am curious about to what extent the fine-tuning alters the model. Does it freeze the weights that have been provided by the pre-trained model and only alter the top classification layer, or does it change the hidden layers that are contained in the already pre-trained BERT model?
  
  **Tags:** bert, pretrained-models, fine-tuning, named-entity-recognition
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2021-07-15. Score: 3 | Accepted answer
  
  Taken directly from [HuggingFace](https://huggingface.co/transformers/training.html)
  
  > Note that if you are used to freezing the body of your pretrained model (like in computer vision) the above may seem a bit strange, as we are directly fine-tuning the whole model without taking any precaution. It actually works better this way for Transformers model (so this is not an oversight on our side). If you’re not familiar with what “freezing the body” of the model means, forget you read this paragraph.
  
  
  ---
  
  