Answers.Body: <p>The usual practice is to the first token embedding as an input to the classifier, which forces the last layer to collect the relevant information from the previous layers to this particular embeddings.</p>
  <p>You might probably view it as wasting the potential of the last layer, where most hidden states are just ignored. From this point of view, an additional attention layer might better use what BERT already knows. On the other hand,  adding an entirely new layer means new parameters that need to be learned from scratch, so more data is needed. Because this is not usually done, I guess the gain from it is probably very small.</p>
  <p>I would rather focus on more traditional ways of improving classification accuracy: cleaning/preprocessing data and better regularization.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-20T22:14:03.913Z
Answers.Text: Please, add information on "where most hidden states are just ignored".
Answers.UserDisplayName: Cloud Cho
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-28T08:49:53.527Z
Answers.LastActivityDate: 2022-04-28T08:49:53.527Z
Answers.UserDisplayName: Jindřich
Answers.Body: <p>A few points to consider:</p>
  <ul>
  <li>Is your dataset big enough for fine-tuning BERT model, a large model might overfit your dataset, and using a small model might help in that case</li>
  <li>Also instead of a whole attention block, try adding only a linear layer to see if that helps, you can also reduce the number of parameters in the layer by using a bottleneck architecture, i.e. adding 2 layers of shape (N x N/4) &amp; (N/4 x N).</li>
  <li>Instead of full fine-tuning, try using LoRA where you can control the rank of matrices according to the size of your dataset.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-20T22:13:11.490Z
Answers.Text: Please, share any reference of LoRA and its experiment.
Answers.UserDisplayName: Cloud Cho
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-21T22:02:36.427Z
Answers.LastActivityDate: 2023-10-21T22:02:36.427Z
Answers.UserDisplayName: basujindal
Body: <p>I'm fine tuning a Bert/Roberta model for a classification task.
  As I need to improve my results, I'm thinking about to add an additional attention layer after Bert model and before dense and dropout layers. Is this a good idea?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-20T22:12:06.233Z
Comments.Text: What method are you using for fine tuning? Do you have your code in GitHub?
Comments.UserDisplayName: Cloud Cho
Tags: <neural-networks><deep-learning><natural-language-processing><attention><bert>
Title: Does it make sense to add an additional attention layer while fine-tuning Bert?
UserDisplayName: Santiago Gonzalez Silot
ThreadText: # Does it make sense to add an additional attention layer while fine-tuning Bert?
  
  (Asked by: username_0 on 2022-04-11)
  
  I'm fine tuning a Bert/Roberta model for a classification task.
  As I need to improve my results, I'm thinking about to add an additional attention layer after Bert model and before dense and dropout layers. Is this a good idea?
  
  **Tags:** neural-networks, deep-learning, natural-language-processing, attention, bert
  > **Comments:**
  > **username_1**: What method are you using for fine tuning? Do you have your code in GitHub?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-04-28. Score: 0
  
  The usual practice is to the first token embedding as an input to the classifier, which forces the last layer to collect the relevant information from the previous layers to this particular embeddings.
  
  You might probably view it as wasting the potential of the last layer, where most hidden states are just ignored. From this point of view, an additional attention layer might better use what BERT already knows. On the other hand, adding an entirely new layer means new parameters that need to be learned from scratch, so more data is needed. Because this is not usually done, I guess the gain from it is probably very small.
  
  I would rather focus on more traditional ways of improving classification accuracy: cleaning/preprocessing data and better regularization.
  
  > **Comments:**
  > **username_1**: Please, add information on "where most hidden states are just ignored".  
  
  ---
  
  ### Answer by username_3 on 2023-10-21. Score: 0
  
  A few points to consider:
  
  * Is your dataset big enough for fine-tuning BERT model, a large model might overfit your dataset, and using a small model might help in that case
  * Also instead of a whole attention block, try adding only a linear layer to see if that helps, you can also reduce the number of parameters in the layer by using a bottleneck architecture, i.e. adding 2 layers of shape (N x N/4) & (N/4 x N).
  * Instead of full fine-tuning, try using LoRA where you can control the rank of matrices according to the size of your dataset.
  
  > **Comments:**
  > **username_1**: Please, share any reference of LoRA and its experiment.  
  
  ---
  
  