Answers.Body: <p>The paper <a href="https://arxiv.org/abs/2012.11881" rel="nofollow noreferrer">Undivided Attention: Are Intermediate Layers Necessary for BERT?</a> should answer it.</p>
  <p>In the abstract, they write</p>
  <blockquote>
  <p>All BERT-based architectures have a self-attention block followed by a block of intermediate layers as the basic building component. However, a strong justification for the inclusion of these intermediate layers remains missing in the literature.</p>
  </blockquote>
  <p>In the conclusion, they write</p>
  <blockquote>
  <p>In this work we proposed a modification to the BERT architecture focusing on <strong>reducing the number of intermediate layers in the network.</strong> With the modified BERTBASE network <strong>we show that the network complexity can be significantly decreased while preserving accuracy on fine-tuning tasks.</strong></p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-15T09:26:33.683Z
Answers.Text: So, why are "these intermediate layers" used? You're quoting the abstract that says that an explanation in the literature is missing, but do the authors of this paper provide it?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-23T02:24:32.523Z
Answers.Text: As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer).
Answers.UserDisplayName: Community
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-15T03:00:48.183Z
Answers.LastActivityDate: 2021-12-15T09:25:13.787Z
Answers.LastEditDate: 2021-12-15T09:25:13.787Z
Answers.UserDisplayName: Tan Eugene
Body: <p>I am reading the BERT paper <a href="https://arxiv.org/pdf/1810.04805.pdf" rel="nofollow noreferrer">BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding</a>.</p>
  
  <p>As I look at the attention mechanism, I don't understand why in the BERT encoder we have an intermediate layer between the attention and neural network layers with a bigger output (<span class="math-container">$4*H$</span>, where <span class="math-container">$H$</span> is the hidden size).
  Perhaps it is the layer normalization, but, by looking at the code, I'm not certain. </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-25T19:47:42.190Z
Comments.Text: I am thinking about the same. did you find out why is it there
Comments.UserDisplayName: mohammad ali Humayun
Tags: <deep-learning><natural-language-processing><papers><attention><bert>
Title: Why does the BERT encoder have an intermediate layer between the attention and neural network layers with a bigger output?
UserDisplayName: NathanaëlBeau
ThreadText: # Why does the BERT encoder have an intermediate layer between the attention and neural network layers with a bigger output?
  
  (Asked by: username_0 on 2019-03-14)
  
  I am reading the BERT paper [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/pdf/1810.04805.pdf).
  
  As I look at the attention mechanism, I don't understand why in the BERT encoder we have an intermediate layer between the attention and neural network layers with a bigger output ($4\*H$, where $H$ is the hidden size).
  Perhaps it is the layer normalization, but, by looking at the code, I'm not certain.
  
  **Tags:** deep-learning, natural-language-processing, papers, attention, bert
  > **Comments:**
  > **username_1**: I am thinking about the same. did you find out why is it there  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-12-15. Score: 2
  
  The paper [Undivided Attention: Are Intermediate Layers Necessary for BERT?](https://arxiv.org/abs/2012.11881) should answer it.
  
  In the abstract, they write
  
  > All BERT-based architectures have a self-attention block followed by a block of intermediate layers as the basic building component. However, a strong justification for the inclusion of these intermediate layers remains missing in the literature.
  
  In the conclusion, they write
  
  > In this work we proposed a modification to the BERT architecture focusing on **reducing the number of intermediate layers in the network.** With the modified BERTBASE network **we show that the network complexity can be significantly decreased while preserving accuracy on fine-tuning tasks.**
  
  > **Comments:**
  > **username_3**: So, why are "these intermediate layers" used? You're quoting the abstract that says that an explanation in the literature is missing, but do the authors of this paper provide it?  
  > **username_4**: As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer).  
  
  ---
  
  