<p>The paper <a href="https://arxiv.org/abs/2012.11881" rel="nofollow noreferrer">Undivided Attention: Are Intermediate Layers Necessary for BERT?</a> should answer it.</p>
<p>In the abstract, they write</p>
<blockquote>
<p>All BERT-based architectures have a self-attention block followed by a block of intermediate layers as the basic building component. However, a strong justification for the inclusion of these intermediate layers remains missing in the literature.</p>
</blockquote>
<p>In the conclusion, they write</p>
<blockquote>
<p>In this work we proposed a modification to the BERT architecture focusing on <strong>reducing the number of intermediate layers in the network.</strong> With the modified BERTBASE network <strong>we show that the network complexity can be significantly decreased while preserving accuracy on fine-tuning tasks.</strong></p>
</blockquote>

CC BY-SA 4.0
2021-12-15T09:26:33.683Z
So, why are "these intermediate layers" used? You're quoting the abstract that says that an explanation in the literature is missing, but do the authors of this paper provide it?
nbro
CC BY-SA 4.0
2021-12-23T02:24:32.523Z
As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer).
Community
CC BY-SA 4.0
2021-12-15T03:00:48.183Z
2021-12-15T09:25:13.787Z
2021-12-15T09:25:13.787Z
Tan Eugene
<p>I am reading the BERT paper <a href="https://arxiv.org/pdf/1810.04805.pdf" rel="nofollow noreferrer">BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding</a>.</p>

<p>As I look at the attention mechanism, I don't understand why in the BERT encoder we have an intermediate layer between the attention and neural network layers with a bigger output (<span class="math-container">$4*H$</span>, where <span class="math-container">$H$</span> is the hidden size).
Perhaps it is the layer normalization, but, by looking at the code, I'm not certain. </p>

CC BY-SA 4.0
2021-10-25T19:47:42.190Z
I am thinking about the same. did you find out why is it there
mohammad ali Humayun
<deep-learning><natural-language-processing><papers><attention><bert>
Why does the BERT encoder have an intermediate layer between the attention and neural network layers with a bigger output?
NathanaëlBeau
# Why does the BERT encoder have an intermediate layer between the attention and neural network layers with a bigger output?

(Asked by: username_0 on 2019-03-14)

I am reading the BERT paper [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/pdf/1810.04805.pdf).

As I look at the attention mechanism, I don't understand why in the BERT encoder we have an intermediate layer between the attention and neural network layers with a bigger output ($4\*H$, where $H$ is the hidden size).
Perhaps it is the layer normalization, but, by looking at the code, I'm not certain.

**Tags:** deep-learning, natural-language-processing, papers, attention, bert

> **Comments:**
> **username_1**: I am thinking about the same. did you find out why is it there  

---

## 1 answers

### Answer by username_2 on 2021-12-15. Score: 2

The paper [Undivided Attention: Are Intermediate Layers Necessary for BERT?](https://arxiv.org/abs/2012.11881) should answer it.

In the abstract, they write

> All BERT-based architectures have a self-attention block followed by a block of intermediate layers as the basic building component. However, a strong justification for the inclusion of these intermediate layers remains missing in the literature.

In the conclusion, they write

> In this work we proposed a modification to the BERT architecture focusing on **reducing the number of intermediate layers in the network.** With the modified BERTBASE network **we show that the network complexity can be significantly decreased while preserving accuracy on fine-tuning tasks.**

> **Comments:**
> **username_3**: So, why are "these intermediate layers" used? You're quoting the abstract that says that an explanation in the literature is missing, but do the authors of this paper provide it?  
> **username_4**: As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer).  

---

