Answers.Body: <p>Feedforward layer is an important part of the transformer architecture.</p>
  <p>Transformer architecture, in addition to the <code>self-attention</code> layer, that aggregates information from the whole sequence and transforms each token due to the attention scores from the <code>queries</code> and <code>values</code> has a feedforward layer, which is mostly a 2-layer MLP, that processes each token separately:
  <span class="math-container">$$
  y = W_2 \sigma(W_1 x + b_1) + b_2
  $$</span></p>
  <p>Where <span class="math-container">$W_1, W_2$</span> are the weights, and <span class="math-container">$b_1, b_2$</span> - biases, <span class="math-container">$\sigma$</span> - is the nonlinearity <code>ReLU, GeLU</code>, e.t.c.</p>
  <p><a href="https://i.sstatic.net/CUBEd.png" rel="noreferrer"><img src="https://i.sstatic.net/CUBEd.png" alt="enter image description here" /></a></p>
  <p>This is kind of a <em>pointwise</em> nonlinear transformation of the sequence.</p>
  <p>I suspect, that <code>intermediate</code> here corresponds to <span class="math-container">$W_1, b_1$</span> and the output is about <span class="math-container">$W_2, b_2$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-16T19:32:41.603Z
Answers.Text: The language "take a moment to think and process this information" is a seriously unhelpful and obfuscating way to describe a matrix multiplication. I would discourage people from unnecessary anthropomorphism of simple operations just because they are in the context of AI.
Answers.UserDisplayName: Jon Deaton
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-26T12:18:50.893Z
Answers.Text: Hi, thank you for your response. I understand that the attention sublayer in the encoder has a fully connected dense layer followed by another output fully connected dense layer. but all pre-trained transformers at hugging face have three fully connected layers in each encoder block. one in attention one intermediate and one output. just for confirmation, are you suggesting that the MLP within attention (before skip connection) has two layers one of which is being referred to as the intermediate layer?  thanx again
Answers.UserDisplayName: mohammad ali Humayun
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-02T09:08:40.180Z
Answers.Text: @mohammadaliHumayun The multihead-attention block has four linear layers - $W_Q$, $W_K$, $W_V$ and $W_O$. The layer $W_O$ is used to combine the concatenated output from the multiple heads. It is called `attention.output.dense` in the model that you are using. After the attention block you have a multi-layer perceptron (MLP). As explained in the answer, the MLP usually has two layers $W_1$ and $W_2$. So $W_1$ is called `intermidiate.dense` and $W_2$ is called `output.dense`.
Answers.UserDisplayName: pi-tau
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-26T07:25:53.980Z
Answers.LastActivityDate: 2021-10-26T07:25:53.980Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Answers.Body: <p><a href="https://huggingface.co/docs/transformers/main/en/model_doc/bert#transformers.BertForMaskedLM" rel="nofollow noreferrer">documentaion of Bert on Hugging Face</a></p>
  <p>you will find that</p>
  <blockquote>
  <p>intermediate_size (int, optional, defaults to 3072) — Dimensionality of the “intermediate” (often named feed-forward) layer in the Transformer encoder.</p>
  </blockquote>
  <p>so intermediate layer is feed-forward layer.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-02T08:45:43.827Z
Answers.LastActivityDate: 2023-08-02T08:45:43.827Z
Answers.UserDisplayName: Ya Wen
Answers.Body: <p>So I came across this myself when trying to explore why models require intermediate layers in general. BERT is fundamentally different than the previous transformer architectures. The intermediate layer acts as a way to ensure the different multi attention heads are able to utilize pre-trained information. So for example if you examine the block directly you will notice they upscale and descale in my experience something like 768 dim to 3072 back down to 768, I am approximating dimensions from memory. This does not correspond to the KQV in the attention layer weights. Now it is still a point of research concern why need the scale and descale of the embedded information to allow for fine tuning of the models. When we remove the intermediate layers we have considerable drop in transfer learning performance and therefore pre-training experience is lost. You will also notice more recent models have this intermediate layer as opposed to the original transformer paper.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-06T16:19:38.350Z
Answers.LastActivityDate: 2023-12-06T16:19:38.350Z
Answers.UserDisplayName: selachianhandler
Body: <p>In PyTorch, transformer (BERT) models have an intermediate dense layer in between attention and output layers whereas the BERT and Transformer papers just mention the attention connected directly to output fully connected layer for the encoder just after adding the residual connection.</p>
  <p>Why is there an intermediate layer within an encoder block?</p>
  <p>For example,</p>
  <blockquote>
  <p>encoder.layer.11.attention.self.query.weight<br />
  encoder.layer.11.attention.self.query.bias<br />
  encoder.layer.11.attention.self.key.weight<br />
  encoder.layer.11.attention.self.key.bias<br />
  encoder.layer.11.attention.self.value.weight<br />
  encoder.layer.11.attention.self.value.bias<br />
  encoder.layer.11.attention.output.dense.weight<br />
  encoder.layer.11.attention.output.dense.bias<br />
  encoder.layer.11.attention.output.LayerNorm.weight<br />
  encoder.layer.11.attention.output.LayerNorm.bias<br />
  <strong>encoder.layer.11.intermediate.dense.weight<br />
  encoder.layer.11.intermediate.dense.bias</strong><br />
  encoder.layer.11.output.dense.weight<br />
  encoder.layer.11.output.dense.bias<br />
  encoder.layer.11.output.LayerNorm.weight<br />
  encoder.layer.11.output.LayerNorm.bias</p>
  </blockquote>
  <p>I am confused by this third (intermediate dense layer) in between attention output and encoder output dense layers</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-26T00:22:46.197Z
Comments.Text: Welcome to SE:AI!  I did a slight edit to draw attention to the question posed.
Comments.UserDisplayName: DukeZhou
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-26T11:41:24.137Z
Comments.Text: Hello. Can you please provide the link to the PyTorch model that has an intermediate layer that you're referring to?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-26T12:15:15.270Z
Comments.Text: Hello thank you for your response. in fact each pre-trained model on hugging face has the same architecture. you can look at the following for example 'bert-base-uncased' at https://huggingface.co/bert-base-uncased (three fully connected layers. one in attention one intermediate and one output. for all encoder blocks)
Comments.UserDisplayName: mohammad ali Humayun
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-26T12:25:09.460Z
Comments.Text: I am confused by this third (intermediate dense layer) in between attention output and encoder output dense layers.
Comments.UserDisplayName: mohammad ali Humayun
Tags: <natural-language-processing><pytorch><transformer><bert>
Title: What is the Intermediate (dense) layer in between attention-output and encoder-output dense layers within transformer block in PyTorch implementation?
UserDisplayName: mohammad ali Humayun
ThreadText: # What is the Intermediate (dense) layer in between attention-output and encoder-output dense layers within transformer block in PyTorch implementation?
  
  (Asked by: username_0 on 2021-10-25)
  
  In PyTorch, transformer (BERT) models have an intermediate dense layer in between attention and output layers whereas the BERT and Transformer papers just mention the attention connected directly to output fully connected layer for the encoder just after adding the residual connection.
  
  Why is there an intermediate layer within an encoder block?
  
  For example,
  
  > encoder.layer.11.attention.self.query.weight  
  > encoder.layer.11.attention.self.query.bias  
  > encoder.layer.11.attention.self.key.weight  
  > encoder.layer.11.attention.self.key.bias  
  > encoder.layer.11.attention.self.value.weight  
  > encoder.layer.11.attention.self.value.bias  
  > encoder.layer.11.attention.output.dense.weight  
  > encoder.layer.11.attention.output.dense.bias  
  > encoder.layer.11.attention.output.LayerNorm.weight  
  > encoder.layer.11.attention.output.LayerNorm.bias  
  > **encoder.layer.11.intermediate.dense.weight  
  > encoder.layer.11.intermediate.dense.bias**  
  > encoder.layer.11.output.dense.weight  
  > encoder.layer.11.output.dense.bias  
  > encoder.layer.11.output.LayerNorm.weight  
  > encoder.layer.11.output.LayerNorm.bias
  
  I am confused by this third (intermediate dense layer) in between attention output and encoder output dense layers
  
  **Tags:** natural-language-processing, pytorch, transformer, bert
  > **Comments:**
  > **username_1**: Welcome to SE:AI!  I did a slight edit to draw attention to the question posed.  
  > **username_2**: Hello. Can you please provide the link to the PyTorch model that has an intermediate layer that you're referring to?  
  > **username_0**: Hello thank you for your response. in fact each pre-trained model on hugging face has the same architecture. you can look at the following for example 'bert-base-uncased' at https://huggingface.co/bert-base-uncased (three fully connected layers. one in attention one intermediate and one output. for all encoder blocks)  
  > **username_0**: I am confused by this third (intermediate dense layer) in between attention output and encoder output dense layers.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2021-10-26. Score: 5
  
  Feedforward layer is an important part of the transformer architecture.
  
  Transformer architecture, in addition to the `self-attention` layer, that aggregates information from the whole sequence and transforms each token due to the attention scores from the `queries` and `values` has a feedforward layer, which is mostly a 2-layer MLP, that processes each token separately:
  $$
  y = W\_2 \sigma(W\_1 x + b\_1) + b\_2
  $$
  
  Where $W\_1, W\_2$ are the weights, and $b\_1, b\_2$ - biases, $\sigma$ - is the nonlinearity `ReLU, GeLU`, e.t.c.
  
  [![enter image description here](https://i.sstatic.net/CUBEd.png)](https://i.sstatic.net/CUBEd.png)
  
  This is kind of a *pointwise* nonlinear transformation of the sequence.
  
  I suspect, that `intermediate` here corresponds to $W\_1, b\_1$ and the output is about $W\_2, b\_2$.
  
  > **Comments:**
  > **username_0**: Hi, thank you for your response. I understand that the attention sublayer in the encoder has a fully connected dense layer followed by another output fully connected dense layer. but all pre-trained transformers at hugging face have three fully connected layers in each encoder block. one in attention one intermediate and one output. just for confirmation, are you suggesting that the MLP within attention (before skip connection) has two layers one of which is being referred to as the intermediate layer?  thanx again  
  > **username_4**: The language "take a moment to think and process this information" is a seriously unhelpful and obfuscating way to describe a matrix multiplication. I would discourage people from unnecessary anthropomorphism of simple operations just because they are in the context of AI.  
  > **username_5**: @mohammadaliHumayun The multihead-attention block has four linear layers - $W_Q$, $W_K$, $W_V$ and $W_O$. The layer $W_O$ is used to combine the concatenated output from the multiple heads. It is called `attention.output.dense` in the model that you are using. After the attention block you have a multi-layer perceptron (MLP). As explained in the answer, the MLP usually has two layers $W_1$ and $W_2$. So $W_1$ is called `intermidiate.dense` and $W_2$ is called `output.dense`.  
  
  ---
  
  ### Answer by username_6 on 2023-08-02. Score: 0
  
  [documentaion of Bert on Hugging Face](https://huggingface.co/docs/transformers/main/en/model_doc/bert#transformers.BertForMaskedLM)
  
  you will find that
  
  > intermediate\_size (int, optional, defaults to 3072) — Dimensionality of the “intermediate” (often named feed-forward) layer in the Transformer encoder.
  
  so intermediate layer is feed-forward layer.
  
  
  ---
  
  ### Answer by username_7 on 2023-12-06. Score: 0
  
  So I came across this myself when trying to explore why models require intermediate layers in general. BERT is fundamentally different than the previous transformer architectures. The intermediate layer acts as a way to ensure the different multi attention heads are able to utilize pre-trained information. So for example if you examine the block directly you will notice they upscale and descale in my experience something like 768 dim to 3072 back down to 768, I am approximating dimensions from memory. This does not correspond to the KQV in the attention layer weights. Now it is still a point of research concern why need the scale and descale of the embedded information to allow for fine tuning of the models. When we remove the intermediate layers we have considerable drop in transfer learning performance and therefore pre-training experience is lost. You will also notice more recent models have this intermediate layer as opposed to the original transformer paper.
  
  
  ---
  
  