Answers.Body: <p>I would suggest that for in-depth details on how stuff is done, stick with the researcher materials, so either the code or the paper itself.</p>
  <p>In particular, no details are given for this manner in the main paper, but the authors published the <a href="https://github.com/jadore801120/attention-is-all-you-need-pytorch/" rel="nofollow noreferrer">code</a></p>
  <p>In particular <a href="https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/transformer/Layers.py#L34" rel="nofollow noreferrer">here</a>, you can see that the encoder output is passed as both key and value.</p>
  <p>So, no, you cannot remove all that final part of the transformer</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-06T10:42:15.920Z
Answers.Text: Thanks for providing further material. Looking at the code really cleared that up for me. My mistake was that I thought we capture the k and v vectors from the encoder, but instead we calculate fresh k and v vectors from the encoder output using the decoder's Wk and Wv weights.
Answers.UserDisplayName: Cerno
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-06T14:55:44.500Z
Answers.Text: @Cerno no worries, i knew the code had the answer because i used to have the same doubt as your, and i found the answer there, good luck with your journey
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-05T18:04:15.623Z
Answers.LastActivityDate: 2024-09-06T14:56:02.933Z
Answers.LastEditDate: 2024-09-06T14:56:02.933Z
Answers.UserDisplayName: Alberto
Body: <p>I am trying to wrap my head around the transformer model using this great resource:</p>
  <p><a href="https://jalammar.github.io/illustrated-transformer/" rel="nofollow noreferrer">https://jalammar.github.io/illustrated-transformer/</a></p>
  <p>I may need some clarification.</p>
  <p>My understanding is that we take the key and value vectors from the last encoder block's self-attention layer and use them as input for the cross-attention layer of each decoder block.</p>
  <p>But if that is the case, then everything after the last encoder block's self-attention layer (i.e. the feed forward and normalization layers) is not necessary anymore, right?</p>
  <p>I assume they are in the illustrations and also implementations to keep things consistent, but for the sake of inference and training time, I think we could technically skip these layers, like this:</p>
  <p><a href="https://i.sstatic.net/ocW2H4A4.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ocW2H4A4.png" alt="transformer_adapted" /></a></p>
  <p>I am asking because the illustration makes it seem like we take the output of the encoder as input of the decoder, but that wouldn't work because the dimensions don't fit (key and value vectors have different length than the embedding vector). Or does the dashed line indicate general flow of information from the encoder as a black box?</p>
  <p>Can anyone confirm that these layers are technically not needed? And as a bonus question: Is this usually taken into account in frameworks like tensorflow to conserve training and inference time?</p>
  <p><strong>Update:</strong> As per @Alberto's answer, my initial premise was wrong. We do not take the key and value vectors from the last encoder's self-attention layer. Instead we take the last encoder's output and compute new key and value vectors in the decoder's cross-attention layer.</p>
  
Tags: <transformer>
Title: Is the feed forward layer in the last encoder block of a transformer unnecessary?
UserDisplayName: Cerno
ThreadText: # Is the feed forward layer in the last encoder block of a transformer unnecessary?
  
  (Asked by: username_0 on 2024-09-05)
  
  I am trying to wrap my head around the transformer model using this great resource:
  
  <https://jalammar.github.io/illustrated-transformer/>
  
  I may need some clarification.
  
  My understanding is that we take the key and value vectors from the last encoder block's self-attention layer and use them as input for the cross-attention layer of each decoder block.
  
  But if that is the case, then everything after the last encoder block's self-attention layer (i.e. the feed forward and normalization layers) is not necessary anymore, right?
  
  I assume they are in the illustrations and also implementations to keep things consistent, but for the sake of inference and training time, I think we could technically skip these layers, like this:
  
  [![transformer_adapted](https://i.sstatic.net/ocW2H4A4.png)](https://i.sstatic.net/ocW2H4A4.png)
  
  I am asking because the illustration makes it seem like we take the output of the encoder as input of the decoder, but that wouldn't work because the dimensions don't fit (key and value vectors have different length than the embedding vector). Or does the dashed line indicate general flow of information from the encoder as a black box?
  
  Can anyone confirm that these layers are technically not needed? And as a bonus question: Is this usually taken into account in frameworks like tensorflow to conserve training and inference time?
  
  **Update:** As per @Alberto's answer, my initial premise was wrong. We do not take the key and value vectors from the last encoder's self-attention layer. Instead we take the last encoder's output and compute new key and value vectors in the decoder's cross-attention layer.
  
  **Tags:** transformer
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-09-05. Score: 1 | Accepted answer
  
  I would suggest that for in-depth details on how stuff is done, stick with the researcher materials, so either the code or the paper itself.
  
  In particular, no details are given for this manner in the main paper, but the authors published the [code](https://github.com/jadore801120/attention-is-all-you-need-pytorch/)
  
  In particular [here](https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/transformer/Layers.py#L34), you can see that the encoder output is passed as both key and value.
  
  So, no, you cannot remove all that final part of the transformer
  
  > **Comments:**
  > **username_0**: Thanks for providing further material. Looking at the code really cleared that up for me. My mistake was that I thought we capture the k and v vectors from the encoder, but instead we calculate fresh k and v vectors from the encoder output using the decoder's Wk and Wv weights.  
  > **username_1**: @Cerno no worries, i knew the code had the answer because i used to have the same doubt as your, and i found the answer there, good luck with your journey  
  
  ---
  
  