Answers.Body: <p>The decoder uses the output of only the final encoder layer for cross attention. The first diagram is correct.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-20T15:04:56.173Z
Answers.LastActivityDate: 2023-04-20T15:05:35.997Z
Answers.LastEditDate: 2023-04-20T15:05:35.997Z
Answers.UserDisplayName: Rohit Saxena
Body: <p>After looking into transformer-based models that used multiple stacked encoders and decoders, I am trying to understand how cross attention in the decoders work. In a transformer with a single encoder/decoder, my understanding is the queries come from the decoder, while the keys and values come from the output of the encoder.</p>
  <p>But when multiple stacked encoders are used, do the decoders only attend to the output of the final encoder layer? Or do they attend to all the encoder layers?</p>
  <p>To put it visually, does cross attention look like this?</p>
  <p><a href="https://i.sstatic.net/ZYdzv.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ZYdzv.png" alt="Attends hidden state of final encoder layer" /></a></p>
  <p>Or like this?</p>
  <p><a href="https://i.sstatic.net/77WM3.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/77WM3.png" alt="Attends all encoder layers" /></a></p>
  
Tags: <neural-networks><machine-learning><transformer>
Title: When multiple stacked encoders are used, do the decoders only attend to the output of the final encoder layer?
UserDisplayName: ndeily
ThreadText: # When multiple stacked encoders are used, do the decoders only attend to the output of the final encoder layer?
  
  (Asked by: username_0 on 2023-02-06)
  
  After looking into transformer-based models that used multiple stacked encoders and decoders, I am trying to understand how cross attention in the decoders work. In a transformer with a single encoder/decoder, my understanding is the queries come from the decoder, while the keys and values come from the output of the encoder.
  
  But when multiple stacked encoders are used, do the decoders only attend to the output of the final encoder layer? Or do they attend to all the encoder layers?
  
  To put it visually, does cross attention look like this?
  
  [![Attends hidden state of final encoder layer](https://i.sstatic.net/ZYdzv.png)](https://i.sstatic.net/ZYdzv.png)
  
  Or like this?
  
  [![Attends all encoder layers](https://i.sstatic.net/77WM3.png)](https://i.sstatic.net/77WM3.png)
  
  **Tags:** neural-networks, machine-learning, transformer
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-04-20. Score: 0 | Accepted answer
  
  The decoder uses the output of only the final encoder layer for cross attention. The first diagram is correct.
  
  
  ---
  
  