Answers.Body: <blockquote>
  <p>The transformer architecture contains a cross attention mechanism which is
  enriching the encoder with information from the decoder. The place where this
  takes place is visualized in the image below:</p>
  </blockquote>
  <p>I think that you got it the other way round. The encoder passes an enriched input sentence to the decoder. Cross attention helps the decoder to attend to every part of this &quot;enriched&quot; input and produce one output at a time recursively.
  Initially, the decoder's first prediction is fixed to  (start of sentence) token. That gets self attended first, then get attended with encoder's output (the &quot;enriched&quot; input) and gives out a prediction from the word vocab list. This word gets appended to the decoder's input and we repeat the process again.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-20T09:16:58.250Z
Answers.LastActivityDate: 2022-12-20T09:16:58.250Z
Answers.UserDisplayName: p0p4k
Answers.Body: <p>The idea of the cross-attention layer is to transform the <strong>input words to output words</strong>. The Decoder provides context of which input words should we pay attention to next based on the already decoded text.</p>
  <p>If the values in the cross-attention came from the decoder as you suggest, we would have to have access to the already translated sentence to translate the sentence, which is absurd.</p>
  <p>For example, suppose we are going to translate the sentence &quot;में बैडमिंटन खेलने जा रहा हूँ &quot; in Hindi to English : &quot;I am going to play badminton&quot;.</p>
  <p>And suppose we are have already translated &quot;I am going to play _&quot; and are looking for the next word : &quot;badminton&quot;. In this case the decoder self-attention guesses that the next word is going to be a sports name. The query cross-attention linear layer now transforms the input vector to align with the corresponding vector representation in Hindi for sports names. Now amongst the input words the hindi representation of 'badminton' aligns the most with query vector and thus we know that what the next word to transform to English is.</p>
  <p>Now what is left, is to do the actual transformation from Hindi representation of badminton to the English representation. The value linear layer handles that part.</p>
  <p>The dot product is able to work because the vector representation of words follows a pattern as shown by the papers referred here : <a href="https://kawine.github.io/blog/nlp/2019/06/21/word-analogies.html" rel="nofollow noreferrer">https://kawine.github.io/blog/nlp/2019/06/21/word-analogies.html</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-16T05:39:25.900Z
Answers.LastActivityDate: 2023-09-16T06:12:40.580Z
Answers.LastEditDate: 2023-09-16T06:12:40.580Z
Answers.UserDisplayName: Abhishek
Answers.Body: <p>My intuition is that the decoder essentially performs a dictionary look-up where the encoder acts as the dictionary that provides keys and corresponding values. In this way, the decoder can &quot;ask&quot; for relevant information to fulfill its needs.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-21T08:32:48.510Z
Answers.LastActivityDate: 2022-12-21T08:32:48.510Z
Answers.UserDisplayName: entropy07
Body: <p>The transformer architecture contains a cross attention mechanism which is enriching the encoder with information from the decoder. The place where this takes place is visualized in the image below:</p>
  <p><a href="https://i.sstatic.net/L3ifH.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/L3ifH.jpg" alt="transformer_architecture" /></a></p>
  <p>The cross attention mechanism within the original transformer architecture is implemented in the following way:</p>
  <p><a href="https://i.sstatic.net/BBwHl.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/BBwHl.jpg" alt="cross_attention_computation" /></a></p>
  <p>The source for the images is <a href="https://www.youtube.com/watch?v=EixI6t5oif0" rel="nofollow noreferrer">this video</a>.
  Why are the values in this step coming from the encoder instead of from the decoder? Is this where e.g. the actual language translation happens within a transformer?</p>
  
Tags: <natural-language-processing><transformer><attention>
Title: Why do the values in the cross attentional mechanism within a transformer come from the encoder and not from the decoder?
UserDisplayName: Mariusmarten
ThreadText: # Why do the values in the cross attentional mechanism within a transformer come from the encoder and not from the decoder?
  
  (Asked by: username_0 on 2022-12-15)
  
  The transformer architecture contains a cross attention mechanism which is enriching the encoder with information from the decoder. The place where this takes place is visualized in the image below:
  
  [![transformer_architecture](https://i.sstatic.net/L3ifH.jpg)](https://i.sstatic.net/L3ifH.jpg)
  
  The cross attention mechanism within the original transformer architecture is implemented in the following way:
  
  [![cross_attention_computation](https://i.sstatic.net/BBwHl.jpg)](https://i.sstatic.net/BBwHl.jpg)
  
  The source for the images is [this video](https://www.youtube.com/watch?v=EixI6t5oif0).
  Why are the values in this step coming from the encoder instead of from the decoder? Is this where e.g. the actual language translation happens within a transformer?
  
  **Tags:** natural-language-processing, transformer, attention
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2022-12-20. Score: 2
  
  > The transformer architecture contains a cross attention mechanism which is
  > enriching the encoder with information from the decoder. The place where this
  > takes place is visualized in the image below:
  
  I think that you got it the other way round. The encoder passes an enriched input sentence to the decoder. Cross attention helps the decoder to attend to every part of this "enriched" input and produce one output at a time recursively.
  Initially, the decoder's first prediction is fixed to (start of sentence) token. That gets self attended first, then get attended with encoder's output (the "enriched" input) and gives out a prediction from the word vocab list. This word gets appended to the decoder's input and we repeat the process again.
  
  
  ---
  
  ### Answer by username_2 on 2023-09-16. Score: 2
  
  The idea of the cross-attention layer is to transform the **input words to output words**. The Decoder provides context of which input words should we pay attention to next based on the already decoded text.
  
  If the values in the cross-attention came from the decoder as you suggest, we would have to have access to the already translated sentence to translate the sentence, which is absurd.
  
  For example, suppose we are going to translate the sentence "में बैडमिंटन खेलने जा रहा हूँ " in Hindi to English : "I am going to play badminton".
  
  And suppose we are have already translated "I am going to play \_" and are looking for the next word : "badminton". In this case the decoder self-attention guesses that the next word is going to be a sports name. The query cross-attention linear layer now transforms the input vector to align with the corresponding vector representation in Hindi for sports names. Now amongst the input words the hindi representation of 'badminton' aligns the most with query vector and thus we know that what the next word to transform to English is.
  
  Now what is left, is to do the actual transformation from Hindi representation of badminton to the English representation. The value linear layer handles that part.
  
  The dot product is able to work because the vector representation of words follows a pattern as shown by the papers referred here : <https://kawine.github.io/blog/nlp/2019/06/21/word-analogies.html>
  
  
  ---
  
  ### Answer by username_3 on 2022-12-21. Score: 2
  
  My intuition is that the decoder essentially performs a dictionary look-up where the encoder acts as the dictionary that provides keys and corresponding values. In this way, the decoder can "ask" for relevant information to fulfill its needs.
  
  
  ---
  
  