Body: <p>Transformer model of the original <a href="https://arxiv.org/abs/1706.03762" rel="nofollow noreferrer">Attention paper</a> has a decoder unit that works differently during Inference than Tranining.</p>
  <p>I'm trying to understand the shapes used during decoder (both self-attention and enc-dec-attention blocks), but it's very confusing. I'm referring to <a href="https://towardsdatascience.com/transformers-explained-visually-part-1-overview-of-functionality-95a6dd460452" rel="nofollow noreferrer">this link</a> and also the original <a href="https://arxiv.org/abs/1706.03762" rel="nofollow noreferrer">Attention paper</a></p>
  <p>In Inference, it uses all previous tokens generated until that time step (say <code>k</code>th time-step), as shown in the diagram below and explained at <a href="https://towardsdatascience.com/transformers-explained-visually-part-1-overview-of-functionality-95a6dd460452" rel="nofollow noreferrer">this link.</a></p>
  <p><a href="https://i.sstatic.net/0Lt8b.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0Lt8b.png" alt="" /></a></p>
  <p>Another diagram that shows self-attention and enc-dec-attention within decoder:</p>
  <p><a href="https://i.sstatic.net/BbCxU.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/BbCxU.png" alt="enter image description here" /></a></p>
  <p><strong>Question:</strong></p>
  <p>However when I look at actual shapes of the QKV projection in the decoder self-attention, and feeding of the decoder self-attention output to the &quot;enc-dec-attention&quot;'s Q matrix, I see only 1 token from the output being used.</p>
  <p>Let's <strong>assume 6 deocder blocks one after the other in the decoder stack</strong> (which is the base transformer model).</p>
  <p>I'm very confused how the shapes for all matrices in the <strong>Decoder blocks after decoder-1</strong> of the decoder-stack (more specifically decoder-block-2 decoder-3, decoder-4..., decoder-6 of the decoder stack) self-attention and enc-dec-attention can match up with variable length of input to the decoder during inference. I looked at several online material but couldn't find answer.
  I see only the BGemms in the decoder's self-attention (not enc-dec-attention) using the variable shapes until all previous <code>k</code> steps, but all other Gemms are fixed size.</p>
  <ul>
  <li>How is that possible? Is only 1 token (last one from decoder output) is being used for qkv matmuls in self-attention and Q-matmul in enc-dec-attention (which is what I see when running the model)?</li>
  <li>Could someone elaborate how all these shapes for QKV in self-attention and Q in enc-dec-attention match up with decoder input length being different at each time-step?**</li>
  </ul>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-09T16:13:21.173Z
Comments.Text: Hello. Rather than saying that you're confused in the title, could you please put your **specific** question there? Thanks!
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-10T02:59:49.603Z
Comments.Text: Updated to specific question in the title per your comment.
Comments.UserDisplayName: Joe Black
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-10T09:52:32.813Z
Comments.Text: Please see [this](https://ai.stackexchange.com/questions/40179/how-does-the-decoder-only-transformer-architecture-work) (newer) post for an answer to your question
Comments.UserDisplayName: Robin van Hoorn
Tags: <transformer><attention><machine-translation><language-model><encoder-decoder>
Title: What is input (and shape) to K/V/Q of self-attention of EACH Decoder block of Language-translation model Transformer's tokens during Inference?
UserDisplayName: Joe Black
ThreadText: # What is input (and shape) to K/V/Q of self-attention of EACH Decoder block of Language-translation model Transformer's tokens during Inference?
  
  (Asked by: username_0 on 2021-11-09)
  
  Transformer model of the original [Attention paper](https://arxiv.org/abs/1706.03762) has a decoder unit that works differently during Inference than Tranining.
  
  I'm trying to understand the shapes used during decoder (both self-attention and enc-dec-attention blocks), but it's very confusing. I'm referring to [this link](https://towardsdatascience.com/transformers-explained-visually-part-1-overview-of-functionality-95a6dd460452) and also the original [Attention paper](https://arxiv.org/abs/1706.03762)
  
  In Inference, it uses all previous tokens generated until that time step (say `k`th time-step), as shown in the diagram below and explained at [this link.](https://towardsdatascience.com/transformers-explained-visually-part-1-overview-of-functionality-95a6dd460452)
  
  [![](https://i.sstatic.net/0Lt8b.png)](https://i.sstatic.net/0Lt8b.png)
  
  Another diagram that shows self-attention and enc-dec-attention within decoder:
  
  [![enter image description here](https://i.sstatic.net/BbCxU.png)](https://i.sstatic.net/BbCxU.png)
  
  **Question:**
  
  However when I look at actual shapes of the QKV projection in the decoder self-attention, and feeding of the decoder self-attention output to the "enc-dec-attention"'s Q matrix, I see only 1 token from the output being used.
  
  Let's **assume 6 deocder blocks one after the other in the decoder stack** (which is the base transformer model).
  
  I'm very confused how the shapes for all matrices in the **Decoder blocks after decoder-1** of the decoder-stack (more specifically decoder-block-2 decoder-3, decoder-4..., decoder-6 of the decoder stack) self-attention and enc-dec-attention can match up with variable length of input to the decoder during inference. I looked at several online material but couldn't find answer.
  I see only the BGemms in the decoder's self-attention (not enc-dec-attention) using the variable shapes until all previous `k` steps, but all other Gemms are fixed size.
  
  * How is that possible? Is only 1 token (last one from decoder output) is being used for qkv matmuls in self-attention and Q-matmul in enc-dec-attention (which is what I see when running the model)?
  * Could someone elaborate how all these shapes for QKV in self-attention and Q in enc-dec-attention match up with decoder input length being different at each time-step?\*\*
  
  **Tags:** transformer, attention, machine-translation, language-model, encoder-decoder
  > **Comments:**
  > **username_1**: Hello. Rather than saying that you're confused in the title, could you please put your **specific** question there? Thanks!  
  > **username_0**: Updated to specific question in the title per your comment.  
  > **username_2**: Please see [this](https://ai.stackexchange.com/questions/40179/how-does-the-decoder-only-transformer-architecture-work) (newer) post for an answer to your question  
  
  ---
  
  ## 0 answers
  
  