<p>Transformer model of the original <a href="https://arxiv.org/abs/1706.03762" rel="nofollow noreferrer">Attention paper</a> has a decoder unit that works differently during Inference than Tranining.</p>
<p>I'm trying to understand the shapes used during decoder (both self-attention and enc-dec-attention blocks), but it's very confusing. I'm referring to <a href="https://towardsdatascience.com/transformers-explained-visually-part-1-overview-of-functionality-95a6dd460452" rel="nofollow noreferrer">this link</a> and also the original <a href="https://arxiv.org/abs/1706.03762" rel="nofollow noreferrer">Attention paper</a></p>
<p>In Inference, it uses all previous tokens generated until that time step (say <code>k</code>th time-step), as shown in the diagram below and explained at <a href="https://towardsdatascience.com/transformers-explained-visually-part-1-overview-of-functionality-95a6dd460452" rel="nofollow noreferrer">this link.</a></p>
<p><a href="https://i.sstatic.net/0Lt8b.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0Lt8b.png" alt="" /></a></p>
<p>Another diagram that shows self-attention and enc-dec-attention within decoder:</p>
<p><a href="https://i.sstatic.net/BbCxU.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/BbCxU.png" alt="enter image description here" /></a></p>
<p><strong>Question:</strong></p>
<p>However when I look at actual shapes of the QKV projection in the decoder self-attention, and feeding of the decoder self-attention output to the &quot;enc-dec-attention&quot;'s Q matrix, I see only 1 token from the output being used.</p>
<p>Let's <strong>assume 6 deocder blocks one after the other in the decoder stack</strong> (which is the base transformer model).</p>
<p>I'm very confused how the shapes for all matrices in the <strong>Decoder blocks after decoder-1</strong> of the decoder-stack (more specifically decoder-block-2 decoder-3, decoder-4..., decoder-6 of the decoder stack) self-attention and enc-dec-attention can match up with variable length of input to the decoder during inference. I looked at several online material but couldn't find answer.
I see only the BGemms in the decoder's self-attention (not enc-dec-attention) using the variable shapes until all previous <code>k</code> steps, but all other Gemms are fixed size.</p>
<ul>
<li>How is that possible? Is only 1 token (last one from decoder output) is being used for qkv matmuls in self-attention and Q-matmul in enc-dec-attention (which is what I see when running the model)?</li>
<li>Could someone elaborate how all these shapes for QKV in self-attention and Q in enc-dec-attention match up with decoder input length being different at each time-step?**</li>
</ul>

CC BY-SA 4.0
2021-11-09T16:13:21.173Z
Hello. Rather than saying that you're confused in the title, could you please put your **specific** question there? Thanks!
nbro
CC BY-SA 4.0
2021-11-10T02:59:49.603Z
Updated to specific question in the title per your comment.
Joe Black
CC BY-SA 4.0
2023-05-10T09:52:32.813Z
Please see [this](https://ai.stackexchange.com/questions/40179/how-does-the-decoder-only-transformer-architecture-work) (newer) post for an answer to your question
Robin van Hoorn
<transformer><attention><machine-translation><language-model><encoder-decoder>
What is input (and shape) to K/V/Q of self-attention of EACH Decoder block of Language-translation model Transformer's tokens during Inference?
Joe Black
# What is input (and shape) to K/V/Q of self-attention of EACH Decoder block of Language-translation model Transformer's tokens during Inference?

(Asked by: username_0 on 2021-11-09)

Transformer model of the original [Attention paper](https://arxiv.org/abs/1706.03762) has a decoder unit that works differently during Inference than Tranining.

I'm trying to understand the shapes used during decoder (both self-attention and enc-dec-attention blocks), but it's very confusing. I'm referring to [this link](https://towardsdatascience.com/transformers-explained-visually-part-1-overview-of-functionality-95a6dd460452) and also the original [Attention paper](https://arxiv.org/abs/1706.03762)

In Inference, it uses all previous tokens generated until that time step (say `k`th time-step), as shown in the diagram below and explained at [this link.](https://towardsdatascience.com/transformers-explained-visually-part-1-overview-of-functionality-95a6dd460452)

[![](https://i.sstatic.net/0Lt8b.png)](https://i.sstatic.net/0Lt8b.png)

Another diagram that shows self-attention and enc-dec-attention within decoder:

[![enter image description here](https://i.sstatic.net/BbCxU.png)](https://i.sstatic.net/BbCxU.png)

**Question:**

However when I look at actual shapes of the QKV projection in the decoder self-attention, and feeding of the decoder self-attention output to the "enc-dec-attention"'s Q matrix, I see only 1 token from the output being used.

Let's **assume 6 deocder blocks one after the other in the decoder stack** (which is the base transformer model).

I'm very confused how the shapes for all matrices in the **Decoder blocks after decoder-1** of the decoder-stack (more specifically decoder-block-2 decoder-3, decoder-4..., decoder-6 of the decoder stack) self-attention and enc-dec-attention can match up with variable length of input to the decoder during inference. I looked at several online material but couldn't find answer.
I see only the BGemms in the decoder's self-attention (not enc-dec-attention) using the variable shapes until all previous `k` steps, but all other Gemms are fixed size.

* How is that possible? Is only 1 token (last one from decoder output) is being used for qkv matmuls in self-attention and Q-matmul in enc-dec-attention (which is what I see when running the model)?
* Could someone elaborate how all these shapes for QKV in self-attention and Q in enc-dec-attention match up with decoder input length being different at each time-step?\*\*

**Tags:** transformer, attention, machine-translation, language-model, encoder-decoder

> **Comments:**
> **username_1**: Hello. Rather than saying that you're confused in the title, could you please put your **specific** question there? Thanks!  
> **username_0**: Updated to specific question in the title per your comment.  
> **username_2**: Please see [this](https://ai.stackexchange.com/questions/40179/how-does-the-decoder-only-transformer-architecture-work) (newer) post for an answer to your question  

---

## 0 answers

