Answers.Body: <p>Yes, if you input into a transformer <span class="math-container">$n$</span> input embeddings, you will receive <span class="math-container">$n$</span> output embeddings. But, when making predictions, the feed forward layer will only use one of those. For example, when a decoder model (e.g., GPT-2) predicts the next token, it will use the last embedding, and BERT has a special [CLS] token that is supposed to pool information from the entire input.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-26T18:11:35.830Z
Answers.Text: So in the multi-head attention connecting the encoder and the decoder, only one of the output embedding is used? But in the masked multi-head attention they are all used?
Answers.UserDisplayName: FluidMechanics Potential Flows
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-26T19:07:41.430Z
Answers.Text: No usually, only one token is used for each prediction for MLM as well. e.g., with BERT, the [CLS] token embedding is used (during pretraining) for the next sentence prediction. The "pooling" happens implicitly as a result of the pretraining task.
Answers.UserDisplayName: Alexander Wan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-27T09:27:38.200Z
Answers.Text: I'm sorry I'm getting confused because in the paper they explicitly say: "At each step the model is auto-regressive, consuming the previously generated symbols as additional input when generating the next". Symbols is plural so surely they use more than one token right?
Answers.UserDisplayName: FluidMechanics Potential Flows
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-27T15:35:46.097Z
Answers.Text: "At each step" means at each (next token) prediction. Each step can use a different token, but each prediction only uses a single token.
Answers.UserDisplayName: Alexander Wan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-26T16:49:40.377Z
Answers.LastActivityDate: 2024-01-26T16:49:40.377Z
Answers.UserDisplayName: Alexander Wan
Body: <p>I'm curious about something in the decoder part of the Transformers architecture. From what I understand, the Keys and Values come from the output of the encoder part of the Transformer. I understand that we have as many queries, keys and values as we have input tokens (actually if it is multiheaded attention we may have more (i.e. 10 heads means 10 queries for each word)).</p>
  <p>In the decoder part, we basically match (by doing a scalar product) a query that the output &quot;asks&quot; to those keys and values to know which part of the input the decoder should look at.</p>
  <p>However, if I'm not mistaken, there are as many queries as there are previously outputted output tokens. Therefore as many outputs of the multi-head attention as there are previously outputted output tokens.</p>
  <p>How / when does this dimensionality that depends on the current amount of output tokens disappear? It has to disappear since &quot;at the end&quot; there is a standard feed-forward network which size is fixed* and cannot depend on the number of output tokens.
  I'm talking about inference when one token at a time is generated but the transformer still inputs to the decoder <strong>all</strong> the previous generated output tokens. I'm not sure about that last part especially since it says &quot;Outputs (shifted right)&quot; in the paper. Maybe it only inputs a fixed number of previously outputted tokens? But that feels wrong.</p>
  <p>* or maybe each of those outputs is fed through the last feed forward network but then how is a word determined with those <span class="math-container">$n_{previously\_outputted\_tokens}$</span> outputs of this last feed forward network?</p>
  
Tags: <neural-networks><transformer><attention>
Title: Handling Variable Output Token Dimensionality in Transformer Decoders During Inference
UserDisplayName: FluidMechanics Potential Flows
ThreadText: # Handling Variable Output Token Dimensionality in Transformer Decoders During Inference
  
  (Asked by: username_0 on 2024-01-26)
  
  I'm curious about something in the decoder part of the Transformers architecture. From what I understand, the Keys and Values come from the output of the encoder part of the Transformer. I understand that we have as many queries, keys and values as we have input tokens (actually if it is multiheaded attention we may have more (i.e. 10 heads means 10 queries for each word)).
  
  In the decoder part, we basically match (by doing a scalar product) a query that the output "asks" to those keys and values to know which part of the input the decoder should look at.
  
  However, if I'm not mistaken, there are as many queries as there are previously outputted output tokens. Therefore as many outputs of the multi-head attention as there are previously outputted output tokens.
  
  How / when does this dimensionality that depends on the current amount of output tokens disappear? It has to disappear since "at the end" there is a standard feed-forward network which size is fixed\* and cannot depend on the number of output tokens.
  I'm talking about inference when one token at a time is generated but the transformer still inputs to the decoder **all** the previous generated output tokens. I'm not sure about that last part especially since it says "Outputs (shifted right)" in the paper. Maybe it only inputs a fixed number of previously outputted tokens? But that feels wrong.
  
  \* or maybe each of those outputs is fed through the last feed forward network but then how is a word determined with those $n\_{previously\\_outputted\\_tokens}$ outputs of this last feed forward network?
  
  **Tags:** neural-networks, transformer, attention
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-01-26. Score: 1
  
  Yes, if you input into a transformer $n$ input embeddings, you will receive $n$ output embeddings. But, when making predictions, the feed forward layer will only use one of those. For example, when a decoder model (e.g., GPT-2) predicts the next token, it will use the last embedding, and BERT has a special [CLS] token that is supposed to pool information from the entire input.
  
  > **Comments:**
  > **username_0**: So in the multi-head attention connecting the encoder and the decoder, only one of the output embedding is used? But in the masked multi-head attention they are all used?  
  > **username_1**: No usually, only one token is used for each prediction for MLM as well. e.g., with BERT, the [CLS] token embedding is used (during pretraining) for the next sentence prediction. The "pooling" happens implicitly as a result of the pretraining task.  
  > **username_0**: I'm sorry I'm getting confused because in the paper they explicitly say: "At each step the model is auto-regressive, consuming the previously generated symbols as additional input when generating the next". Symbols is plural so surely they use more than one token right?  
  > **username_1**: "At each step" means at each (next token) prediction. Each step can use a different token, but each prediction only uses a single token.  
  
  ---
  
  