Answers.Body: <p>(This type of) autoregressive LLM always works by predicting <em>one</em> next token based on a series of previous tokens. First you run the model with input &quot;today is a&quot; and the prediction is &quot;good&quot;. Then you run the model with input &quot;today is a good&quot; and the prediction is &quot;day&quot;, and so on. Each token is predicted by running the entire model from start to finish on its previous input.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-09T12:52:09.063Z
Answers.LastActivityDate: 2023-05-09T12:52:09.063Z
Answers.UserDisplayName: Criticize SE actions means ban
Body: <p>For a transformer decoder, how exactly are K, Q, and V for each decoding step?</p>
  <p>Assume my input prompt is &quot;today is a&quot; (good day).</p>
  <p>At t= 0 (generation step 0):
  K, Q, and V are the projections of the sequence (&quot;today is a&quot;)
  Then say the next token generated is &quot;good&quot;.</p>
  <p>At <code>t=1</code> (generation step 1):
  Which one is true:</p>
  <ol>
  <li>K, Q, and V are the projections of the sequence (&quot;today is a good&quot;)</li>
  <li>K, Q, are the projections of the sequence (&quot;today is a&quot;), and V is the projection of the sequence (&quot;good&quot;)?</li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-09T07:03:57.083Z
Comments.Text: Please see if [the explanation](https://ai.stackexchange.com/questions/40179/how-does-the-decoder-only-transformer-architecture-work) here helps you in any way.
  In short, during the forward pass, K, Q, and V are projections of the input sequence ("today is a"). The output of the forward pass is "good".
Comments.UserDisplayName: Robin van Hoorn
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-10T02:33:58.030Z
Comments.Text: @RobinvanHoorn, I see, and thanks. My question is the next next time step (t+2), so now your entire sequence is "today is a good", do you just repeat the same thing (with K, Q, V as the projections of the "today is a good" sequence?)
Comments.UserDisplayName: wrek
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-10T09:50:20.783Z
Comments.Text: your question was (and the question still only mentions) `t=1`. If you again want to predict a next word, then K,Q and V are again projections of the input sequence ("today is a good") and then the output becomes (i.e.) "day".
Comments.UserDisplayName: Robin van Hoorn
Tags: <transformer><autoencoders><large-language-models><encoder-decoder>
Title: For a transformer decoder, how exactly are K, Q, and V for each decoding step?
UserDisplayName: wrek
ThreadText: # For a transformer decoder, how exactly are K, Q, and V for each decoding step?
  
  (Asked by: username_0 on 2023-05-09)
  
  For a transformer decoder, how exactly are K, Q, and V for each decoding step?
  
  Assume my input prompt is "today is a" (good day).
  
  At t= 0 (generation step 0):
  K, Q, and V are the projections of the sequence ("today is a")
  Then say the next token generated is "good".
  
  At `t=1` (generation step 1):
  Which one is true:
  
  1. K, Q, and V are the projections of the sequence ("today is a good")
  2. K, Q, are the projections of the sequence ("today is a"), and V is the projection of the sequence ("good")?
  
  **Tags:** transformer, autoencoders, large-language-models, encoder-decoder
  > **Comments:**
  > **username_1**: Please see if [the explanation](https://ai.stackexchange.com/questions/40179/how-does-the-decoder-only-transformer-architecture-work) here helps you in any way. In short, during the forward pass, K, Q, and V are projections of the input sequence ("today is a"). The output of the forward pass is "good".  
  > **username_0**: @RobinvanHoorn, I see, and thanks. My question is the next next time step (t+2), so now your entire sequence is "today is a good", do you just repeat the same thing (with K, Q, V as the projections of the "today is a good" sequence?)  
  > **username_1**: your question was (and the question still only mentions) `t=1`. If you again want to predict a next word, then K,Q and V are again projections of the input sequence ("today is a good") and then the output becomes (i.e.) "day".  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-05-09. Score: 2 | Accepted answer
  
  (This type of) autoregressive LLM always works by predicting *one* next token based on a series of previous tokens. First you run the model with input "today is a" and the prediction is "good". Then you run the model with input "today is a good" and the prediction is "day", and so on. Each token is predicted by running the entire model from start to finish on its previous input.
  
  
  ---
  
  