Answers.Body: <p>Apperently KV Caching is a thing, which means you only need to run the transformer on the next token you are infering</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-16T08:30:01.093Z
Answers.LastActivityDate: 2024-09-16T08:30:01.093Z
Answers.UserDisplayName: user2741831
Body: <p>We have a transformer with a single layer, a single head, an embedding dimension of 8 and a vocab size of 10. That makes W_k,W_v,W_q and W_ff all 8x8 and the out_ff 8x10. Now we run inference twice on the input &quot;roses are&quot;. For the first inference we have to run both &quot;roses&quot; and &quot;are&quot; through all of the W_* matrices (and of course, perform self-attention).</p>
  <p>But I was wondering, since during training, the transformer used masked self attention (meaning words are only every influenced by previous word and never following words), is masked self attention also used during inference?</p>
  <p>And if so, after we have generated the next words (presumably &quot;red&quot;), can we cache and reuse the hidden states of &quot;roses&quot; and &quot;are&quot; now? Since they wouldn't change if we used masked-self-attention? Or do we have to run all words (&quot;roses&quot;,&quot;are&quot; and &quot;red&quot;) through all W_* matrices again, to gain new hidden states?</p>
  
Tags: <transformer>
Title: How often do you have to recalculate the hidden states of each token while running inference?
UserDisplayName: user2741831
ThreadText: # How often do you have to recalculate the hidden states of each token while running inference?
  
  (Asked by: username_0 on 2024-09-15)
  
  We have a transformer with a single layer, a single head, an embedding dimension of 8 and a vocab size of 10. That makes W\_k,W\_v,W\_q and W\_ff all 8x8 and the out\_ff 8x10. Now we run inference twice on the input "roses are". For the first inference we have to run both "roses" and "are" through all of the W\_\* matrices (and of course, perform self-attention).
  
  But I was wondering, since during training, the transformer used masked self attention (meaning words are only every influenced by previous word and never following words), is masked self attention also used during inference?
  
  And if so, after we have generated the next words (presumably "red"), can we cache and reuse the hidden states of "roses" and "are" now? Since they wouldn't change if we used masked-self-attention? Or do we have to run all words ("roses","are" and "red") through all W\_\* matrices again, to gain new hidden states?
  
  **Tags:** transformer
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2024-09-16. Score: 1
  
  Apperently KV Caching is a thing, which means you only need to run the transformer on the next token you are infering
  
  
  ---
  
  