Answers.Body: <p>From <a href="https://arxiv.org/abs/2211.05102" rel="nofollow noreferrer">Efficiently Scaling Transformer Inference</a>:</p>
  <blockquote>
  <p>Each example ... has L_input tokens of input text, and generates L_gen
  tokens of output text. ... Since prefill can run in parallel over
  L_input, but decode must run sequentially over L_gen, the two phases
  have different performance characteristics and we analyze them
  separately.</p>
  </blockquote>
  <p>This means the KV cache can be filled fully in parallel for all L_input tokens. In contrast, in the decoding phase the output has to be computed autoregressively, i.e. one output token after the other because the current output token is dependent on the last output token.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-04T16:28:39.257Z
Answers.LastActivityDate: 2023-10-04T16:28:39.257Z
Answers.UserDisplayName: gizzmole
Body: <p>I've seen conflicting information about this online so I'm looking for clarification. I'm dealing with the causal LLaMAF model specifically.</p>
  <p>I used to think that a sequence of tokens is generated in, and a sequence of probabilities for the next token in the sequence is generated as output. This generated output token is then appended to the sequence of tokens, and fed in the model again.</p>
  <p>However, I know understand that there's a key value cache that's generated for each sequence of tokens fed in. This key value cache stores a precomputed matrix that can be used in future computations to prevent having to recompute previously seen tokens.</p>
  <p>So, the new workflow is feed in token list -&gt; generate next token and key value cache -&gt; feed in next token -&gt; generate next token and key value cache for current token</p>
  <p>What is considered the pre-fill, and what is considered the decoding phase in this process? Does the prefill phase involve feeding through the token list one by one passes to generate the kv cache? Why does the prefill phase take significantly longer?</p>
  
Tags: <machine-learning><deep-learning><natural-language-processing><transformer><large-language-models>
Title: What is considered the pre-fill, and what is considered the decoding phase in this process?
UserDisplayName: jgeddes
ThreadText: # What is considered the pre-fill, and what is considered the decoding phase in this process?
  
  (Asked by: username_0 on 2023-07-12)
  
  I've seen conflicting information about this online so I'm looking for clarification. I'm dealing with the causal LLaMAF model specifically.
  
  I used to think that a sequence of tokens is generated in, and a sequence of probabilities for the next token in the sequence is generated as output. This generated output token is then appended to the sequence of tokens, and fed in the model again.
  
  However, I know understand that there's a key value cache that's generated for each sequence of tokens fed in. This key value cache stores a precomputed matrix that can be used in future computations to prevent having to recompute previously seen tokens.
  
  So, the new workflow is feed in token list -> generate next token and key value cache -> feed in next token -> generate next token and key value cache for current token
  
  What is considered the pre-fill, and what is considered the decoding phase in this process? Does the prefill phase involve feeding through the token list one by one passes to generate the kv cache? Why does the prefill phase take significantly longer?
  
  **Tags:** machine-learning, deep-learning, natural-language-processing, transformer, large-language-models
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-10-04. Score: 0
  
  From [Efficiently Scaling Transformer Inference](https://arxiv.org/abs/2211.05102):
  
  > Each example ... has L\_input tokens of input text, and generates L\_gen
  > tokens of output text. ... Since prefill can run in parallel over
  > L\_input, but decode must run sequentially over L\_gen, the two phases
  > have different performance characteristics and we analyze them
  > separately.
  
  This means the KV cache can be filled fully in parallel for all L\_input tokens. In contrast, in the decoding phase the output has to be computed autoregressively, i.e. one output token after the other because the current output token is dependent on the last output token.
  
  
  ---
  
  