Answers.Body: <p>During inference, autoregressive models predicts text one token at a time sequentially. At each prediction step, attention mechanism takes query from only the current token and scores with all previous tokens' keys to compute a (softmax) weighted sum of the values associated with each of these previous tokens as output of the attention layer, and thus with runtime complexity of <span class="math-container">$O(T^2)$</span> for each layer where <span class="math-container">$T$</span> is the length of all the generated tokens in a sequence.</p>
  <p>The attention mechanism along with multiple attention heads are designed to use the entire sequence history to capture multiple aspects of dependencies, which enables it to understand and generate contextually relevant responses at both training and inference stages.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-04T02:04:53.427Z
Answers.LastActivityDate: 2024-11-04T06:53:04.673Z
Answers.LastEditDate: 2024-11-04T06:53:04.673Z
Answers.UserDisplayName: cinch
Body: <p>So while predicting the next word in autoregressive models(LLM) will the attention mechanism use queries from starting word or only previous word. Like for predicting after sentence &quot;I love&quot; attention mechanism takes query value for I and love and after predicting lets say as pizza, the next word attention mechanism does it take query value for I ,love ,pizza or only pizza.</p>
  
Tags: <natural-language-processing><transformer><large-language-models><decoder>
Title: Autoregressive Models(LLM) inference Prediction
UserDisplayName: adithya
ThreadText: # Autoregressive Models(LLM) inference Prediction
  
  (Asked by: username_0 on 2024-11-03)
  
  So while predicting the next word in autoregressive models(LLM) will the attention mechanism use queries from starting word or only previous word. Like for predicting after sentence "I love" attention mechanism takes query value for I and love and after predicting lets say as pizza, the next word attention mechanism does it take query value for I ,love ,pizza or only pizza.
  
  **Tags:** natural-language-processing, transformer, large-language-models, decoder
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-11-04. Score: 0 | Accepted answer
  
  During inference, autoregressive models predicts text one token at a time sequentially. At each prediction step, attention mechanism takes query from only the current token and scores with all previous tokens' keys to compute a (softmax) weighted sum of the values associated with each of these previous tokens as output of the attention layer, and thus with runtime complexity of $O(T^2)$ for each layer where $T$ is the length of all the generated tokens in a sequence.
  
  The attention mechanism along with multiple attention heads are designed to use the entire sequence history to capture multiple aspects of dependencies, which enables it to understand and generate contextually relevant responses at both training and inference stages.
  
  
  ---
  
  