Answers.Body: <p>This number is a little old and not some magic, both the latest <code>chatgpt-4o-latest</code> and <code>gpt-4o-mini</code> models have higher output token number as <span class="math-container">$16,384$</span> according to <a href="https://platform.openai.com/docs/models/gpt-3-5" rel="nofollow noreferrer">OpenAI API</a>.</p>
  <p>In general since for self-attention mechanism each position of the output sequence must attend to all other positions, it requires <span class="math-container">$O(n^2)$</span> memory for sequence length <span class="math-container">$n$</span>. This means <span class="math-container">$4096^2 = 16,777,216$</span> attention matrix elements for the said cap which already required a lot of memory and acted as some heuristic threshold for many old GPT models with their corresponding hardware resources. Also <span class="math-container">$4096$</span> tokens are close to human readability and comprehension limits and most real-world use cases don't need longer outputs whose quality may tend to degrade with very long generations</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-02T23:57:01.203Z
Answers.LastActivityDate: 2024-12-02T23:57:01.203Z
Answers.UserDisplayName: cinch
Body: <p>LLMs' maximum amount of output tokens is usually capped to 4096. Is there a reason for this?</p>
  
Tags: <large-language-models><tokenization>
Title: Why the maximum amount of output tokens is usually 4096?
UserDisplayName: Hephaestus
ThreadText: # Why the maximum amount of output tokens is usually 4096?
  
  (Asked by: username_0 on 2024-12-02)
  
  LLMs' maximum amount of output tokens is usually capped to 4096. Is there a reason for this?
  
  **Tags:** large-language-models, tokenization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-12-02. Score: 0 | Accepted answer
  
  This number is a little old and not some magic, both the latest `chatgpt-4o-latest` and `gpt-4o-mini` models have higher output token number as $16,384$ according to [OpenAI API](https://platform.openai.com/docs/models/gpt-3-5).
  
  In general since for self-attention mechanism each position of the output sequence must attend to all other positions, it requires $O(n^2)$ memory for sequence length $n$. This means $4096^2 = 16,777,216$ attention matrix elements for the said cap which already required a lot of memory and acted as some heuristic threshold for many old GPT models with their corresponding hardware resources. Also $4096$ tokens are close to human readability and comprehension limits and most real-world use cases don't need longer outputs whose quality may tend to degrade with very long generations
  
  
  ---
  
  