Answers.Body: <p>Most state of the art LLMs nowadays are based on an architecture called <em>Transformer</em>, which uses a technique called <em>attention</em>, which scales quadratically with the input size. Thus, for an input of 1 million tokens, you'll have to create a matrix of size <span class="math-container">$(10^6)^2 = 10^{12}$</span> numbers, which are usually <code>float32</code>, which are composed of 4 bytes each. In other words, a matrix that is 4 Terabytes in size.</p>
  <p>Alternatives have been proposed. For example linear transformers, sparse attention, RMTs, and quantization, but they are not as effective as the full transformer model.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-07T22:54:08.607Z
Answers.LastActivityDate: 2023-12-08T18:49:24.853Z
Answers.LastEditDate: 2023-12-08T18:49:24.853Z
Answers.UserDisplayName: Alberto
Body: <p>Is there a limitation in current large language models (LLMs) in terms of practical processing time or memory resources when it comes to digesting the context provided by users? What I mean regarding context is the data fed to LLMs post-training, like feeding a selection of books about a certain topic.</p>
  
Tags: <large-language-models><gpt>
Title: What is the current limit for LLMs regarding the size of the context that can be fed to them?
UserDisplayName: Miki
ThreadText: # What is the current limit for LLMs regarding the size of the context that can be fed to them?
  
  (Asked by: username_0 on 2023-12-07)
  
  Is there a limitation in current large language models (LLMs) in terms of practical processing time or memory resources when it comes to digesting the context provided by users? What I mean regarding context is the data fed to LLMs post-training, like feeding a selection of books about a certain topic.
  
  **Tags:** large-language-models, gpt
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-12-07. Score: 3
  
  Most state of the art LLMs nowadays are based on an architecture called *Transformer*, which uses a technique called *attention*, which scales quadratically with the input size. Thus, for an input of 1 million tokens, you'll have to create a matrix of size $(10^6)^2 = 10^{12}$ numbers, which are usually `float32`, which are composed of 4 bytes each. In other words, a matrix that is 4 Terabytes in size.
  
  Alternatives have been proposed. For example linear transformers, sparse attention, RMTs, and quantization, but they are not as effective as the full transformer model.
  
  
  ---
  
  