<p>Most state of the art LLMs nowadays are based on an architecture called <em>Transformer</em>, which uses a technique called <em>attention</em>, which scales quadratically with the input size. Thus, for an input of 1 million tokens, you'll have to create a matrix of size <span class="math-container">$(10^6)^2 = 10^{12}$</span> numbers, which are usually <code>float32</code>, which are composed of 4 bytes each. In other words, a matrix that is 4 Terabytes in size.</p>
<p>Alternatives have been proposed. For example linear transformers, sparse attention, RMTs, and quantization, but they are not as effective as the full transformer model.</p>

CC BY-SA 4.0
2023-12-07T22:54:08.607Z
2023-12-08T18:49:24.853Z
2023-12-08T18:49:24.853Z
Alberto
<p>Is there a limitation in current large language models (LLMs) in terms of practical processing time or memory resources when it comes to digesting the context provided by users? What I mean regarding context is the data fed to LLMs post-training, like feeding a selection of books about a certain topic.</p>

<large-language-models><gpt>
What is the current limit for LLMs regarding the size of the context that can be fed to them?
Miki
# What is the current limit for LLMs regarding the size of the context that can be fed to them?

(Asked by: username_0 on 2023-12-07)

Is there a limitation in current large language models (LLMs) in terms of practical processing time or memory resources when it comes to digesting the context provided by users? What I mean regarding context is the data fed to LLMs post-training, like feeding a selection of books about a certain topic.

**Tags:** large-language-models, gpt


---

## 1 answers

### Answer by username_1 on 2023-12-07. Score: 3

Most state of the art LLMs nowadays are based on an architecture called *Transformer*, which uses a technique called *attention*, which scales quadratically with the input size. Thus, for an input of 1 million tokens, you'll have to create a matrix of size $(10^6)^2 = 10^{12}$ numbers, which are usually `float32`, which are composed of 4 bytes each. In other words, a matrix that is 4 Terabytes in size.

Alternatives have been proposed. For example linear transformers, sparse attention, RMTs, and quantization, but they are not as effective as the full transformer model.


---

