Answers.Body: <p><strong>Context length is fixed</strong></p>
  <p>128K refers to Llama 3.1's ability to take up to 128,000 tokens in a single forward pass. This limit is fixed during training.</p>
  <p><strong>Inference on local hardware</strong></p>
  <p>When running inference locally (assuming your question isn't about hardware for training), your hardware significantly influences the speed of the model's performance but doesn't affect the fixed context length. There are optimization techniques, such as offloading model layers between RAM and VRAM and using quantization to reduce floating-point precision (which may impact response quality).</p>
  <p><strong>But why &quot;128&quot;?</strong></p>
  <p>According to the <a href="https://arxiv.org/pdf/2407.21783" rel="nofollow noreferrer">to the paper</a>:</p>
  <blockquote>
  <p>We increase the supported context length in increments, pre-training until the model has successfully adapted to the increased context length. We assess successful adaptation by measuring whether (1) model performance on short-context evaluations has recovered completely and (2) the model perfectly solves &quot;needle in a haystack&quot; tasks up to that length. In Llama 3 405B pre-training, we increased context length gradually in six stages, starting from the original 8K context window and ending in the final 128K context window. This long-context pre-training stage was performed using approximately 800B training tokens.</p>
  </blockquote>
  <p>The <a href="https://www.youtube.com/watch?v=KwRRuiCCdmc" rel="nofollow noreferrer">&quot;needle in a haystack&quot;</a> test involves hiding a key fact within a large amount of information. The model is then asked a question that requires that specific fact to answer correctly. This tests the model's ability to recover that small fact in a such a large context, it would catch if it's answering solely on pre-existing knowledge within it's weights for instance (and not using the context effectively).</p>
  <p>It appears that the &quot;128k&quot; was chosen based on the quality of the response instead of on hardware specifications.</p>
  <p>P.S.: It's a bit odd to me that they tried &quot;six stages&quot; from 8K and 128K instead of five (8K, 16K, 32K, 64K, 128K).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-18T19:55:17.750Z
Answers.Text: _To increase the context length the model would need to be retrained._ have I skipped a lecture on how a transformer works?
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-18T23:15:36.137Z
Answers.Text: Hi @Alberto, I've removed the sentence because it might be indeed misleading, thank you for the comment. My understanding is that if you increase the context length ideally you have to retraining/fine tuning it, to better generalize with such bigger context length. I confess I never tested it myself, so I wonder how poorer it will be.
Answers.UserDisplayName: talles
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-19T08:51:27.937Z
Answers.Text: Thanks, so do you mean that in inference time if my hardware allows, i can put as many as tokens that i want and pass the 128k limit but it will perform poorly or it gives an error ?
Answers.UserDisplayName: Parsa Foroozmand
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-19T13:27:05.673Z
Answers.Text: @ParsaForoozmand the architecture has no limitation on the input length, however they are still ML models, thus trained on a distribution of data (in this case, the max-length of the sentences), thus even though it allows it, it might have very poor performances
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-18T15:13:17.680Z
Answers.LastActivityDate: 2024-09-18T23:12:28.217Z
Answers.LastEditDate: 2024-09-18T23:12:28.217Z
Answers.UserDisplayName: talles
Body: <p>I'm trying to understand how context length is calculated in transformer models, especially in NLP tasks using open-source models like llama, which can be run locally. I'm confused about how this is defined and whether hardware specifications (like GPU/CPU memory) affect the context length.</p>
  <p>From what I understand, the maximum context length might be limited by memory, but I don't often see hardware specifications mentioned when discussing context length limits. For models that can be run locally, how exactly does hardware influence this, and why isn't it typically part of the context length discussion when dealing with transformer models?
  for example when they say llama 3.1 context length is 128k, based on what requirements they say that ?</p>
  
Tags: <natural-language-processing><transformer><large-language-models>
Title: How is context length calculated in transformers, and why isn't hardware specification considered?
UserDisplayName: Parsa Foroozmand
ThreadText: # How is context length calculated in transformers, and why isn't hardware specification considered?
  
  (Asked by: username_0 on 2024-09-18)
  
  I'm trying to understand how context length is calculated in transformer models, especially in NLP tasks using open-source models like llama, which can be run locally. I'm confused about how this is defined and whether hardware specifications (like GPU/CPU memory) affect the context length.
  
  From what I understand, the maximum context length might be limited by memory, but I don't often see hardware specifications mentioned when discussing context length limits. For models that can be run locally, how exactly does hardware influence this, and why isn't it typically part of the context length discussion when dealing with transformer models?
  for example when they say llama 3.1 context length is 128k, based on what requirements they say that ?
  
  **Tags:** natural-language-processing, transformer, large-language-models
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-09-18. Score: 1 | Accepted answer
  
  **Context length is fixed**
  
  128K refers to Llama 3.1's ability to take up to 128,000 tokens in a single forward pass. This limit is fixed during training.
  
  **Inference on local hardware**
  
  When running inference locally (assuming your question isn't about hardware for training), your hardware significantly influences the speed of the model's performance but doesn't affect the fixed context length. There are optimization techniques, such as offloading model layers between RAM and VRAM and using quantization to reduce floating-point precision (which may impact response quality).
  
  **But why "128"?**
  
  According to the [to the paper](https://arxiv.org/pdf/2407.21783):
  
  > We increase the supported context length in increments, pre-training until the model has successfully adapted to the increased context length. We assess successful adaptation by measuring whether (1) model performance on short-context evaluations has recovered completely and (2) the model perfectly solves "needle in a haystack" tasks up to that length. In Llama 3 405B pre-training, we increased context length gradually in six stages, starting from the original 8K context window and ending in the final 128K context window. This long-context pre-training stage was performed using approximately 800B training tokens.
  
  The ["needle in a haystack"](https://www.youtube.com/watch?v=KwRRuiCCdmc) test involves hiding a key fact within a large amount of information. The model is then asked a question that requires that specific fact to answer correctly. This tests the model's ability to recover that small fact in a such a large context, it would catch if it's answering solely on pre-existing knowledge within it's weights for instance (and not using the context effectively).
  
  It appears that the "128k" was chosen based on the quality of the response instead of on hardware specifications.
  
  P.S.: It's a bit odd to me that they tried "six stages" from 8K and 128K instead of five (8K, 16K, 32K, 64K, 128K).
  
  > **Comments:**
  > **username_2**: _To increase the context length the model would need to be retrained._ have I skipped a lecture on how a transformer works?  
  > **username_1**: Hi @Alberto, I've removed the sentence because it might be indeed misleading, thank you for the comment. My understanding is that if you increase the context length ideally you have to retraining/fine tuning it, to better generalize with such bigger context length. I confess I never tested it myself, so I wonder how poorer it will be.  
  > **username_0**: Thanks, so do you mean that in inference time if my hardware allows, i can put as many as tokens that i want and pass the 128k limit but it will perform poorly or it gives an error ?  
  > **username_2**: @ParsaForoozmand the architecture has no limitation on the input length, however they are still ML models, thus trained on a distribution of data (in this case, the max-length of the sentences), thus even though it allows it, it might have very poor performances  
  
  ---
  
  