Answers.Body: <p>Yes, you have the right idea. There's been a lot of work recently regarding extending the context-length of existing models, mostly looking at the Llama family of models.</p>
  <p>You should check out <a href="https://lmsys.org/blog/2023-06-29-longchat/" rel="nofollow noreferrer">this post/paper</a> on LongChat by the team behind the Vicuna models (LMSYS). They extend the pretrained Llama models (with 2048-token contexts) to be able to handle 16k-token contexts.</p>
  <p>You may seem some parallels between the techniques they discuss and what you mentioned in your post.</p>
  <p>(1) They use Rotary Embeddings, which have an infinite number of positions. However, instead of extending past the original length, they instead <em>interpolate</em> between the existing values.</p>
  <p>(2) They also finetune the model on long-context sequences (long conversational data).</p>
  <p>The resulting performance on different context lengths:
  <a href="https://i.sstatic.net/eYH9Q.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/eYH9Q.png" alt="Context length vs accuracy on a retrieval task" /></a>
  This plot from the LMSYS org shows the LongChat model (described above) versus accuracy on a retrieval task, compared to some other models.</p>
  <p>This technique is elaborated on in this paper: <em><a href="https://arxiv.org/abs/2306.15595" rel="nofollow noreferrer">&quot;Extending Context Window of Large Language Models via Position Interpolation&quot;</a></em></p>
  <blockquote>
  <p>Position Interpolation can easily enable very long context windows (e.g. 32768), requiring only fine-tuning for 1000 steps on the Pile (Gao et al., 2020) to achieve a good quality. The cost of fine-tuning is negligible compared to the pre-training costs. This confirms our hypothesis that it is relatively easy for the models to adapt to interpolated position encodings.</p>
  </blockquote>
  <p>This work suggests that your second suggestion may even be overkill -- you don't need to do too much additional tuning to allow models to use the larger context. They also discuss why one should interpolate between embeddings rather than extrapolate beyond the original context length: namely, that it can result in catastrophic explosions in attention scores when the position differences are too large.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-05T06:35:13.620Z
Answers.Text: So the reason for the interpolation is that the pre-training on the original position encoding still kind of makes sense and we don't have to pre-train from scratch, right?
Answers.UserDisplayName: Robert Wegner
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-05T06:37:37.803Z
Answers.Text: Yeah kinda, you don't have to pre-train from scratch, and interpolation (rather than extrapolation) is used because the model is "used to" the *range* of positional embeddings.
Answers.UserDisplayName: Alexander Wan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-04T16:09:17.383Z
Answers.LastActivityDate: 2023-10-04T18:32:33.327Z
Answers.LastEditDate: 2023-10-04T18:32:33.327Z
Answers.UserDisplayName: Alexander Wan
Body: <p>Let's focus on the case of decoder-only transformers, where I am using algorithm 10 from &quot;Formal Algorithms for Transformers&quot; by Mary Phung and Marcus Hutter as a reference.</p>
  <p><a href="https://i.sstatic.net/Me8rq.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Me8rq.png" alt="Algorithm 10 from &quot;Formal Algorithms for Transformers&quot; by Mary Phung and Marcus Hutter" /></a>: <a href="https://i.sstatic.net/ZWC9o.png" rel="nofollow noreferrer">https://i.sstatic.net/ZWC9o.png</a></p>
  <p>Previously I thought that the maximum context length is very much <strong>built into</strong> the transformer, for example as the dimension of a layer of weights. After studying this algorithm I am surprised because it seems more like an <strong>artificial restriction</strong>! Because this is a topic of active research I would like to know if I am misunderstanding something.</p>
  <p>The way I see it, if I had access to the weights of GPT-2 right now, I could almost execute it on any number of tokens I like right away (If I had sufficient memory to compute this). The MHA algorithm is just carried out over a larger sequenece. There are only two issues, which are points where the context window <span class="math-container">$l_{max}$</span> appears:</p>
  <ol>
  <li>The positional encoding has only <span class="math-container">$l_{max}$</span> positions</li>
  <li>During training the weights were never optimized to attend over more than <span class="math-container">$l_{max}$</span> tokens.</li>
  </ol>
  <p>But these issues seem rather easy to resolve:</p>
  <ol>
  <li><p>Use some positional encoding which has infinitely many positions. The first encoding vectors are nicely spread around while the later ones are closer to eachother, due to the nature of fitting an infinite sequence of vectors into more or less a unit ball/sphere. But this is not an issue: It is natural for the positional encoding to become more vague as the token is further and further in the past.</p>
  </li>
  <li><p>Train 50% on context lengths around <span class="math-container">$l_{max}$</span>, 25% on context lengths around <span class="math-container">$2 l_{max}$</span>, 12.5% on context lengths around <span class="math-container">$4 l_{max}$</span> and so on...</p>
  </li>
  </ol>
  <p>I can imagine the following issues appearing:</p>
  <p>A) Memory becomes larger than what is available on a single &quot;unit&quot; (GPU?) so you have to start moving data back and forth to execute your transformer, which is just terribly inefficient during training and also inference, so it is really pointless to train on such large context windows.</p>
  <p>B) Perhaps the transformer just doesn't learn well with this procedure for some reason.</p>
  <p>These issues are still rather &quot;soft&quot; issues though. As far as I can tell, I could use the architecture of GPT-2 (modified positional encoding) to create 1000000 context window LLMs, in theory. So, am I missing something?</p>
  <p>Thank you!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-10-04T20:31:32.020Z
Comments.Text: In theory you can spend your way to any context length. That said, transformers show really bad information retrieval/context understanding for the middle of long sequences. They show a bias towards information at the start or at the end of the sequence. Can't remember the paper that showed this
Comments.UserDisplayName: Karl
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-10-04T20:56:54.020Z
Comments.Text: @Karl I believe the paper was called "Lost in the middle"
Comments.UserDisplayName: Alexander Wan
Tags: <natural-language-processing><transformer><attention><gpt><decoder>
Title: Aren't context lengths for transformers an artificial restriction?
UserDisplayName: Robert Wegner
ThreadText: # Aren't context lengths for transformers an artificial restriction?
  
  (Asked by: username_0 on 2023-10-04)
  
  Let's focus on the case of decoder-only transformers, where I am using algorithm 10 from "Formal Algorithms for Transformers" by Mary Phung and Marcus Hutter as a reference.
  
  [![Algorithm 10 from ](https://i.sstatic.net/Me8rq.png)](https://i.sstatic.net/Me8rq.png): <https://i.sstatic.net/ZWC9o.png>
  
  Previously I thought that the maximum context length is very much **built into** the transformer, for example as the dimension of a layer of weights. After studying this algorithm I am surprised because it seems more like an **artificial restriction**! Because this is a topic of active research I would like to know if I am misunderstanding something.
  
  The way I see it, if I had access to the weights of GPT-2 right now, I could almost execute it on any number of tokens I like right away (If I had sufficient memory to compute this). The MHA algorithm is just carried out over a larger sequenece. There are only two issues, which are points where the context window $l\_{max}$ appears:
  
  1. The positional encoding has only $l\_{max}$ positions
  2. During training the weights were never optimized to attend over more than $l\_{max}$ tokens.
  
  But these issues seem rather easy to resolve:
  
  1. Use some positional encoding which has infinitely many positions. The first encoding vectors are nicely spread around while the later ones are closer to eachother, due to the nature of fitting an infinite sequence of vectors into more or less a unit ball/sphere. But this is not an issue: It is natural for the positional encoding to become more vague as the token is further and further in the past.
  2. Train 50% on context lengths around $l\_{max}$, 25% on context lengths around $2 l\_{max}$, 12.5% on context lengths around $4 l\_{max}$ and so on...
  
  I can imagine the following issues appearing:
  
  A) Memory becomes larger than what is available on a single "unit" (GPU?) so you have to start moving data back and forth to execute your transformer, which is just terribly inefficient during training and also inference, so it is really pointless to train on such large context windows.
  
  B) Perhaps the transformer just doesn't learn well with this procedure for some reason.
  
  These issues are still rather "soft" issues though. As far as I can tell, I could use the architecture of GPT-2 (modified positional encoding) to create 1000000 context window LLMs, in theory. So, am I missing something?
  
  Thank you!
  
  **Tags:** natural-language-processing, transformer, attention, gpt, decoder
  > **Comments:**
  > **username_1**: In theory you can spend your way to any context length. That said, transformers show really bad information retrieval/context understanding for the middle of long sequences. They show a bias towards information at the start or at the end of the sequence. Can't remember the paper that showed this  
  > **username_2**: @Karl I believe the paper was called "Lost in the middle"  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-10-04. Score: 3 | Accepted answer
  
  Yes, you have the right idea. There's been a lot of work recently regarding extending the context-length of existing models, mostly looking at the Llama family of models.
  
  You should check out [this post/paper](https://lmsys.org/blog/2023-06-29-longchat/) on LongChat by the team behind the Vicuna models (LMSYS). They extend the pretrained Llama models (with 2048-token contexts) to be able to handle 16k-token contexts.
  
  You may seem some parallels between the techniques they discuss and what you mentioned in your post.
  
  (1) They use Rotary Embeddings, which have an infinite number of positions. However, instead of extending past the original length, they instead *interpolate* between the existing values.
  
  (2) They also finetune the model on long-context sequences (long conversational data).
  
  The resulting performance on different context lengths:
  [![Context length vs accuracy on a retrieval task](https://i.sstatic.net/eYH9Q.png)](https://i.sstatic.net/eYH9Q.png)
  This plot from the LMSYS org shows the LongChat model (described above) versus accuracy on a retrieval task, compared to some other models.
  
  This technique is elaborated on in this paper: *["Extending Context Window of Large Language Models via Position Interpolation"](https://arxiv.org/abs/2306.15595)*
  
  > Position Interpolation can easily enable very long context windows (e.g. 32768), requiring only fine-tuning for 1000 steps on the Pile (Gao et al., 2020) to achieve a good quality. The cost of fine-tuning is negligible compared to the pre-training costs. This confirms our hypothesis that it is relatively easy for the models to adapt to interpolated position encodings.
  
  This work suggests that your second suggestion may even be overkill -- you don't need to do too much additional tuning to allow models to use the larger context. They also discuss why one should interpolate between embeddings rather than extrapolate beyond the original context length: namely, that it can result in catastrophic explosions in attention scores when the position differences are too large.
  
  > **Comments:**
  > **username_0**: So the reason for the interpolation is that the pre-training on the original position encoding still kind of makes sense and we don't have to pre-train from scratch, right?  
  > **username_2**: Yeah kinda, you don't have to pre-train from scratch, and interpolation (rather than extrapolation) is used because the model is "used to" the *range* of positional embeddings.  
  
  ---
  
  