Answers.Body: <p>To some extent, this is true; The piecewise feedforward layers can be added or subtracted to fit the sequence length. The matrix operations can similarly be scaled to fit sequence length.</p>
  <p>However, the computational complexity comes from the matrix operations in the attention layer. Those are not trained; There are no trained parameters in the attention mechanism (see figure 2 and equation (1) in Vaswani et al). So, those have to be computed during inference as well.</p>
  <p>Another challenge would be the output layer. That layer is a regular feedforward layer and thus has a fixed input size; That is, you cannot add new parameters during inference.</p>
  <p>Of course, there is a caveat to this; There are now transformers now that allow recurrence, such as Transformer-XL and Memformer. These do, in a way, allow longer input sequences than &quot;max sequence length&quot;.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-29T22:16:43.287Z
Answers.Text: I think the part about the output layer is incorrect. Usually the output layer is repeated for thesequence length, so that should not be an issue. Also the point about the matrix operations does not answer why transfoermers CANNOT be used for sequences larger than what they were trained on.
Answers.UserDisplayName: chessprogrammer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-29T22:31:06.013Z
Answers.Text: @chessprogrammer It's not strictly necessary to repeat it for the sequence length; It helps with training, but the output is in principle a single token (especially if you want to avoid teacher forcing, although that's of course still ongoing research). As Vaswani say in the first paragraph of section 3 in their paper: "Given z, the decoder then generates an output
  sequence (y1, ..., ym) of symbols one element at a time."
Answers.UserDisplayName: Avatrin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-29T23:09:46.077Z
Answers.Text: @chessprogrammer I only mentioned the matrix complexity point because I interpreted your question as being premised on the quadratic complexity being the reason for the limitation.
Answers.UserDisplayName: Avatrin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-29T17:47:20.517Z
Answers.LastActivityDate: 2021-11-29T17:47:20.517Z
Answers.UserDisplayName: Avatrin
Answers.Body: <p>If you used something like Rotary Positional Embedding -- such as the LLama3 family of models -- then technically speaking, you could do inference with arbitrary long sequence (assuming you have enough computational resources). The transformer itself does not have any inherent limitation.</p>
  <p>The real question is more like: will it work well, if no such sequence has been ever seen in training? Will it understand how to interpret a token -2000 from current position, if in training, it has seen at most tokens -1000 from current position?</p>
  <p>I tried to overwrite the max_length configuration in llama3-7b, just for fun, and got this warning:
  <code>This is a friendly reminder - the current text generation call will exceed the model's predefined maximum length (2048). Depending on the model, you may observe exceptions, performance degradation, or nothing at all.</code></p>
  <p>It is common to use shorter sequences on inference, but not longer.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-14T18:03:50.780Z
Answers.LastActivityDate: 2024-08-14T18:03:50.780Z
Answers.UserDisplayName: Peter Franek
Answers.Body: <p>Transformer models have limited sequence length at inference time
  because of positional embeddings. But there are workarounds.</p>
  <p>Self-attention in transformer does not distinguish the order of keys/values,
  it works as if the sequence is a bag of words.</p>
  <p>So to expose the sequence order to the model, one typically adds an extra
  &quot;positional embedding&quot; vector to each input token embedding.
  This extra positional information then allows the model to construct primitives like &quot;attention head that looks at previous token&quot;.</p>
  <p>There are several variants of how to do this exactly.</p>
  <p>One way is to use an extra trainable parameter matrix [L,M]
  where L is maximum input length and M is model dimension.
  After training a model this way, it simply becomes impossible
  to embed tokens with position &gt; L. So we get a hard limit for which there is no workaround.</p>
  <p>Another way is to use a non-trainable matrix [L,M]
  initialized with a specially crafted set of sinusoids
  (original &quot;attention is all you need&quot; paper does this).
  A model trained this way would not have a length limit,
  because this matrix of sinusoids you can extend to arbitrary size.</p>
  <p>Yes another way is to use &quot;relative positional encoding&quot;.
  With this, for each pair of tokens you take relative position,
  which would be in range [-L ... +L], and embed that instead.
  Then you inject this vector into attention layer in the right way
  (section 3.3 in <a href="https://arxiv.org/pdf/1901.02860.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1901.02860.pdf</a>).</p>
  <p>Now you still kind of have a limit of [-L, L].
  But you can always &quot;clip&quot; relative position to this range,
  and pretend that all pairs of tokens at distance &gt;= L
  have the same relative position (&quot;very far from each other&quot;).
  And this allows to run inference with longer inputs.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-24T11:16:35.970Z
Answers.Text: What are the relative positions relative to?!
Answers.UserDisplayName: HappyFace
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-24T14:36:59.743Z
Answers.Text: Thanks for this answer! I had thought positional embeddings were the problem but.I couldn't find anywhere where that was stated explicitly. Thanks for clarifying
Answers.UserDisplayName: chessprogrammer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-24T14:41:50.897Z
Answers.Text: One follow up question: even if we are using the sinusoid positional embeddings, would there still be a problem. Granted it is easy to extend the embeddings arbitrarily,  but still since the network has never seen positions beyond the training length limit, wouldnt the network weights be unfamiliar with that part of the function
Answers.UserDisplayName: chessprogrammer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-24T15:57:50.177Z
Answers.Text: @HappyFace: "relative position" means distance between query and key in self-attention
Answers.UserDisplayName: user2907934
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-23T21:07:25.463Z
Answers.LastActivityDate: 2022-01-23T21:07:25.463Z
Answers.UserDisplayName: user2907934
Body: <p>As far as I understand, Transformer's time complexity increases quadratically with respect to the sequence length. As a result, during training to make training feasible, a maximum sequence limit is set, and to allow batching, all sequences smaller are padded.</p>
  <p>However, after a Transformer is trained, and we want to run it on a single sequence at inference time, the computational costs are far less than training. Thus, it seems reasonable that I would want to run the transformer on a larger input sequence length during inference time. From a technical perspective, this should be feasible.</p>
  <p>I keep reading online that a Transformer cannot be run on a sequence size larger than the one seen during training. Why is this? Is it because the network weights will be unfamiliar with sequences of this length? Or is it more fundamental?</p>
  
Tags: <machine-learning><natural-language-processing><transformer><architecture><sequence-modeling>
Title: Why do Transformers have a sequence limit at inference time?
UserDisplayName: chessprogrammer
ThreadText: # Why do Transformers have a sequence limit at inference time?
  
  (Asked by: username_0 on 2021-11-26)
  
  As far as I understand, Transformer's time complexity increases quadratically with respect to the sequence length. As a result, during training to make training feasible, a maximum sequence limit is set, and to allow batching, all sequences smaller are padded.
  
  However, after a Transformer is trained, and we want to run it on a single sequence at inference time, the computational costs are far less than training. Thus, it seems reasonable that I would want to run the transformer on a larger input sequence length during inference time. From a technical perspective, this should be feasible.
  
  I keep reading online that a Transformer cannot be run on a sequence size larger than the one seen during training. Why is this? Is it because the network weights will be unfamiliar with sequences of this length? Or is it more fundamental?
  
  **Tags:** machine-learning, natural-language-processing, transformer, architecture, sequence-modeling
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2022-01-23. Score: 4 | Accepted answer
  
  Transformer models have limited sequence length at inference time
  because of positional embeddings. But there are workarounds.
  
  Self-attention in transformer does not distinguish the order of keys/values,
  it works as if the sequence is a bag of words.
  
  So to expose the sequence order to the model, one typically adds an extra
  "positional embedding" vector to each input token embedding.
  This extra positional information then allows the model to construct primitives like "attention head that looks at previous token".
  
  There are several variants of how to do this exactly.
  
  One way is to use an extra trainable parameter matrix [L,M]
  where L is maximum input length and M is model dimension.
  After training a model this way, it simply becomes impossible
  to embed tokens with position > L. So we get a hard limit for which there is no workaround.
  
  Another way is to use a non-trainable matrix [L,M]
  initialized with a specially crafted set of sinusoids
  (original "attention is all you need" paper does this).
  A model trained this way would not have a length limit,
  because this matrix of sinusoids you can extend to arbitrary size.
  
  Yes another way is to use "relative positional encoding".
  With this, for each pair of tokens you take relative position,
  which would be in range [-L ... +L], and embed that instead.
  Then you inject this vector into attention layer in the right way
  (section 3.3 in <https://arxiv.org/pdf/1901.02860.pdf>).
  
  Now you still kind of have a limit of [-L, L].
  But you can always "clip" relative position to this range,
  and pretend that all pairs of tokens at distance >= L
  have the same relative position ("very far from each other").
  And this allows to run inference with longer inputs.
  
  > **Comments:**
  > **username_4**: What are the relative positions relative to?!  
  > **username_0**: Thanks for this answer! I had thought positional embeddings were the problem but.I couldn't find anywhere where that was stated explicitly. Thanks for clarifying  
  > **username_0**: One follow up question: even if we are using the sinusoid positional embeddings, would there still be a problem. Granted it is easy to extend the embeddings arbitrarily,  but still since the network has never seen positions beyond the training length limit, wouldnt the network weights be unfamiliar with that part of the function  
  > **username_3**: @HappyFace: "relative position" means distance between query and key in self-attention  
  
  ---
  
  ### Answer by username_1 on 2021-11-29. Score: 1
  
  To some extent, this is true; The piecewise feedforward layers can be added or subtracted to fit the sequence length. The matrix operations can similarly be scaled to fit sequence length.
  
  However, the computational complexity comes from the matrix operations in the attention layer. Those are not trained; There are no trained parameters in the attention mechanism (see figure 2 and equation (1) in Vaswani et al). So, those have to be computed during inference as well.
  
  Another challenge would be the output layer. That layer is a regular feedforward layer and thus has a fixed input size; That is, you cannot add new parameters during inference.
  
  Of course, there is a caveat to this; There are now transformers now that allow recurrence, such as Transformer-XL and Memformer. These do, in a way, allow longer input sequences than "max sequence length".
  
  > **Comments:**
  > **username_0**: I think the part about the output layer is incorrect. Usually the output layer is repeated for thesequence length, so that should not be an issue. Also the point about the matrix operations does not answer why transfoermers CANNOT be used for sequences larger than what they were trained on.  
  > **username_1**: @chessprogrammer It's not strictly necessary to repeat it for the sequence length; It helps with training, but the output is in principle a single token (especially if you want to avoid teacher forcing, although that's of course still ongoing research). As Vaswani say in the first paragraph of section 3 in their paper: "Given z, the decoder then generates an output sequence (y1, ..., ym) of symbols one element at a time."  
  > **username_1**: @chessprogrammer I only mentioned the matrix complexity point because I interpreted your question as being premised on the quadratic complexity being the reason for the limitation.  
  
  ---
  
  ### Answer by username_2 on 2024-08-14. Score: 1
  
  If you used something like Rotary Positional Embedding -- such as the LLama3 family of models -- then technically speaking, you could do inference with arbitrary long sequence (assuming you have enough computational resources). The transformer itself does not have any inherent limitation.
  
  The real question is more like: will it work well, if no such sequence has been ever seen in training? Will it understand how to interpret a token -2000 from current position, if in training, it has seen at most tokens -1000 from current position?
  
  I tried to overwrite the max\_length configuration in llama3-7b, just for fun, and got this warning:
  `This is a friendly reminder - the current text generation call will exceed the model's predefined maximum length (2048). Depending on the model, you may observe exceptions, performance degradation, or nothing at all.`
  
  It is common to use shorter sequences on inference, but not longer.
  
  
  ---
  
  