Answers.Body: <p>The accepted answer is wonderful; this answer provides an alternative approach for dealing with variable length inputs.  More specifically, what might be done when the input is longer than the maximum sequence length supported by the transformer you have built.</p>
  <p>We have found it useful to wrap our transformer in a class that allows us to programmatically use a sliding window across inputs that are longer than the supported transformer input length.  If the input is less than or equal to the supported length, it is simply processed.  If it is longer, we iteratively slide across the data, passing each window into the transformer and then aggregate the outputs.</p>
  <p>When we take this approach, we do not typically slide the window one element (word embedding) at a time, but instead use a longer stride, usually two to five embeddings at a time.  We have been intending to do some research into evaluating whether using multiple strides improves overall performance, but have not yet done so because of the prohibitive computational performance characteristics of using multiple strides.  Using a sliding window already significantly impacts the time to predict since we are running the predictions multiple times.</p>
  <p>If this approach seems useful, a simple insight is that you need not pass the inputs in sequentially; instead, we typically build a batch with all of the windows and pass them through all at once.</p>
  <p>Of course, there is a downside to this approach that might make you decide to choose to split the windows based on sentences or paragraphs.  otherwise, your positional encoding will end up being &quot;off&quot; since you are sliding across the inputs.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-04T12:55:16.933Z
Answers.LastActivityDate: 2023-01-04T18:15:58.457Z
Answers.LastEditDate: 2023-01-04T18:15:58.457Z
Answers.UserDisplayName: David Hoelzer
Answers.Body: <p>Actually, there is usually an upper bound for inputs of transformers, due to the inability of handling long-sequence. Usually, the value is set as 512 or 1024 at current stage.</p>
  <p>However, if you are asking handling the various input size, adding padding token such as [PAD] in BERT model is a common solution. The position of [PAD] token could be masked in self-attention, therefore, causes no influence. Let's say we use a transformer model with 512 limit of sequence length, then we pass a input sequence of 103 tokens. We padded it to 512 tokens. In the attention layer, positions from 104 to 512 are all masked, that is, they are not attending or being attended.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-11T13:35:39.403Z
Answers.Text: Thanks for the answer. Could you clarify one more point: What do you do when the input sequence is *longer* than 512/1024? Is a transformer simply inappropriate in that case?
Answers.UserDisplayName: chessprogrammer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-12T07:12:53.570Z
Answers.Text: Yes, it's the limitation of BERT. Actually, there are tons of researches trying to solve this issue such as SpanBERT, Longformer, Reformer, Sparse transformer, etc. We could categorize them into two categories roughly. One is embracing the length limitation and divide the sequence into sections, while the others assume the attention could be sparsified.
Answers.UserDisplayName: tsu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-28T14:49:08.057Z
Answers.Text: @tsu is that limitation still relevant to GPT 4?
Answers.UserDisplayName: Weier
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-29T16:11:11.197Z
Answers.Text: @Weier After ~4 years, the limitation has been pushed up to 128K by OpenAI and 100M claimed by Google. IIUC, the main architecture of LLMs is still multi-head attention based transformer although a variety of researches on sparse and efficient transformers are proposed. I would say the main improvement on pushing the sequecne limitation might be attributed to quatization and the optimization on GPUs.
Answers.UserDisplayName: tsu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-10T05:43:22.993Z
Answers.LastActivityDate: 2020-08-10T05:43:22.993Z
Answers.UserDisplayName: tsu
Body: <p>The transformer, introduced in the paper <a href="https://arxiv.org/abs/1706.03762" rel="noreferrer">Attention Is All You Need</a>, is a popular new neural network architecture that is commonly viewed as an alternative to recurrent neural networks, like LSTMs and GRUs.</p>
  <p>However, having gone through the paper, as well as several online explanations, I still have trouble wrapping my head around how they work. How can a non-recurrent structure be able to deal with inputs of arbitrary length?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-10T05:41:37.267Z
Comments.Text: Transformer cannot handle arbitrary length. You must have misunderstood something. For example, BERT is a transformer architecture (encoder only), and it has a maximum length of 512 tokens. A very useful blog for understanding transformers is [The Illustrated Transformer](https://jalammar.github.io/illustrated-transformer/)
Comments.UserDisplayName: Astariul
Tags: <natural-language-processing><recurrent-neural-networks><long-short-term-memory><transformer>
Title: How can Transformers handle arbitrary length input?
UserDisplayName: chessprogrammer
ThreadText: # How can Transformers handle arbitrary length input?
  
  (Asked by: username_0 on 2020-08-10)
  
  The transformer, introduced in the paper [Attention Is All You Need](https://arxiv.org/abs/1706.03762), is a popular new neural network architecture that is commonly viewed as an alternative to recurrent neural networks, like LSTMs and GRUs.
  
  However, having gone through the paper, as well as several online explanations, I still have trouble wrapping my head around how they work. How can a non-recurrent structure be able to deal with inputs of arbitrary length?
  
  **Tags:** natural-language-processing, recurrent-neural-networks, long-short-term-memory, transformer
  > **Comments:**
  > **username_1**: Transformer cannot handle arbitrary length. You must have misunderstood something. For example, BERT is a transformer architecture (encoder only), and it has a maximum length of 512 tokens. A very useful blog for understanding transformers is [The Illustrated Transformer](https://jalammar.github.io/illustrated-transformer/)  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2020-08-10. Score: 33 | Accepted answer
  
  Actually, there is usually an upper bound for inputs of transformers, due to the inability of handling long-sequence. Usually, the value is set as 512 or 1024 at current stage.
  
  However, if you are asking handling the various input size, adding padding token such as [PAD] in BERT model is a common solution. The position of [PAD] token could be masked in self-attention, therefore, causes no influence. Let's say we use a transformer model with 512 limit of sequence length, then we pass a input sequence of 103 tokens. We padded it to 512 tokens. In the attention layer, positions from 104 to 512 are all masked, that is, they are not attending or being attended.
  
  > **Comments:**
  > **username_0**: Thanks for the answer. Could you clarify one more point: What do you do when the input sequence is *longer* than 512/1024? Is a transformer simply inappropriate in that case?  
  > **username_3**: Yes, it's the limitation of BERT. Actually, there are tons of researches trying to solve this issue such as SpanBERT, Longformer, Reformer, Sparse transformer, etc. We could categorize them into two categories roughly. One is embracing the length limitation and divide the sequence into sections, while the others assume the attention could be sparsified.  
  > **username_4**: @tsu is that limitation still relevant to GPT 4?  
  > **username_3**: @Weier After ~4 years, the limitation has been pushed up to 128K by OpenAI and 100M claimed by Google. IIUC, the main architecture of LLMs is still multi-head attention based transformer although a variety of researches on sparse and efficient transformers are proposed. I would say the main improvement on pushing the sequecne limitation might be attributed to quatization and the optimization on GPUs.  
  
  ---
  
  ### Answer by username_2 on 2023-01-04. Score: 8
  
  The accepted answer is wonderful; this answer provides an alternative approach for dealing with variable length inputs. More specifically, what might be done when the input is longer than the maximum sequence length supported by the transformer you have built.
  
  We have found it useful to wrap our transformer in a class that allows us to programmatically use a sliding window across inputs that are longer than the supported transformer input length. If the input is less than or equal to the supported length, it is simply processed. If it is longer, we iteratively slide across the data, passing each window into the transformer and then aggregate the outputs.
  
  When we take this approach, we do not typically slide the window one element (word embedding) at a time, but instead use a longer stride, usually two to five embeddings at a time. We have been intending to do some research into evaluating whether using multiple strides improves overall performance, but have not yet done so because of the prohibitive computational performance characteristics of using multiple strides. Using a sliding window already significantly impacts the time to predict since we are running the predictions multiple times.
  
  If this approach seems useful, a simple insight is that you need not pass the inputs in sequentially; instead, we typically build a batch with all of the windows and pass them through all at once.
  
  Of course, there is a downside to this approach that might make you decide to choose to split the windows based on sentences or paragraphs. otherwise, your positional encoding will end up being "off" since you are sliding across the inputs.
  
  
  ---
  
  