Answers.Body: <p>As @N. Kiefer wrote, a trained transformer can handle input sequences of a length smaller than it maximum sequence length. I thought about this and I realized how this is a direct consequence of transformers' design.</p>
  <p>Input embeddings are augmented with positional encoding. This works &quot;out of the box&quot; as the positional encoding is a function of the token placement within the input sequence <code>[0, max_seq_len]</code>. So the positional encoding happens as usual.</p>
  <p>Then, they go through the encoder stack. In a single encoder, they are first converted to <code>Q</code>uery, <code>K</code>ey and <code>V</code>alue matrices. This conversion uses predetermined per-head matrices. Then <code>Z</code> matrices are computed per each token per each head. This also works as usual, as the sizes of <code>Q</code>, <code>V</code>, and <code>K</code> match. Combining multi-headed attention also works, as this is, again, a per token process. Then, the residual original encoding is added on top and the values proceed to the linear layer, where each token is processed individually. So, apparently, the encoder works fine with different input lengths.</p>
  <p>Encoder-decoder cross-attention also works. That's because the <code>Q</code> matrix taken from decoder's masked self-attention can be multiplied with any number of keys from the encoder. And the key and value matrix sizes match, as they are both coming from the encoder.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-05T07:20:22.227Z
Answers.Text: The author of the post mentioned GPT-4, which to my knowledge, does not actually have an encoder stack as it is a decoder-only architecture. How would it work for a decoder-only architecture?
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-27T13:07:01.857Z
Answers.LastActivityDate: 2023-04-27T13:07:01.857Z
Answers.UserDisplayName: janekb04
Answers.Body: <p>you only pad sequences when you combine sequences of different lengths. This way you get a tensor of same sized sequences. when you only want to process one sequence, Transformers are capable of handling different sizes.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-26T11:06:02.373Z
Answers.LastActivityDate: 2023-04-26T11:06:02.373Z
Answers.UserDisplayName: N. Kiefer
Body: <p>As far as I understand, a Transformer has a specific input sequence length that depends on its architecture. So a model like <code>gpt-4</code> has a sequence length of 8192 tokens. As such, I am interested what happens when the input prompt is shorter than that. <a href="https://ai.stackexchange.com/questions/22957/how-can-transformers-handle-arbitrary-length-input">This question and answers</a> suggest that the input is simply padded out to the full input sequence length. But that doesn't really seem plausible to me. That would mean that the cost of processing a 100 token prompt would be the same as the cost of processing a 1000 token prompt. And it clearly isn't, as they charge a different price for them - they charge for token count, not API request count.</p>
  <p>I am interested to know how variable length input sequences are handled for experimenting with trying to implement my own language model. I have two possible hypothesis. One could be that they actually combine multiple prompts into a single large prompt and tell the model to complete them all. But I don't think that it is the case as the risk of mixing the prompts from different users and the data they have would probably drastically degrade the quality of the output. I also thought that maybe they have different model sizes - like <code>gpt4-1k</code>, <code>gpt4-2k</code>, <code>gpt4-3k</code>, ... - but that seems like a large overhead, as each of these would need to be trained separately.</p>
  
Tags: <natural-language-processing><transformer><inference><production-systems>
Title: How to optimize transformer inference for prompts shorter than the maximum sequence length?
UserDisplayName: janekb04
ThreadText: # How to optimize transformer inference for prompts shorter than the maximum sequence length?
  
  (Asked by: username_0 on 2023-04-26)
  
  As far as I understand, a Transformer has a specific input sequence length that depends on its architecture. So a model like `gpt-4` has a sequence length of 8192 tokens. As such, I am interested what happens when the input prompt is shorter than that. [This question and answers](https://ai.stackexchange.com/questions/22957/how-can-transformers-handle-arbitrary-length-input) suggest that the input is simply padded out to the full input sequence length. But that doesn't really seem plausible to me. That would mean that the cost of processing a 100 token prompt would be the same as the cost of processing a 1000 token prompt. And it clearly isn't, as they charge a different price for them - they charge for token count, not API request count.
  
  I am interested to know how variable length input sequences are handled for experimenting with trying to implement my own language model. I have two possible hypothesis. One could be that they actually combine multiple prompts into a single large prompt and tell the model to complete them all. But I don't think that it is the case as the risk of mixing the prompts from different users and the data they have would probably drastically degrade the quality of the output. I also thought that maybe they have different model sizes - like `gpt4-1k`, `gpt4-2k`, `gpt4-3k`, ... - but that seems like a large overhead, as each of these would need to be trained separately.
  
  **Tags:** natural-language-processing, transformer, inference, production-systems
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2023-04-27. Score: 1
  
  As @username_2 wrote, a trained transformer can handle input sequences of a length smaller than it maximum sequence length. I thought about this and I realized how this is a direct consequence of transformers' design.
  
  Input embeddings are augmented with positional encoding. This works "out of the box" as the positional encoding is a function of the token placement within the input sequence `[0, max_seq_len]`. So the positional encoding happens as usual.
  
  Then, they go through the encoder stack. In a single encoder, they are first converted to `Q`uery, `K`ey and `V`alue matrices. This conversion uses predetermined per-head matrices. Then `Z` matrices are computed per each token per each head. This also works as usual, as the sizes of `Q`, `V`, and `K` match. Combining multi-headed attention also works, as this is, again, a per token process. Then, the residual original encoding is added on top and the values proceed to the linear layer, where each token is processed individually. So, apparently, the encoder works fine with different input lengths.
  
  Encoder-decoder cross-attention also works. That's because the `Q` matrix taken from decoder's masked self-attention can be multiplied with any number of keys from the encoder. And the key and value matrix sizes match, as they are both coming from the encoder.
  
  > **Comments:**
  > **username_1**: The author of the post mentioned GPT-4, which to my knowledge, does not actually have an encoder stack as it is a decoder-only architecture. How would it work for a decoder-only architecture?  
  
  ---
  
  ### Answer by username_2 on 2023-04-26. Score: 0
  
  you only pad sequences when you combine sequences of different lengths. This way you get a tensor of same sized sequences. when you only want to process one sequence, Transformers are capable of handling different sizes.
  
  
  ---
  
  