Answers.Body: <p>Transformers can be run with a variable dimension <code>N_INPUT</code> as pointed out by <a href="https://ai.stackexchange.com/questions/41352/why-does-llm-inference-cost-scale-in-both-input-tokens-and-output-tokens#comment61395_41352">@pcpthm's comment</a>. That means that the answer to <a href="https://ai.stackexchange.com/questions/37624/why-do-transformers-have-a-fixed-input-length">&quot;Why do transformers have a fixed input length?&quot;</a> are wrong, and the answers to <a href="https://ai.stackexchange.com/questions/37267/how-do-transformer-decoders-handle-arbitrary-length-input/37289#37289">&quot;How do transformers handle arbitrary-length input?&quot;</a> are more correct.</p>
  <p>If the input of dimension <code>N_INPUT * N_VOCAB</code> and <code>N_INPUT</code> is variable then that simply means <code>Q, K, V, Z</code> have one variable dimension (<code>N_INPUT * D_ATTN_HEAD</code>) and input/output of MLP are just <code>N_INPUT * WIDTH</code>.</p>
  <p>For some reason I thought the hidden layers would be fixed size and/or require a fixed size input, but this is not the case since it turns out that <code>N_INPUT</code> is always the first dimension, except in the result of <code>Q * K^T</code> which has dimension <code>N_INPUT * N_INPUT</code>, but is only multiplied on the right by <code>V</code> which has first dimension <code>N_INPUT</code>.</p>
  <p>In the NNs that I'd learned about previously, N_INPUT is the second dimension so even if one were to put in a variable input, all of the hidden layers would still be fixed-size.</p>
  <p>Particularly what I did not realize is that the &quot;context window&quot; is not actually a part of the model (<a href="https://ai.stackexchange.com/a/37289/68775">except in the positional encoding</a>). It seems to be purely a training time / inference time restriction.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-21T02:36:38.200Z
Answers.LastActivityDate: 2023-07-21T02:43:05.503Z
Answers.LastEditDate: 2023-07-21T02:43:05.503Z
Answers.UserDisplayName: llllvvuu
Body: <h2>EDIT</h2>
  <p>This question was flawed. See <a href="https://ai.stackexchange.com/a/41388/68775">my answer</a> with help from commenters.</p>
  <hr />
  <h2>Original question</h2>
  <p>This question has been asked in other forums <a href="https://community.openai.com/t/why-does-pricing-vary-by-input-tokens-instead-of-only-output-tokens/21833" rel="nofollow noreferrer">[1]</a> <a href="https://www.reddit.com/r/OpenAI/comments/11vvx66/whats_the_rationale_for_charging_for_input_tokens/" rel="nofollow noreferrer">[2]</a> but I'm not sure I understand the claims, which are (EDIT: the following are based on my faulty assumption that pad tokens are added up to the maximum context window):</p>
  <ol>
  <li>Each forward pass takes less resources when more of the context window is padding.</li>
  <li>Forward passes are run on the input tokens.</li>
  <li>Forward passes with fewer non-pad input tokens are smaller tensor operations.</li>
  </ol>
  <p>Hypothesis 1 seems the most plausible to me from a performance engineering standpoint (sparse math, etc). Does it fall out naturally from just writing basic JAX code or would it require manual optimization (if so, what tricks can be used?)? There does <a href="https://arxiv.org/abs/2210.03052" rel="nofollow noreferrer">seem to be some research on this</a>.</p>
  <p>Hypothesis 2 and 3 seem wrong based on my surface-level understanding of the Transformer architecture.</p>
  <p>I've tried a few open-source LLMs locally and neither on those nor ChatGPT have I noticed any difference in latency based on how much text was in the context window. But I haven't done actual rigorous benchmarking yet.</p>
  <p>The reason this is relevant is due to document lookup-based applications. Looking at the <a href="https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_embeddings.ipynb" rel="nofollow noreferrer">OpenAI cookbook for Q&amp;A using embeddings</a>, I can see that:</p>
  <ul>
  <li>most of the token usage seems to come from the long Wikipedia article pasted into the context</li>
  <li>the price per query is quite prohibitive to frequent usage:</li>
  </ul>
  <blockquote>
  <p>For gpt-3.5-turbo using ~1,000 tokens per query, it costs ~0.002 per query, or ~500 queries per dollar (as of Apr 2023)
  For gpt-4, again assuming ~1,000 tokens per query, it costs ~0.03 per query, or ~30 queries per dollar (as of Apr 2023)</p>
  </blockquote>
  <p>My prior intuition would have been that optimal usage of LLMs would be to keep the context filled with inexpensive text (e.g. from NN search and/or cheaper LMs) and to have the LLM generate terse responses. But the input token cost model changes the strategy, as it means that users need to be sparing about the size and quantity of documents that they paste into the context window.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-19T09:15:45.780Z
Comments.Text: you have a (ignoring possible improvements from FastAttention) a modules that scales quadratically with the sequence length (attention), thus doubling the length, will x4 the cost of that... thus, it's expensive to have both an long input, and a long output (as words are generated autoregressively)
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-19T09:44:25.430Z
Comments.Text: So with padding it does not do the full tensor dot?
Comments.UserDisplayName: llllvvuu
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-19T10:13:35.630Z
Comments.Text: Unfortunately, this is a bit opinion-based, so needs to be closed. Asking why companies charge you money is a bit opinion-based. Formulate the question to make it a **theoretical** question about AI. Please, change the title accordingly. Maybe the title is the only thing you need to reformulate.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-20T12:31:58.657Z
Comments.Text: The Transformer architecture doesn't require padding and can natively process variable-length input. Why do you think there are pad tokens added? (in practice it may be more complicated because one wants to batch multiple inputs)
Comments.UserDisplayName: pcpthm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-20T19:34:34.913Z
Comments.Text: One can only speculate about pricing decisions and furthermore, how OpenAI are batching requests. For example, they might well batch similarly sized requests together. This avoids a situation where most tokens in a batch are padding because one input is huge and the rest are small. Pad tokens are still expensive to calculate, so this is sensible. This  would mean shorter inputs are cheaper than long ones, due to lack of padding (not more of it).
Comments.UserDisplayName: Venna Banana
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-21T02:05:46.217Z
Comments.Text: OK, I understood now. If the input of dimension N_INPUT * N_VOCAB and N_INPUT is variable then that simply means Q, K, V, Z have one variable dimension (N_INPUT * D_ATTN_HEAD) and input/output of MLP are just N_INPUT * WIDTH. For some reason I thought there would be a mismatch in some matrix multiplication if N_INPUT is not fixed, but it turns out that N_INPUT is always the first dimension (except in the result of Q * K_T but since it is only multipled with V on the right which has first dimension N_INPUT it is okay) so this is not the case.
Comments.UserDisplayName: llllvvuu
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-21T02:25:32.297Z
Comments.Text: I guess this also makes the answers here wrong? https://ai.stackexchange.com/questions/37624/why-do-transformers-have-a-fixed-input-length?rq=1
  
  With this question's answer being more correct: https://ai.stackexchange.com/a/37289/68775
Comments.UserDisplayName: llllvvuu
Tags: <transformer><open-ai><embeddings><large-language-models><inference>
Title: Why does LLM inference cost scale in both input tokens and output tokens?
UserDisplayName: llllvvuu
ThreadText: # Why does LLM inference cost scale in both input tokens and output tokens?
  
  (Asked by: username_0 on 2023-07-19)
  
  ## EDIT
  
  This question was flawed. See [my answer](https://ai.stackexchange.com/a/41388/68775) with help from commenters.
  
  ---
  
  ## Original question
  
  This question has been asked in other forums [[1]](https://community.openai.com/t/why-does-pricing-vary-by-input-tokens-instead-of-only-output-tokens/21833) [[2]](https://www.reddit.com/r/OpenAI/comments/11vvx66/whats_the_rationale_for_charging_for_input_tokens/) but I'm not sure I understand the claims, which are (EDIT: the following are based on my faulty assumption that pad tokens are added up to the maximum context window):
  
  1. Each forward pass takes less resources when more of the context window is padding.
  2. Forward passes are run on the input tokens.
  3. Forward passes with fewer non-pad input tokens are smaller tensor operations.
  
  Hypothesis 1 seems the most plausible to me from a performance engineering standpoint (sparse math, etc). Does it fall out naturally from just writing basic JAX code or would it require manual optimization (if so, what tricks can be used?)? There does [seem to be some research on this](https://arxiv.org/abs/2210.03052).
  
  Hypothesis 2 and 3 seem wrong based on my surface-level understanding of the Transformer architecture.
  
  I've tried a few open-source LLMs locally and neither on those nor ChatGPT have I noticed any difference in latency based on how much text was in the context window. But I haven't done actual rigorous benchmarking yet.
  
  The reason this is relevant is due to document lookup-based applications. Looking at the [OpenAI cookbook for Q&A using embeddings](https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_embeddings.ipynb), I can see that:
  
  * most of the token usage seems to come from the long Wikipedia article pasted into the context
  * the price per query is quite prohibitive to frequent usage:
  
  > For gpt-3.5-turbo using ~1,000 tokens per query, it costs ~0.002 per query, or ~500 queries per dollar (as of Apr 2023)
  > For gpt-4, again assuming ~1,000 tokens per query, it costs ~0.03 per query, or ~30 queries per dollar (as of Apr 2023)
  
  My prior intuition would have been that optimal usage of LLMs would be to keep the context filled with inexpensive text (e.g. from NN search and/or cheaper LMs) and to have the LLM generate terse responses. But the input token cost model changes the strategy, as it means that users need to be sparing about the size and quantity of documents that they paste into the context window.
  
  **Tags:** transformer, open-ai, embeddings, large-language-models, inference
  
  > **Comments:**
  > **username_1**: you have a (ignoring possible improvements from FastAttention) a modules that scales quadratically with the sequence length (attention), thus doubling the length, will x4 the cost of that... thus, it's expensive to have both an long input, and a long output (as words are generated autoregressively)  
  > **username_0**: So with padding it does not do the full tensor dot?  
  > **username_2**: Unfortunately, this is a bit opinion-based, so needs to be closed. Asking why companies charge you money is a bit opinion-based. Formulate the question to make it a **theoretical** question about AI. Please, change the title accordingly. Maybe the title is the only thing you need to reformulate.  
  > **username_3**: The Transformer architecture doesn't require padding and can natively process variable-length input. Why do you think there are pad tokens added? (in practice it may be more complicated because one wants to batch multiple inputs)  
  > **username_4**: One can only speculate about pricing decisions and furthermore, how OpenAI are batching requests. For example, they might well batch similarly sized requests together. This avoids a situation where most tokens in a batch are padding because one input is huge and the rest are small. Pad tokens are still expensive to calculate, so this is sensible. This  would mean shorter inputs are cheaper than long ones, due to lack of padding (not more of it).  
  > **username_0**: OK, I understood now. If the input of dimension N_INPUT * N_VOCAB and N_INPUT is variable then that simply means Q, K, V, Z have one variable dimension (N_INPUT * D_ATTN_HEAD) and input/output of MLP are just N_INPUT * WIDTH. For some reason I thought there would be a mismatch in some matrix multiplication if N_INPUT is not fixed, but it turns out that N_INPUT is always the first dimension (except in the result of Q * K_T but since it is only multipled with V on the right which has first dimension N_INPUT it is okay) so this is not the case.  
  > **username_0**: I guess this also makes the answers here wrong? https://ai.stackexchange.com/questions/37624/why-do-transformers-have-a-fixed-input-length?rq=1  With this question's answer being more correct: https://ai.stackexchange.com/a/37289/68775  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2023-07-21. Score: 0 | Accepted answer
  
  Transformers can be run with a variable dimension `N_INPUT` as pointed out by [@username_3's comment](https://ai.stackexchange.com/questions/41352/why-does-llm-inference-cost-scale-in-both-input-tokens-and-output-tokens#comment61395_41352). That means that the answer to ["Why do transformers have a fixed input length?"](https://ai.stackexchange.com/questions/37624/why-do-transformers-have-a-fixed-input-length) are wrong, and the answers to ["How do transformers handle arbitrary-length input?"](https://ai.stackexchange.com/questions/37267/how-do-transformer-decoders-handle-arbitrary-length-input/37289#37289) are more correct.
  
  If the input of dimension `N_INPUT * N_VOCAB` and `N_INPUT` is variable then that simply means `Q, K, V, Z` have one variable dimension (`N_INPUT * D_ATTN_HEAD`) and input/output of MLP are just `N_INPUT * WIDTH`.
  
  For some reason I thought the hidden layers would be fixed size and/or require a fixed size input, but this is not the case since it turns out that `N_INPUT` is always the first dimension, except in the result of `Q * K^T` which has dimension `N_INPUT * N_INPUT`, but is only multiplied on the right by `V` which has first dimension `N_INPUT`.
  
  In the NNs that I'd learned about previously, N\_INPUT is the second dimension so even if one were to put in a variable input, all of the hidden layers would still be fixed-size.
  
  Particularly what I did not realize is that the "context window" is not actually a part of the model ([except in the positional encoding](https://ai.stackexchange.com/a/37289/68775)). It seems to be purely a training time / inference time restriction.
  
  
  ---
  
  