Answers.Body: <p>Transformers parameters count is invariant to the context window (which by definition can be infinite, though the <span class="math-container">$O(n^2)$</span> complexity might hurt)</p>
  <p>Consider that the context window that you see in the LLMs documentations usually refers to the one used in training, but nothing stops you from taking a model train with a 10k context window and using it with a 50k context window (but usually performances deplete quickly extending the context window)</p>
  <hr />
  <p>For the details, I'd suggest you to read the <a href="https://arxiv.org/abs/1706.03762" rel="nofollow noreferrer">original paper</a>, but the main point is the following:<br />
  Transformers are made by 2 main building blocks, attention and FFNN</p>
  <p>The FFNN applies a transformation element-wise, thus it's for sure able to apply it to any sequence length size</p>
  <p>The Attention block is nothing more than a (dynamic) linear combination of all the tokens... if you take the attention formula, and multiply the matrices, you will find that the output shape is the same as the input, thus making it length-free</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-28T12:03:55.053Z
Answers.Text: Isn't there a maximum number of input tokens specified prior to training in the normal transformer architecture?
Answers.UserDisplayName: Ggjj11
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-28T18:22:55.400Z
Answers.Text: @Ggjj11 no, it’s done mainly to make the training faster by using tensor operations, but no, you can feed any sequence to the same architecture
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-28T21:44:55.250Z
Answers.Text: Could you please help via describing in your answer why the model architecture theoretically can compute it's output even with arbitrary length inputs (let's say for num_tokens 100,1000,10000,...?) the attention output must work with arbitrary length of this dimension (which is true, because the self-attention matrix just dynamically could scale to nxn entries). But then ... I think you are right, but it would be nice to see a written explanation with all the details taking into account the classification head for next token prediction :)
Answers.UserDisplayName: Ggjj11
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-28T11:06:34.003Z
Answers.LastActivityDate: 2024-04-29T00:51:29.990Z
Answers.LastEditDate: 2024-04-29T00:51:29.990Z
Answers.UserDisplayName: Alberto
Body: <p>What the question says.</p>
  <p>In a transformer architecture, is there a relationship between number of tokens and number of parameters?</p>
  <p>Can you have a LLM with a small number of parameters but a large context window (many tokens) or viceversa?</p>
  
Tags: <transformer><large-language-models>
Title: Is there a relationship between tokens and parameters in LLMs?
UserDisplayName: A. Darwin
ThreadText: # Is there a relationship between tokens and parameters in LLMs?
  
  (Asked by: username_0 on 2024-04-27)
  
  What the question says.
  
  In a transformer architecture, is there a relationship between number of tokens and number of parameters?
  
  Can you have a LLM with a small number of parameters but a large context window (many tokens) or viceversa?
  
  **Tags:** transformer, large-language-models
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-04-28. Score: 4 | Accepted answer
  
  Transformers parameters count is invariant to the context window (which by definition can be infinite, though the $O(n^2)$ complexity might hurt)
  
  Consider that the context window that you see in the LLMs documentations usually refers to the one used in training, but nothing stops you from taking a model train with a 10k context window and using it with a 50k context window (but usually performances deplete quickly extending the context window)
  
  ---
  
  For the details, I'd suggest you to read the [original paper](https://arxiv.org/abs/1706.03762), but the main point is the following:  
  Transformers are made by 2 main building blocks, attention and FFNN
  
  The FFNN applies a transformation element-wise, thus it's for sure able to apply it to any sequence length size
  
  The Attention block is nothing more than a (dynamic) linear combination of all the tokens... if you take the attention formula, and multiply the matrices, you will find that the output shape is the same as the input, thus making it length-free
  
  > **Comments:**
  > **username_2**: Isn't there a maximum number of input tokens specified prior to training in the normal transformer architecture?  
  > **username_1**: @Ggjj11 no, it’s done mainly to make the training faster by using tensor operations, but no, you can feed any sequence to the same architecture  
  > **username_2**: Could you please help via describing in your answer why the model architecture theoretically can compute it's output even with arbitrary length inputs (let's say for num_tokens 100,1000,10000,...?) the attention output must work with arbitrary length of this dimension (which is true, because the self-attention matrix just dynamically could scale to nxn entries). But then ... I think you are right, but it would be nice to see a written explanation with all the details taking into account the classification head for next token prediction :)  
  
  ---
  
  