Answers.Body: <p>Not in terms of the definition in the linked post. That kind of dimensionality would refer to the dimensionality of the input.</p>
  <p>As discussed in post, there's also some ambiguity here, as data augmentation can change the dimensionality of the input. You could consider the dimensionality of the text input (although it might be hard to quantify the dimensionality of strings). You could consider the dimensionality of the text embeddings, which would be <code>max_input_length x embedding_dim</code>.</p>
  <p>Both of these are not <em>necessarily</em> related to the number of parameters. (E.g., you can have a model with 130b parameters but only takes inputs with 10 tokens or 1000 tokens)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-20T03:34:03.703Z
Answers.LastActivityDate: 2024-01-20T03:34:03.703Z
Answers.UserDisplayName: Alexander Wan
Body: <p>For example, would a Large Language Model (LLM) with parameter size 140 Billion have 140 Billion dimensions <a href="https://stats.stackexchange.com/questions/144770/how-to-determine-the-number-of-dimensions-to-a-machine-learning-problem">as defined in deep learning as the number of nodes in the input layer?</a></p>
  <p>Another way to ask this might be: Is 140B parameters the same as saying 140B nodes/dimensions in the encoder of the LLM?</p>
  
Tags: <deep-learning><large-language-models><dimensionality>
Title: Are LLM parameters synonymous with dimensions?
UserDisplayName: geominded
ThreadText: # Are LLM parameters synonymous with dimensions?
  
  (Asked by: username_0 on 2024-01-20)
  
  For example, would a Large Language Model (LLM) with parameter size 140 Billion have 140 Billion dimensions [as defined in deep learning as the number of nodes in the input layer?](https://stats.stackexchange.com/questions/144770/how-to-determine-the-number-of-dimensions-to-a-machine-learning-problem)
  
  Another way to ask this might be: Is 140B parameters the same as saying 140B nodes/dimensions in the encoder of the LLM?
  
  **Tags:** deep-learning, large-language-models, dimensionality
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-01-20. Score: 0
  
  Not in terms of the definition in the linked post. That kind of dimensionality would refer to the dimensionality of the input.
  
  As discussed in post, there's also some ambiguity here, as data augmentation can change the dimensionality of the input. You could consider the dimensionality of the text input (although it might be hard to quantify the dimensionality of strings). You could consider the dimensionality of the text embeddings, which would be `max_input_length x embedding_dim`.
  
  Both of these are not *necessarily* related to the number of parameters. (E.g., you can have a model with 130b parameters but only takes inputs with 10 tokens or 1000 tokens)
  
  
  ---
  
  