Answers.Body: <p>At first I thought that they were just leaving some headroom by using less-than power-of-two number of parameters, but 33 and 65 actually break this pattern. But I still think it is memory-usage related and they aim to have the full system working at 16, 32, 64 etc. GBs of memory.</p>
  <p>There are some hints at <a href="https://web.archive.org/web/20230530172538/https://github.com/facebookresearch/llama/blob/main/FAQ.md#3" rel="nofollow noreferrer">their GitHub repository</a> (sunset as of 2024)</p>
  <blockquote>
  <p>Accounting for 14GB of memory for the model weights (7B model), this leaves 16GB available for the decoding cache which stores 2 * 2 * n_layers * max_batch_size * max_seq_len * n_heads * head_dim bytes.</p>
  <p>With default parameters, this cache was about 17GB (2 * 2 * 32 * 32 * 1024 * 32 * 128) for the 7B model.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-08T12:31:51.293Z
Answers.LastActivityDate: 2024-11-05T14:03:29.667Z
Answers.LastEditDate: 2024-11-05T14:03:29.667Z
Answers.UserDisplayName: NikoNyrh
Answers.Body: <p>I don't think developers of Llama (and LLMs in general) always work backwards from a chosen model size. They instead choose model hyperparameters such as model dimension, vocabulary size, and number of decoder blocks. For Llama 3, these numbers are 4096, ~128K, and 32.</p>
  <p>Model size is then a consequence of chosen hyperparameters.</p>
  <p>Llama 3 8B model for example has exactly 8,030,257,152 trainable parameters. See <a href="https://adithyask.medium.com/from-7b-to-8b-parameters-understanding-weight-matrix-changes-in-llama-transformer-models-31ea7ed5fd88" rel="nofollow noreferrer">this post</a> for complete derivation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-29T19:09:22.737Z
Answers.LastActivityDate: 2024-12-29T19:09:22.737Z
Answers.UserDisplayName: Fijoy Vadakkumpadan
Body: <p><a href="https://ai.meta.com/blog/large-language-model-llama-meta-ai/" rel="nofollow noreferrer">LLaMa</a> was released in several sizes, with 7B, 13B, 33B, and 65B parameters. These values look a little weird, because they are very close to powers of two (8, 16, 32, 64) which are more conventionally considered “round numbers” in software. Why were these specific numbers chosen?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-06T10:07:22.350Z
Comments.Text: Can you please provide a reference that explains what these "round numbers" are in "software"? I've been programming for a few years and I've never encountered such an expression, which, of course, doesn't mean that it doesn't exist or doesn't define a value concept.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-11-04T14:30:56.477Z
Comments.Text: @nbro [round numbers](https://en.m.wikipedia.org/wiki/Round_number)
Comments.UserDisplayName: Him
Tags: <large-language-models>
Title: Why do LLaMa and its variants have non-“round” numbers of parameters?
UserDisplayName: amh
ThreadText: # Why do LLaMa and its variants have non-“round” numbers of parameters?
  
  (Asked by: username_0 on 2023-06-05)
  
  [LLaMa](https://ai.meta.com/blog/large-language-model-llama-meta-ai/) was released in several sizes, with 7B, 13B, 33B, and 65B parameters. These values look a little weird, because they are very close to powers of two (8, 16, 32, 64) which are more conventionally considered “round numbers” in software. Why were these specific numbers chosen?
  
  **Tags:** large-language-models
  
  > **Comments:**
  > **username_1**: Can you please provide a reference that explains what these "round numbers" are in "software"? I've been programming for a few years and I've never encountered such an expression, which, of course, doesn't mean that it doesn't exist or doesn't define a value concept.  
  > **username_2**: @username_1 [round numbers](https://en.m.wikipedia.org/wiki/Round_number)  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2023-06-08. Score: 1
  
  At first I thought that they were just leaving some headroom by using less-than power-of-two number of parameters, but 33 and 65 actually break this pattern. But I still think it is memory-usage related and they aim to have the full system working at 16, 32, 64 etc. GBs of memory.
  
  There are some hints at [their GitHub repository](https://web.archive.org/web/20230530172538/https://github.com/facebookresearch/llama/blob/main/FAQ.md#3) (sunset as of 2024)
  
  > Accounting for 14GB of memory for the model weights (7B model), this leaves 16GB available for the decoding cache which stores 2 \* 2 \* n\_layers \* max\_batch\_size \* max\_seq\_len \* n\_heads \* head\_dim bytes.
  >
  > With default parameters, this cache was about 17GB (2 \* 2 \* 32 \* 32 \* 1024 \* 32 \* 128) for the 7B model.
  
  
  ---
  
  ### Answer by username_4 on 2024-12-29. Score: 0
  
  I don't think developers of Llama (and LLMs in general) always work backwards from a chosen model size. They instead choose model hyperparameters such as model dimension, vocabulary size, and number of decoder blocks. For Llama 3, these numbers are 4096, ~128K, and 32.
  
  Model size is then a consequence of chosen hyperparameters.
  
  Llama 3 8B model for example has exactly 8,030,257,152 trainable parameters. See [this post](https://adithyask.medium.com/from-7b-to-8b-parameters-understanding-weight-matrix-changes-in-llama-transformer-models-31ea7ed5fd88) for complete derivation.
  
  
  ---
  
  