Answers.Body: <p>Kaplan et al. (2020) <a href="https://arxiv.org/abs/2001.08361" rel="nofollow noreferrer">&quot;<em>Scaling Laws for Neural Language Models</em>&quot;</a> established fundamental relationships between model size, compute, performance and showed power-law relationships in model scaling as OpenAI's seminal work on model scaling.</p>
  <blockquote>
  <p>The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range... Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models
  on a relatively modest amount of data and stopping significantly before convergence... Model size (ranging in size from 768 to 1.5 billion non-embedding parameters)... Transformer performance depends very weakly on the shape parameters <span class="math-container">$n_{layer}$</span>, <span class="math-container">$n_{heads}$</span>, and <span class="math-container">$d_{ff}$</span> when we hold the total non-embedding parameter count <span class="math-container">$N$</span> fixed.</p>
  </blockquote>
  <p>DeepMind's Hoffmann et al. (2022) <a href="https://arxiv.org/pdf/2203.15556" rel="nofollow noreferrer">&quot;<em>Training Compute-Optimal Large Language Models</em>&quot;</a> argues many models are actually undertrained and suggested 70B parameters model trained on 1.4T tokens is better than 280B parameters model with 1/4 training tokens.</p>
  <blockquote>
  <p>We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant...  We test this hypothesis by training a predicted computeoptimal model, Chinchilla, that uses the same compute budget as Gopher but with 70B parameters and 4× more more data. Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B),
  Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks.</p>
  </blockquote>
  <p>Recent <a href="https://arxiv.org/abs/2401.02385" rel="nofollow noreferrer">research</a> also argues tiny model with 1.1B parameters trained on 1T tokens could generate surprising capabilities.</p>
  <blockquote>
  <p>We present TinyLlama, a compact 1.1B language model pretrained on around 1 trillion tokens for approximately 3 epochs... Despite its relatively small size, TinyLlama demonstrates remarkable performance in a series of downstream tasks. It significantly outperforms existing open-source language models with comparable sizes.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-26T19:05:28.870Z
Answers.LastActivityDate: 2024-11-26T19:05:28.870Z
Answers.UserDisplayName: cinch
Body: <p>Given how model size seems to be a big determinant for both performance and impact (electricity / CO2 costs), I was wondering if there is some theoretical bound for model size. That is, without a particular purpose in mind, I would expect to have some measure of language complexity, and thus a minimal number of parameters needed to represent that complexity.</p>
  <p>Is there some meaningful work in this domain?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-11-27T10:19:49.773Z
Comments.Text: What do you mean by "meaningful work"?
Comments.UserDisplayName: nbro
Tags: <reference-request><large-language-models><computational-learning-theory>
Title: Lower Bound for LLM size?
UserDisplayName: Lucas Morin
ThreadText: # Lower Bound for LLM size?
  
  (Asked by: username_0 on 2024-11-26)
  
  Given how model size seems to be a big determinant for both performance and impact (electricity / CO2 costs), I was wondering if there is some theoretical bound for model size. That is, without a particular purpose in mind, I would expect to have some measure of language complexity, and thus a minimal number of parameters needed to represent that complexity.
  
  Is there some meaningful work in this domain?
  
  **Tags:** reference-request, large-language-models, computational-learning-theory
  
  > **Comments:**
  > **username_1**: What do you mean by "meaningful work"?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-11-26. Score: 0
  
  Kaplan et al. (2020) ["*Scaling Laws for Neural Language Models*"](https://arxiv.org/abs/2001.08361) established fundamental relationships between model size, compute, performance and showed power-law relationships in model scaling as OpenAI's seminal work on model scaling.
  
  > The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range... Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models
  > on a relatively modest amount of data and stopping significantly before convergence... Model size (ranging in size from 768 to 1.5 billion non-embedding parameters)... Transformer performance depends very weakly on the shape parameters $n\_{layer}$, $n\_{heads}$, and $d\_{ff}$ when we hold the total non-embedding parameter count $N$ fixed.
  
  DeepMind's Hoffmann et al. (2022) ["*Training Compute-Optimal Large Language Models*"](https://arxiv.org/pdf/2203.15556) argues many models are actually undertrained and suggested 70B parameters model trained on 1.4T tokens is better than 280B parameters model with 1/4 training tokens.
  
  > We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant... We test this hypothesis by training a predicted computeoptimal model, Chinchilla, that uses the same compute budget as Gopher but with 70B parameters and 4× more more data. Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B),
  > Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks.
  
  Recent [research](https://arxiv.org/abs/2401.02385) also argues tiny model with 1.1B parameters trained on 1T tokens could generate surprising capabilities.
  
  > We present TinyLlama, a compact 1.1B language model pretrained on around 1 trillion tokens for approximately 3 epochs... Despite its relatively small size, TinyLlama demonstrates remarkable performance in a series of downstream tasks. It significantly outperforms existing open-source language models with comparable sizes.
  
  
  ---
  
  