Answers.Body: <blockquote>
  <p>is the set of values estimated by the Chinchilla scaling laws optimal for these smaller models with optimized data too?</p>
  </blockquote>
  <p>This is an open research question, but some research such as <a href="https://arxiv.org/abs/2206.14486" rel="nofollow noreferrer">Beyond neural scaling laws: beating power law scaling via data pruning</a> and <a href="https://arxiv.org/abs/2305.07759" rel="nofollow noreferrer">Tiny Data</a> seem to imply that higher quality data gives better scaling laws than Chinchilla estimates.</p>
  <blockquote>
  <p>Do we have any model to estimate the quality of some datasets and some scaling laws that take it into account?</p>
  </blockquote>
  <p>Not yet!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-07T02:39:21.270Z
Answers.LastActivityDate: 2023-11-07T02:39:21.270Z
Answers.UserDisplayName: Stella Biderman
Body: <p>DeepMind released the <a href="https://arxiv.org/abs/2203.15556" rel="nofollow noreferrer">Training Compute-Optimal Large Language Models</a> paper in 2022 which describe some scaling laws for LLMs. As far as I understand this is the most accredited reference to estimate the optimal relation between dataset size, compute power and model size.</p>
  <p>Recently a number of models have been developed using far less data, parameters and compute than the bigger LLMs. Yet these models achieved great results thanks to much better data quality. For instance models like <a href="https://arxiv.org/abs/2304.12244" rel="nofollow noreferrer">WizardLM</a>, <a href="https://arxiv.org/abs/2305.07759" rel="nofollow noreferrer">TinyStories</a> and <a href="https://arxiv.org/abs/2306.11644" rel="nofollow noreferrer">phi-1</a>. The recent article published on nature <a href="https://www.nature.com/articles/s41586-023-06668-3" rel="nofollow noreferrer">Human-like systematic generalization through a meta-learning neural network</a> seems to imply that data quality is the key.</p>
  <p>I'm curious about what role the data quality plays in the training of LLMs. I'm wondering things like: is the set of values estimated by the Chinchilla scaling laws optimal for these smaller models with optimized data too? Do we have any model to estimate the quality of some datasets and some scaling laws that take it into account?</p>
  
Tags: <datasets><large-language-models>
Title: What role does data quality plays in the LLM scaling laws?
UserDisplayName: Blue Nebula
ThreadText: # What role does data quality plays in the LLM scaling laws?
  
  (Asked by: username_0 on 2023-07-12)
  
  DeepMind released the [Training Compute-Optimal Large Language Models](https://arxiv.org/abs/2203.15556) paper in 2022 which describe some scaling laws for LLMs. As far as I understand this is the most accredited reference to estimate the optimal relation between dataset size, compute power and model size.
  
  Recently a number of models have been developed using far less data, parameters and compute than the bigger LLMs. Yet these models achieved great results thanks to much better data quality. For instance models like [WizardLM](https://arxiv.org/abs/2304.12244), [TinyStories](https://arxiv.org/abs/2305.07759) and [phi-1](https://arxiv.org/abs/2306.11644). The recent article published on nature [Human-like systematic generalization through a meta-learning neural network](https://www.nature.com/articles/s41586-023-06668-3) seems to imply that data quality is the key.
  
  I'm curious about what role the data quality plays in the training of LLMs. I'm wondering things like: is the set of values estimated by the Chinchilla scaling laws optimal for these smaller models with optimized data too? Do we have any model to estimate the quality of some datasets and some scaling laws that take it into account?
  
  **Tags:** datasets, large-language-models
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-11-07. Score: 3
  
  > is the set of values estimated by the Chinchilla scaling laws optimal for these smaller models with optimized data too?
  
  This is an open research question, but some research such as [Beyond neural scaling laws: beating power law scaling via data pruning](https://arxiv.org/abs/2206.14486) and [Tiny Data](https://arxiv.org/abs/2305.07759) seem to imply that higher quality data gives better scaling laws than Chinchilla estimates.
  
  > Do we have any model to estimate the quality of some datasets and some scaling laws that take it into account?
  
  Not yet!
  
  
  ---
  
  