title: DataDecide: How to Predict Best Pretraining Data with Small Experiments
summary: Because large language models are expensive to pretrain on different
  datasets, using smaller-scale experiments to decide on data is crucial for
  reducing costs. Which benchmarks and methods of making decisions from observed
  performance at small scale most accurately predict the datasets that yield the
  best large models? To empower open exploration of this question, we release
  models, data, and evaluations in DataDecide -- the most extensive open suite of
  models over differences in data and scale. We conduct controlled pretraining
  experiments across 25 corpora with differing sources, deduplication, and
  filtering up to 100B tokens, model sizes up to 1B parameters, and 3 random
  seeds. We find that the ranking of models at a single, small size (e.g., 150M
  parameters) is a strong baseline for predicting best models at our larger
  target scale (1B) (~80% of com parisons correct). No scaling law methods among
  8 baselines exceed the compute-decision frontier of single-scale predictions,
  but DataDecide can measure improvement in future scaling laws. We also identify
  that using continuous likelihood metrics as proxies in small experiments makes
  benchmarks including MMLU, ARC, HellaSwag, MBPP, and HumanEval >80% predictable
  at the target 1B scale with just 0.01% of the compute.
paper_authors._id: 67fffbd0fd5600e97f8deeff
paper_authors.name: Ian Magnusson
paper_authors._id: 67fffbd0fd5600e97f8def00
paper_authors.name: Nguyen Tai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-22T10:14:10.388Z
paper_authors._id: 6359a17e3568d26d3f0c4f23
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6359a17e3568d26d3f0c4f23/l523tmkBiZ9a-lGUMz8g8.jpeg
paper_authors.fullname: Tai Nguyen
paper_authors.type: user
paper_authors.user: taidng
paper_authors._id: 67fffbd0fd5600e97f8def01
paper_authors.name: Ben Bogin
paper_authors._id: 67fffbd0fd5600e97f8def02
paper_authors.name: David Heineman
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-22T10:14:07.187Z
paper_authors._id: 6105c8136377c5f8ef2494b1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6105c8136377c5f8ef2494b1/mlifkrHYMhXlXSa9DFbDh.jpeg
paper_authors.fullname: David Heineman
paper_authors.type: user
paper_authors.user: davidheineman
paper_authors._id: 67fffbd0fd5600e97f8def03
paper_authors.name: Jena D. Hwang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-05T07:29:23.920Z
paper_authors._id: 65c157eaf9957360824b0d87
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65c157eaf9957360824b0d87/c1NWTKsFLATJLR7W6Kmqz.jpeg
paper_authors.fullname: jena hwang
paper_authors.type: user
paper_authors.user: jenah
paper_authors._id: 67fffbd0fd5600e97f8def04
paper_authors.name: Luca Soldaini
paper_authors._id: 67fffbd0fd5600e97f8def05
paper_authors.name: Akshita Bhagia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-18T09:40:20.663Z
paper_authors._id: 6282937a9624ab5f75c977e1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6282937a9624ab5f75c977e1/ZOysQbGzEzmrN_SaTQgOP.jpeg
paper_authors.fullname: Akshita Bhagia
paper_authors.type: user
paper_authors.user: akshitab
paper_authors._id: 67fffbd0fd5600e97f8def06
paper_authors.name: Jiacheng Liu
paper_authors._id: 67fffbd0fd5600e97f8def07
paper_authors.name: Dirk Groeneveld
paper_authors._id: 67fffbd0fd5600e97f8def08
paper_authors.name: Oyvind Tafjord
paper_authors._id: 67fffbd0fd5600e97f8def09
paper_authors.name: Noah A. Smith
paper_authors._id: 67fffbd0fd5600e97f8def0a
paper_authors.name: Pang Wei Koh
paper_authors._id: 67fffbd0fd5600e97f8def0b
paper_authors.name: Jesse Dodge
paper_title: DataDecide: How to Predict Best Pretraining Data with Small Experiments
paper_summary: Because large language models are expensive to pretrain on different
  datasets, using smaller-scale experiments to decide on data is crucial for
  reducing costs. Which benchmarks and methods of making decisions from observed
  performance at small scale most accurately predict the datasets that yield the
  best large models? To empower open exploration of this question, we release
  models, data, and evaluations in DataDecide -- the most extensive open suite of
  models over differences in data and scale. We conduct controlled pretraining
  experiments across 25 corpora with differing sources, deduplication, and
  filtering up to 100B tokens, model sizes up to 1B parameters, and 3 random
  seeds. We find that the ranking of models at a single, small size (e.g., 150M
  parameters) is a strong baseline for predicting best models at our larger
  target scale (1B) (~80% of com parisons correct). No scaling law methods among
  8 baselines exceed the compute-decision frontier of single-scale predictions,
  but DataDecide can measure improvement in future scaling laws. We also identify
  that using continuous likelihood metrics as proxies in small experiments makes
  benchmarks including MMLU, ARC, HellaSwag, MBPP, and HumanEval >80% predictable
  at the target 1B scale with just 0.01% of the compute.
paper_ai_summary: Small-scale experiments accurately predict the best large language models when using continuous likelihood metrics as proxies.
paper_ai_keywords: large language models pretraining benchmarks scaling laws DataDecide MMLU ARC HellaSwag MBPP HumanEval continuous likelihood metrics