title: Loss-to-Loss Prediction: Scaling Laws for All Datasets
summary: While scaling laws provide a reliable methodology for predicting train loss
  across compute scales for a single data distribution, less is known about how
  these predictions should change as we change the distribution. In this paper,
  we derive a strategy for predicting one loss from another and apply it to
  predict across different pre-training datasets and from pre-training data to
  downstream task data. Our predictions extrapolate well even at 20x the largest
  FLOP budget used to fit the curves. More precisely, we find that there are
  simple shifted power law relationships between (1) the train losses of two
  models trained on two separate datasets when the models are paired by training
  compute (train-to-train), (2) the train loss and the test loss on any
  downstream distribution for a single model (train-to-test), and (3) the test
  losses of two models trained on two separate train datasets (test-to-test). The
  results hold up for pre-training datasets that differ substantially (some are
  entirely code and others have no code at all) and across a variety of
  downstream tasks. Finally, we find that in some settings these shifted power
  law relationships can yield more accurate predictions than extrapolating
  single-dataset scaling laws.
paper_authors._id: 673fa5f0628151f1a4841454
paper_authors.name: David Brandfonbrener
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-11-21T21:30:59.999Z
paper_authors._id: 6314bdae4f283a6af50711da
paper_authors.avatarUrl: /avatars/ca3fcc8609b492124b4f97fab29a63f1.svg
paper_authors.fullname: David Brandfonbrener
paper_authors.type: user
paper_authors.user: davidbrandfonbrener
paper_authors._id: 673fa5f0628151f1a4841455
paper_authors.name: Nikhil Anand
paper_authors._id: 673fa5f0628151f1a4841456
paper_authors.name: Nikhil Vyas
paper_authors._id: 673fa5f0628151f1a4841457
paper_authors.name: Eran Malach
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-11-21T21:28:18.086Z
paper_authors._id: 637b41b756db0404b7c615a5
paper_authors.avatarUrl: /avatars/ce33c956900cc711a237b532c12bdc66.svg
paper_authors.fullname: Eran Malach
paper_authors.type: user
paper_authors.user: emalach
paper_authors._id: 673fa5f0628151f1a4841458
paper_authors.name: Sham Kakade
paper_title: Loss-to-Loss Prediction: Scaling Laws for All Datasets
paper_summary: While scaling laws provide a reliable methodology for predicting train loss
  across compute scales for a single data distribution, less is known about how
  these predictions should change as we change the distribution. In this paper,
  we derive a strategy for predicting one loss from another and apply it to
  predict across different pre-training datasets and from pre-training data to
  downstream task data. Our predictions extrapolate well even at 20x the largest
  FLOP budget used to fit the curves. More precisely, we find that there are
  simple shifted power law relationships between (1) the train losses of two
  models trained on two separate datasets when the models are paired by training
  compute (train-to-train), (2) the train loss and the test loss on any
  downstream distribution for a single model (train-to-test), and (3) the test
  losses of two models trained on two separate train datasets (test-to-test). The
  results hold up for pre-training datasets that differ substantially (some are
  entirely code and others have no code at all) and across a variety of
  downstream tasks. Finally, we find that in some settings these shifted power
  law relationships can yield more accurate predictions than extrapolating
  single-dataset scaling laws.
paper_ai_keywords: shifted power law relationships train-to-train train-to-test test-to-test