Loss-to-Loss Prediction: Scaling Laws for All Datasets
While scaling laws provide a reliable methodology for predicting train loss
across compute scales for a single data distribution, less is known about how
these predictions should change as we change the distribution. In this paper,
we derive a strategy for predicting one loss from another and apply it to
predict across different pre-training datasets and from pre-training data to
downstream task data. Our predictions extrapolate well even at 20x the largest
FLOP budget used to fit the curves. More precisely, we find that there are
simple shifted power law relationships between (1) the train losses of two
models trained on two separate datasets when the models are paired by training
compute (train-to-train), (2) the train loss and the test loss on any
downstream distribution for a single model (train-to-test), and (3) the test
losses of two models trained on two separate train datasets (test-to-test). The
results hold up for pre-training datasets that differ substantially (some are
entirely code and others have no code at all) and across a variety of
downstream tasks. Finally, we find that in some settings these shifted power
law relationships can yield more accurate predictions than extrapolating
single-dataset scaling laws.
673fa5f0628151f1a4841454
David Brandfonbrener
extracted_confirmed
2024-11-21T21:30:59.999Z
6314bdae4f283a6af50711da
/avatars/ca3fcc8609b492124b4f97fab29a63f1.svg
David Brandfonbrener
user
davidbrandfonbrener
673fa5f0628151f1a4841455
Nikhil Anand
673fa5f0628151f1a4841456
Nikhil Vyas
673fa5f0628151f1a4841457
Eran Malach
extracted_pending
2024-11-21T21:28:18.086Z
637b41b756db0404b7c615a5
/avatars/ce33c956900cc711a237b532c12bdc66.svg
Eran Malach
user
emalach
673fa5f0628151f1a4841458
Sham Kakade
Loss-to-Loss Prediction: Scaling Laws for All Datasets
While scaling laws provide a reliable methodology for predicting train loss
across compute scales for a single data distribution, less is known about how
these predictions should change as we change the distribution. In this paper,
we derive a strategy for predicting one loss from another and apply it to
predict across different pre-training datasets and from pre-training data to
downstream task data. Our predictions extrapolate well even at 20x the largest
FLOP budget used to fit the curves. More precisely, we find that there are
simple shifted power law relationships between (1) the train losses of two
models trained on two separate datasets when the models are paired by training
compute (train-to-train), (2) the train loss and the test loss on any
downstream distribution for a single model (train-to-test), and (3) the test
losses of two models trained on two separate train datasets (test-to-test). The
results hold up for pre-training datasets that differ substantially (some are
entirely code and others have no code at all) and across a variety of
downstream tasks. Finally, we find that in some settings these shifted power
law relationships can yield more accurate predictions than extrapolating
single-dataset scaling laws.
The paper presents a strategy to predict model performance across different datasets and tasks using shifted power law relationships derived from scaling laws.
scaling laws train loss FLOP budget power law relationships train-to-train train-to-test test-to-test