Establishing Task Scaling Laws via Compute-Efficient Model Ladders
We develop task scaling laws and model ladders to predict the individual task
performance of pretrained language models (LMs) in the overtrained setting.
Standard power laws for language modeling loss cannot accurately model task
performance. Therefore, we leverage a two-step prediction approach: first use
model and data size to predict a task-specific loss, and then use this task
loss to predict task performance. We train a set of small-scale "ladder"
models, collect data points to fit the parameterized functions of the two
prediction steps, and make predictions for two target models: a 7B model
trained to 4T tokens and a 13B model trained to 5T tokens. Training the ladder
models only costs 1% of the compute used for the target models. On four
multiple-choice tasks written in ranked classification format, we can predict
the accuracy of both target models within 2 points of absolute error. We have
higher prediction error on four other tasks (average absolute error 6.9) and
find that these are often tasks with higher variance in task metrics. We also
find that using less compute to train fewer ladder models tends to deteriorate
predictions. Finally, we empirically show that our design choices and the
two-step approach lead to superior performance in establishing scaling laws.
67538c699b57b4baf0e752af
Akshita Bhagia
claimed_verified
2025-04-18T09:41:57.482Z
6282937a9624ab5f75c977e1
https://cdn-avatars.huggingface.co/v1/production/uploads/6282937a9624ab5f75c977e1/ZOysQbGzEzmrN_SaTQgOP.jpeg
Akshita Bhagia
user
akshitab
67538c699b57b4baf0e752b0
Jiacheng Liu
claimed_verified
2025-01-07T08:42:58.257Z
635f46d1928a42bc95cfcf7c
https://cdn-avatars.huggingface.co/v1/production/uploads/635f46d1928a42bc95cfcf7c/5KF8aLiDCJdl7B1SdJ-7V.png
Jiacheng Liu
user
liujch1998
67538c699b57b4baf0e752b1
Alexander Wettig
67538c699b57b4baf0e752b2
David Heineman
claimed_verified
2025-04-22T10:17:18.572Z
6105c8136377c5f8ef2494b1
https://cdn-avatars.huggingface.co/v1/production/uploads/6105c8136377c5f8ef2494b1/mlifkrHYMhXlXSa9DFbDh.jpeg
David Heineman
user
davidheineman
67538c699b57b4baf0e752b3
Oyvind Tafjord
67538c699b57b4baf0e752b4
Ananya Harsh Jha
67538c699b57b4baf0e752b5
Luca Soldaini
claimed_verified
2024-12-20T08:33:56.749Z
5f04d8c45d08220171a0ad32
https://cdn-avatars.huggingface.co/v1/production/uploads/5f04d8c45d08220171a0ad32/uXEta6nqBabrUlAOXnS5g.jpeg
Luca Soldaini
user
soldni
67538c699b57b4baf0e752b6
Noah A. Smith
67538c699b57b4baf0e752b7
Dirk Groeneveld
67538c699b57b4baf0e752b8
Pang Wei Koh
67538c699b57b4baf0e752b9
Jesse Dodge
67538c699b57b4baf0e752ba
Hannaneh Hajishirzi
Establishing Task Scaling Laws via Compute-Efficient Model Ladders
We develop task scaling laws and model ladders to predict the individual task
performance of pretrained language models (LMs) in the overtrained setting.
Standard power laws for language modeling loss cannot accurately model task
performance. Therefore, we leverage a two-step prediction approach: first use
model and data size to predict a task-specific loss, and then use this task
loss to predict task performance. We train a set of small-scale "ladder"
models, collect data points to fit the parameterized functions of the two
prediction steps, and make predictions for two target models: a 7B model
trained to 4T tokens and a 13B model trained to 5T tokens. Training the ladder
models only costs 1% of the compute used for the target models. On four
multiple-choice tasks written in ranked classification format, we can predict
the accuracy of both target models within 2 points of absolute error. We have
higher prediction error on four other tasks (average absolute error 6.9) and
find that these are often tasks with higher variance in task metrics. We also
find that using less compute to train fewer ladder models tends to deteriorate
predictions. Finally, we empirically show that our design choices and the
two-step approach lead to superior performance in establishing scaling laws.
A two-step prediction approach using task-specific loss models improves the prediction of pretrained language model performance on various tasks.
task scaling laws model ladders pretrained language models language modeling loss multiple-choice tasks ranked classification format target models compute cost variance in task metrics