title: Establishing Task Scaling Laws via Compute-Efficient Model Ladders
summary: We develop task scaling laws and model ladders to predict the individual task
  performance of pretrained language models (LMs) in the overtrained setting.
  Standard power laws for language modeling loss cannot accurately model task
  performance. Therefore, we leverage a two-step prediction approach: first use
  model and data size to predict a task-specific loss, and then use this task
  loss to predict task performance. We train a set of small-scale "ladder"
  models, collect data points to fit the parameterized functions of the two
  prediction steps, and make predictions for two target models: a 7B model
  trained to 4T tokens and a 13B model trained to 5T tokens. Training the ladder
  models only costs 1% of the compute used for the target models. On four
  multiple-choice tasks written in ranked classification format, we can predict
  the accuracy of both target models within 2 points of absolute error. We have
  higher prediction error on four other tasks (average absolute error 6.9) and
  find that these are often tasks with higher variance in task metrics. We also
  find that using less compute to train fewer ladder models tends to deteriorate
  predictions. Finally, we empirically show that our design choices and the
  two-step approach lead to superior performance in establishing scaling laws.
paper_authors._id: 67538c699b57b4baf0e752af
paper_authors.name: Akshita Bhagia
paper_authors._id: 67538c699b57b4baf0e752b0
paper_authors.name: Jiacheng Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-07T08:42:58.257Z
paper_authors._id: 635f46d1928a42bc95cfcf7c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/635f46d1928a42bc95cfcf7c/5KF8aLiDCJdl7B1SdJ-7V.png
paper_authors.fullname: Jiacheng Liu
paper_authors.type: user
paper_authors.user: liujch1998
paper_authors._id: 67538c699b57b4baf0e752b1
paper_authors.name: Alexander Wettig
paper_authors._id: 67538c699b57b4baf0e752b2
paper_authors.name: David Heineman
paper_authors._id: 67538c699b57b4baf0e752b3
paper_authors.name: Oyvind Tafjord
paper_authors._id: 67538c699b57b4baf0e752b4
paper_authors.name: Ananya Harsh Jha
paper_authors._id: 67538c699b57b4baf0e752b5
paper_authors.name: Luca Soldaini
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-20T08:33:56.749Z
paper_authors._id: 5f04d8c45d08220171a0ad32
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5f04d8c45d08220171a0ad32/uXEta6nqBabrUlAOXnS5g.jpeg
paper_authors.fullname: Luca Soldaini
paper_authors.type: user
paper_authors.user: soldni
paper_authors._id: 67538c699b57b4baf0e752b6
paper_authors.name: Noah A. Smith
paper_authors._id: 67538c699b57b4baf0e752b7
paper_authors.name: Dirk Groeneveld
paper_authors._id: 67538c699b57b4baf0e752b8
paper_authors.name: Pang Wei Koh
paper_authors._id: 67538c699b57b4baf0e752b9
paper_authors.name: Jesse Dodge
paper_authors._id: 67538c699b57b4baf0e752ba
paper_authors.name: Hannaneh Hajishirzi
paper_title: Establishing Task Scaling Laws via Compute-Efficient Model Ladders
paper_summary: We develop task scaling laws and model ladders to predict the individual task
  performance of pretrained language models (LMs) in the overtrained setting.
  Standard power laws for language modeling loss cannot accurately model task
  performance. Therefore, we leverage a two-step prediction approach: first use
  model and data size to predict a task-specific loss, and then use this task
  loss to predict task performance. We train a set of small-scale "ladder"
  models, collect data points to fit the parameterized functions of the two
  prediction steps, and make predictions for two target models: a 7B model
  trained to 4T tokens and a 13B model trained to 5T tokens. Training the ladder
  models only costs 1% of the compute used for the target models. On four
  multiple-choice tasks written in ranked classification format, we can predict
  the accuracy of both target models within 2 points of absolute error. We have
  higher prediction error on four other tasks (average absolute error 6.9) and
  find that these are often tasks with higher variance in task metrics. We also
  find that using less compute to train fewer ladder models tends to deteriorate
  predictions. Finally, we empirically show that our design choices and the
  two-step approach lead to superior performance in establishing scaling laws.
paper_ai_keywords: task scaling laws model ladders pretrained language models (LMs) overtrained setting power laws for language modeling loss task performance task-specific loss parameterized functions ladder models multiple-choice format ranked classification format accuracy