title: Predicting Emergent Capabilities by Finetuning
summary: A fundamental open challenge in modern LLM scaling is the lack of
  understanding around emergent capabilities. In particular, language model
  pretraining loss is known to be highly predictable as a function of compute.
  However, downstream capabilities are far less predictable -- sometimes even
  exhibiting emergent jumps -- which makes it challenging to anticipate the
  capabilities of future models. In this work, we first pose the task of
  emergence prediction: given access to current LLMs that have random few-shot
  accuracy on a task, can we predict whether future models (GPT-N+1) will have
  non-trivial accuracy on that task? We then discover a simple insight for this
  problem: finetuning LLMs on a given task can shift the point in scaling at
  which emergence occurs towards less capable models. To operationalize this
  insight, we can finetune LLMs with varying amounts of data and fit a parametric
  function that predicts when emergence will occur (i.e., "emergence laws"). We
  validate this approach using four standard NLP benchmarks where large-scale
  open-source LLMs already demonstrate emergence (MMLU, GSM8K, CommonsenseQA, and
  CoLA). Using only small-scale LLMs, we find that, in some cases, we can
  accurately predict whether models trained with up to 4x more compute have
  emerged. Finally, we present a case study of two realistic uses for emergence
  prediction.
paper_authors._id: 674550443dd665eec817d2f3
paper_authors.name: Charlie Snell
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-11-26T04:49:22.384Z
paper_authors._id: 605d07fa8a3450814bada877
paper_authors.avatarUrl: /avatars/eafe986982057fbaba962b99d5543477.svg
paper_authors.fullname: Charlie Snell
paper_authors.type: user
paper_authors.user: sea-snell
paper_authors._id: 674550443dd665eec817d2f4
paper_authors.name: Eric Wallace
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-26T22:58:54.291Z
paper_authors._id: 63112d2431257261d20d5754
paper_authors.avatarUrl: /avatars/502a68c0fb2f0c6989fe2869d0a7e3f4.svg
paper_authors.fullname: Eric Wallace
paper_authors.type: user
paper_authors.user: EricWallace
paper_authors._id: 674550443dd665eec817d2f5
paper_authors.name: Dan Klein
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-26T22:59:01.446Z
paper_authors._id: 66e1fbf2a8e06d2e99c7aac1
paper_authors.avatarUrl: /avatars/13939a47f995a66a4ab32004c8cc21d0.svg
paper_authors.fullname: Daniel Klein
paper_authors.type: user
paper_authors.user: danklein
paper_authors._id: 674550443dd665eec817d2f6
paper_authors.name: Sergey Levine
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-26T22:59:06.568Z
paper_authors._id: 665ce54120a307a3754849dd
paper_authors.avatarUrl: /avatars/e698726e9be61dd50ce2efe372ed5dac.svg
paper_authors.fullname: Sergey Levine
paper_authors.type: user
paper_authors.user: svlevine
paper_title: Predicting Emergent Capabilities by Finetuning
paper_summary: A fundamental open challenge in modern LLM scaling is the lack of
  understanding around emergent capabilities. In particular, language model
  pretraining loss is known to be highly predictable as a function of compute.
  However, downstream capabilities are far less predictable -- sometimes even
  exhibiting emergent jumps -- which makes it challenging to anticipate the
  capabilities of future models. In this work, we first pose the task of
  emergence prediction: given access to current LLMs that have random few-shot
  accuracy on a task, can we predict whether future models (GPT-N+1) will have
  non-trivial accuracy on that task? We then discover a simple insight for this
  problem: finetuning LLMs on a given task can shift the point in scaling at
  which emergence occurs towards less capable models. To operationalize this
  insight, we can finetune LLMs with varying amounts of data and fit a parametric
  function that predicts when emergence will occur (i.e., "emergence laws"). We
  validate this approach using four standard NLP benchmarks where large-scale
  open-source LLMs already demonstrate emergence (MMLU, GSM8K, CommonsenseQA, and
  CoLA). Using only small-scale LLMs, we find that, in some cases, we can
  accurately predict whether models trained with up to 4x more compute have
  emerged. Finally, we present a case study of two realistic uses for emergence
  prediction.
paper_ai_keywords: language model pretraining loss compute downstream capabilities emergence few-shot accuracy finetuning parametric function emergence laws MMLU GSM8K conceptsenseQA CoLA large-scale open-source LLMs emergence prediction