Predicting Emergent Capabilities by Finetuning
A fundamental open challenge in modern LLM scaling is the lack of
understanding around emergent capabilities. In particular, language model
pretraining loss is known to be highly predictable as a function of compute.
However, downstream capabilities are far less predictable -- sometimes even
exhibiting emergent jumps -- which makes it challenging to anticipate the
capabilities of future models. In this work, we first pose the task of
emergence prediction: given access to current LLMs that have random few-shot
accuracy on a task, can we predict whether future models (GPT-N+1) will have
non-trivial accuracy on that task? We then discover a simple insight for this
problem: finetuning LLMs on a given task can shift the point in scaling at
which emergence occurs towards less capable models. To operationalize this
insight, we can finetune LLMs with varying amounts of data and fit a parametric
function that predicts when emergence will occur (i.e., "emergence laws"). We
validate this approach using four standard NLP benchmarks where large-scale
open-source LLMs already demonstrate emergence (MMLU, GSM8K, CommonsenseQA, and
CoLA). Using only small-scale LLMs, we find that, in some cases, we can
accurately predict whether models trained with up to 4x more compute have
emerged. Finally, we present a case study of two realistic uses for emergence
prediction.
674550443dd665eec817d2f3
Charlie Snell
extracted_confirmed
2024-11-26T04:49:22.384Z
605d07fa8a3450814bada877
/avatars/eafe986982057fbaba962b99d5543477.svg
Charlie Snell
user
sea-snell
674550443dd665eec817d2f4
Eric Wallace
admin_assigned
2024-11-26T22:58:54.291Z
63112d2431257261d20d5754
/avatars/502a68c0fb2f0c6989fe2869d0a7e3f4.svg
Eric Wallace
user
EricWallace
674550443dd665eec817d2f5
Dan Klein
admin_assigned
2024-11-26T22:59:01.446Z
66e1fbf2a8e06d2e99c7aac1
/avatars/13939a47f995a66a4ab32004c8cc21d0.svg
Daniel Klein
user
danklein
674550443dd665eec817d2f6
Sergey Levine
admin_assigned
2024-11-26T22:59:06.568Z
665ce54120a307a3754849dd
/avatars/e698726e9be61dd50ce2efe372ed5dac.svg
Sergey Levine
user
svlevine
Predicting Emergent Capabilities by Finetuning
A fundamental open challenge in modern LLM scaling is the lack of
understanding around emergent capabilities. In particular, language model
pretraining loss is known to be highly predictable as a function of compute.
However, downstream capabilities are far less predictable -- sometimes even
exhibiting emergent jumps -- which makes it challenging to anticipate the
capabilities of future models. In this work, we first pose the task of
emergence prediction: given access to current LLMs that have random few-shot
accuracy on a task, can we predict whether future models (GPT-N+1) will have
non-trivial accuracy on that task? We then discover a simple insight for this
problem: finetuning LLMs on a given task can shift the point in scaling at
which emergence occurs towards less capable models. To operationalize this
insight, we can finetune LLMs with varying amounts of data and fit a parametric
function that predicts when emergence will occur (i.e., "emergence laws"). We
validate this approach using four standard NLP benchmarks where large-scale
open-source LLMs already demonstrate emergence (MMLU, GSM8K, CommonsenseQA, and
CoLA). Using only small-scale LLMs, we find that, in some cases, we can
accurately predict whether models trained with up to 4x more compute have
emerged. Finally, we present a case study of two realistic uses for emergence
prediction.
Researchers predict the emergence of capabilities in large language models through fine-tuning and parameteric functions to forecast when models will achieve non-trivial accuracy on a task.
emergence prediction emergence laws fine-tuning LLMs few-shot accuracy MMLU GSM8K CommonsenseQA CoLA