title: Observational Scaling Laws and the Predictability of Language Model
    Performance
summary: Understanding how language model performance varies with scale is critical to
  benchmark and algorithm development. Scaling laws are one approach to building
  this understanding, but the requirement of training models across many
  different scales has limited their use. We propose an alternative,
  observational approach that bypasses model training and instead builds scaling
  laws from ~80 publically available models. Building a single scaling law from
  multiple model families is challenging due to large variations in their
  training compute efficiencies and capabilities. However, we show that these
  variations are consistent with a simple, generalized scaling law where language
  model performance is a function of a low-dimensional capability space, and
  model families only vary in their efficiency in converting training compute to
  capabilities. Using this approach, we show the surprising predictability of
  complex scaling phenomena: we show that several emergent phenomena follow a
  smooth, sigmoidal behavior and are predictable from small models; we show that
  the agent performance of models such as GPT-4 can be precisely predicted from
  simpler non-agentic benchmarks; and we show how to predict the impact of
  post-training interventions like Chain-of-Thought and Self-Consistency as
  language model capabilities continue to improve.
paper_authors._id: 664b64cc5dfaf30f2d2c05cf
paper_authors.name: Yangjun Ruan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-20T15:23:10.344Z
paper_authors._id: 65619949d2e4352d64365606
paper_authors.avatarUrl: /avatars/4241b6e901e146a9ff8b5bc7de3f960a.svg
paper_authors.fullname: Yangjun Ruan
paper_authors.type: user
paper_authors.user: ryoungj
paper_authors._id: 664b64cc5dfaf30f2d2c05d0
paper_authors.name: Chris J. Maddison
paper_authors._id: 664b64cc5dfaf30f2d2c05d1
paper_authors.name: Tatsunori Hashimoto
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-05-20T14:57:17.446Z
paper_authors._id: 661595d1b3d0b21da55cde7d
paper_authors.avatarUrl: /avatars/ba3fa065536518637d21a5c46cee5dd1.svg
paper_authors.fullname: Tatsu Hashimoto
paper_authors.type: user
paper_authors.user: thashim
paper_title: Observational Scaling Laws and the Predictability of Language Model
    Performance
paper_summary: Understanding how language model performance varies with scale is critical to
  benchmark and algorithm development. Scaling laws are one approach to building
  this understanding, but the requirement of training models across many
  different scales has limited their use. We propose an alternative,
  observational approach that bypasses model training and instead builds scaling
  laws from ~80 publically available models. Building a single scaling law from
  multiple model families is challenging due to large variations in their
  training compute efficiencies and capabilities. However, we show that these
  variations are consistent with a simple, generalized scaling law where language
  model performance is a function of a low-dimensional capability space, and
  model families only vary in their efficiency in converting training compute to
  capabilities. Using this approach, we show the surprising predictability of
  complex scaling phenomena: we show that several emergent phenomena follow a
  smooth, sigmoidal behavior and are predictable from small models; we show that
  the agent performance of models such as GPT-4 can be precisely predicted from
  simpler non-agentic benchmarks; and we show how to predict the impact of
  post-training interventions like Chain-of-Thought and Self-Consistency as
  language model capabilities continue to improve.
paper_ai_summary: An observational approach to scaling laws in language models can predict complex phenomena and agent performance using capability space and efficiency in converting training compute to capabilities.
paper_ai_keywords: scaling laws language model performance capability space training compute emergent phenomena sigmoidal behavior predictive modeling Chain-of-Thought Self-Consistency