title: Beyond Chinchilla-Optimal: Accounting for Inference in Language Model
    Scaling Laws
summary: Large language model (LLM) scaling laws are empirical formulas that estimate
  changes in model quality as a result of increasing parameter count and training
  data. However, these formulas, including the popular DeepMind Chinchilla
  scaling laws, neglect to include the cost of inference. We modify the
  Chinchilla scaling laws to calculate the optimal LLM parameter count and
  pre-training data size to train and deploy a model of a given quality and
  inference demand. We conduct our analysis both in terms of a compute budget and
  real-world costs and find that LLM researchers expecting reasonably large
  inference demand (~1B requests) should train models smaller and longer than
  Chinchilla-optimal.
paper_authors._id: 65938e54dbdeb5bf078f5644
paper_authors.name: Nikhil Sardana
paper_authors.status: extracted_infirmed
paper_authors.statusLastChangedAt: 2024-01-02T22:38:06.642Z
paper_authors._id: 65938e54dbdeb5bf078f5645
paper_authors.name: Jonathan Frankle
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-01-02T04:17:25.214Z
paper_authors._id: 62b0b7f9a596c50b1783fe9e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b0b7f9a596c50b1783fe9e/nzctJXoyiEwCnOHVJEx3_.png
paper_authors.fullname: Jonathan Frankle
paper_authors.type: user
paper_authors.user: jfrankle
paper_title: Beyond Chinchilla-Optimal: Accounting for Inference in Language Model
    Scaling Laws
paper_summary: Large language model (LLM) scaling laws are empirical formulas that estimate
  changes in model quality as a result of increasing parameter count and training
  data. However, these formulas, including the popular DeepMind Chinchilla
  scaling laws, neglect to include the cost of inference. We modify the
  Chinchilla scaling laws to calculate the optimal LLM parameter count and
  pre-training data size to train and deploy a model of a given quality and
  inference demand. We conduct our analysis both in terms of a compute budget and
  real-world costs and find that LLM researchers expecting reasonably large
  inference demand (~1B requests) should train models smaller and longer than
  Chinchilla-optimal.
paper_ai_summary: The modified Chinchilla scaling laws incorporate inference cost, suggesting that models for large inference demand should be smaller and trained longer than originally proposed.
paper_ai_keywords: Large language model scaling laws DeepMind Chinchilla parameter count pre-training data inference demand compute budget real-world costs Chinchilla-optimal