title: Cost-of-Pass: An Economic Framework for Evaluating Language Models
summary: The widespread adoption of AI systems in the economy hinges on their ability
  to generate economic value that outweighs their inference costs. Evaluating
  this tradeoff requires metrics that account for both performance and costs. We
  propose a framework grounded in production theory for evaluating language
  models by combining accuracy and inference cost. We introduce "cost-of-pass",
  the expected monetary cost of generating a correct solution. We then define the
  "frontier cost-of-pass" as the minimum cost-of-pass achievable across available
  models or the "human-expert, using the approximate cost of hiring an expert.
  Our analysis reveals distinct economic insights. First, lightweight models are
  most cost-effective for basic quantitative tasks, large models for
  knowledge-intensive ones, and reasoning models for complex quantitative
  problems, despite higher per-token costs. Second, tracking this frontier
  cost-of-pass over the past year reveals significant progress, particularly for
  complex quantitative tasks where the cost has roughly halved every few months.
  Third, to trace key innovations driving this progress, we examine
  counterfactual frontiers: estimates of cost-efficiency without specific model
  classes. We find that innovations in lightweight, large, and reasoning models
  have been essential for pushing the frontier in basic quantitative,
  knowledge-intensive, and complex quantitative tasks, respectively. Finally, we
  assess the cost-reductions afforded by common inference-time techniques like
  majority voting and self-refinement, finding that their marginal accuracy gains
  rarely justify their costs. Our findings underscore that complementary
  model-level innovations are the primary drivers of cost-efficiency, and our
  economic framework provides a principled tool for measuring this progress and
  guiding deployment.
paper_authors._id: 680690f08a4cbdf2a11ba1ce
paper_authors.name: Mehmet Hamza Erol
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-22T09:51:57.293Z
paper_authors._id: 6404a5c40ab5e22719f3e105
paper_authors.avatarUrl: /avatars/f683bd957502162bad10039c28c8c244.svg
paper_authors.fullname: Mehmet Hamza Erol
paper_authors.type: user
paper_authors.user: mhamzaerol
paper_authors._id: 680690f08a4cbdf2a11ba1cf
paper_authors.name: Batu El
paper_authors._id: 680690f08a4cbdf2a11ba1d0
paper_authors.name: Mirac Suzgun
paper_authors._id: 680690f08a4cbdf2a11ba1d1
paper_authors.name: Mert Yuksekgonul
paper_authors._id: 680690f08a4cbdf2a11ba1d2
paper_authors.name: James Zou
paper_title: Cost-of-Pass: An Economic Framework for Evaluating Language Models
paper_summary: The widespread adoption of AI systems in the economy hinges on their ability
  to generate economic value that outweighs their inference costs. Evaluating
  this tradeoff requires metrics that account for both performance and costs. We
  propose a framework grounded in production theory for evaluating language
  models by combining accuracy and inference cost. We introduce "cost-of-pass",
  the expected monetary cost of generating a correct solution. We then define the
  "frontier cost-of-pass" as the minimum cost-of-pass achievable across available
  models or the "human-expert, using the approximate cost of hiring an expert.
  Our analysis reveals distinct economic insights. First, lightweight models are
  most cost-effective for basic quantitative tasks, large models for
  knowledge-intensive ones, and reasoning models for complex quantitative
  problems, despite higher per-token costs. Second, tracking this frontier
  cost-of-pass over the past year reveals significant progress, particularly for
  complex quantitative tasks where the cost has roughly halved every few months.
  Third, to trace key innovations driving this progress, we examine
  counterfactual frontiers: estimates of cost-efficiency without specific model
  classes. We find that innovations in lightweight, large, and reasoning models
  have been essential for pushing the frontier in basic quantitative,
  knowledge-intensive, and complex quantitative tasks, respectively. Finally, we
  assess the cost-reductions afforded by common inference-time techniques like
  majority voting and self-refinement, finding that their marginal accuracy gains
  rarely justify their costs. Our findings underscore that complementary
  model-level innovations are the primary drivers of cost-efficiency, and our
  economic framework provides a principled tool for measuring this progress and
  guiding deployment.
paper_ai_summary: The framework combines accuracy and inference cost to evaluate language models and reveals that innovation in specialized models, rather than inference techniques, drives cost-efficiency in AI deployment.
paper_ai_keywords: language models cost-of-pass frontier cost-of-pass lightweight models large models reasoning models cost-efficiency majority voting self-refinement