title: Sleep-time Compute: Beyond Inference Scaling at Test-time
summary: Scaling test-time compute has emerged as a key ingredient for enabling large
  language models (LLMs) to solve difficult problems, but comes with high latency
  and inference cost. We introduce sleep-time compute, which allows models to
  "think" offline about contexts before queries are presented: by anticipating
  what queries users might ask and pre-computing useful quantities, we can
  significantly reduce the compute requirements at test-time. To demonstrate the
  efficacy of our method, we create modified versions of two reasoning tasks -
  Stateful GSM-Symbolic and Stateful AIME. We find that sleep-time compute can
  reduce the amount of test-time compute needed to achieve the same accuracy by ~
  5x on Stateful GSM-Symbolic and Stateful AIME and that by scaling sleep-time
  compute we can further increase accuracy by up to 13% on Stateful GSM-Symbolic
  and 18% on Stateful AIME. Furthermore, we introduce Multi-Query GSM-Symbolic,
  which extends GSM-Symbolic by including multiple related queries per context.
  By amortizing sleep-time compute across related queries about the same context
  using Multi-Query GSM-Symbolic, we can decrease the average cost per query by
  2.5x. We then conduct additional analysis to understand when sleep-time compute
  is most effective, finding the predictability of the user query to be well
  correlated with the efficacy of sleep-time compute. Finally, we conduct a
  case-study of applying sleep-time compute to a realistic agentic SWE task.
paper_authors._id: 680210b4b2ae01ba08b04189
paper_authors.name: Kevin Lin
paper_authors._id: 680210b4b2ae01ba08b0418a
paper_authors.name: Charlie Snell
paper_authors._id: 680210b4b2ae01ba08b0418b
paper_authors.name: Yu Wang
paper_authors._id: 680210b4b2ae01ba08b0418c
paper_authors.name: Charles Packer
paper_authors._id: 680210b4b2ae01ba08b0418d
paper_authors.name: Sarah Wooders
paper_authors._id: 680210b4b2ae01ba08b0418e
paper_authors.name: Ion Stoica
paper_authors._id: 680210b4b2ae01ba08b0418f
paper_authors.name: Joseph E. Gonzalez
paper_title: Sleep-time Compute: Beyond Inference Scaling at Test-time
paper_summary: Scaling test-time compute has emerged as a key ingredient for enabling large
  language models (LLMs) to solve difficult problems, but comes with high latency
  and inference cost. We introduce sleep-time compute, which allows models to
  "think" offline about contexts before queries are presented: by anticipating
  what queries users might ask and pre-computing useful quantities, we can
  significantly reduce the compute requirements at test-time. To demonstrate the
  efficacy of our method, we create modified versions of two reasoning tasks -
  Stateful GSM-Symbolic and Stateful AIME. We find that sleep-time compute can
  reduce the amount of test-time compute needed to achieve the same accuracy by ~
  5x on Stateful GSM-Symbolic and Stateful AIME and that by scaling sleep-time
  compute we can further increase accuracy by up to 13% on Stateful GSM-Symbolic
  and 18% on Stateful AIME. Furthermore, we introduce Multi-Query GSM-Symbolic,
  which extends GSM-Symbolic by including multiple related queries per context.
  By amortizing sleep-time compute across related queries about the same context
  using Multi-Query GSM-Symbolic, we can decrease the average cost per query by
  2.5x. We then conduct additional analysis to understand when sleep-time compute
  is most effective, finding the predictability of the user query to be well
  correlated with the efficacy of sleep-time compute. Finally, we conduct a
  case-study of applying sleep-time compute to a realistic agentic SWE task.
paper_ai_summary: Introducing sleep-time compute to performing off-line pre-computations reduces test-time compute requirements and enhances accuracy in reasoning tasks for large language models.
paper_ai_keywords: large language models (LLMs) sleep-time compute GSM-Symbolic AIME Stateful GSM-Symbolic Stateful AIME Multi-Query GSM-Symbolic test-time compute query prediction cost per query agentic SWE task