title: A Probabilistic Inference Approach to Inference-Time Scaling of LLMs
    using Particle-Based Monte Carlo Methods
summary: Large language models (LLMs) have achieved significant performance gains via
  scaling up model sizes and/or data. However, recent evidence suggests
  diminishing returns from such approaches, motivating scaling the computation
  spent at inference time. Existing inference-time scaling methods, usually with
  reward models, cast the task as a search problem, which tends to be vulnerable
  to reward hacking as a consequence of approximation errors in reward models. In
  this paper, we instead cast inference-time scaling as a probabilistic inference
  task and leverage sampling-based techniques to explore the typical set of the
  state distribution of a state-space model with an approximate likelihood,
  rather than optimize for its mode directly. We propose a novel inference-time
  scaling approach by adapting particle-based Monte Carlo methods to this task.
  Our empirical evaluation demonstrates that our methods have a 4-16x better
  scaling rate over our deterministic search counterparts on various challenging
  mathematical reasoning tasks. Using our approach, we show that
  Qwen2.5-Math-1.5B-Instruct can surpass GPT-4o accuracy in only 4 rollouts,
  while Qwen2.5-Math-7B-Instruct scales to o1 level accuracy in only 32 rollouts.
  Our work not only presents an effective method to inference-time scaling, but
  also connects the rich literature in probabilistic inference with
  inference-time scaling of LLMs to develop more robust algorithms in future
  work. Code and further information is available at
  https://probabilistic-inference-scaling.github.io.
paper_authors._id: 67a438d26bb8caaab06f5a5e
paper_authors.name: Isha Puri
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-02-06T04:21:39.202Z
paper_authors._id: 64c2abe8c43875b438efef25
paper_authors.avatarUrl: /avatars/6efda081f52cf56db2d29a5ec05cb557.svg
paper_authors.fullname: isha
paper_authors.type: user
paper_authors.user: ishapuri-mit
paper_authors._id: 67a438d26bb8caaab06f5a5f
paper_authors.name: Shivchander Sudalairaj
paper_authors._id: 67a438d26bb8caaab06f5a60
paper_authors.name: Guangxuan Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-06T16:07:05.566Z
paper_authors._id: 66104696134c832243bde60d
paper_authors.avatarUrl: /avatars/b5a0d194d0e12c60fc5599f81f75c205.svg
paper_authors.fullname: Guangxuan Xu
paper_authors.type: user
paper_authors.user: gx-ai-architect
paper_authors._id: 67a438d26bb8caaab06f5a61
paper_authors.name: Kai Xu
paper_authors._id: 67a438d26bb8caaab06f5a62
paper_authors.name: Akash Srivastava
paper_title: A Probabilistic Inference Approach to Inference-Time Scaling of LLMs
    using Particle-Based Monte Carlo Methods
paper_summary: Large language models (LLMs) have achieved significant performance gains via
  scaling up model sizes and/or data. However, recent evidence suggests
  diminishing returns from such approaches, motivating scaling the computation
  spent at inference time. Existing inference-time scaling methods, usually with
  reward models, cast the task as a search problem, which tends to be vulnerable
  to reward hacking as a consequence of approximation errors in reward models. In
  this paper, we instead cast inference-time scaling as a probabilistic inference
  task and leverage sampling-based techniques to explore the typical set of the
  state distribution of a state-space model with an approximate likelihood,
  rather than optimize for its mode directly. We propose a novel inference-time
  scaling approach by adapting particle-based Monte Carlo methods to this task.
  Our empirical evaluation demonstrates that our methods have a 4-16x better
  scaling rate over our deterministic search counterparts on various challenging
  mathematical reasoning tasks. Using our approach, we show that
  Qwen2.5-Math-1.5B-Instruct can surpass GPT-4o accuracy in only 4 rollouts,
  while Qwen2.5-Math-7B-Instruct scales to o1 level accuracy in only 32 rollouts.
  Our work not only presents an effective method to inference-time scaling, but
  also connects the rich literature in probabilistic inference with
  inference-time scaling of LLMs to develop more robust algorithms in future
  work. Code and further information is available at
  https://probabilistic-inference-scaling.github.io.
paper_ai_keywords: inference-time scaling reward models probabilistic inference sampling-based techniques typical set state distribution state-space model approximate likelihood particle-based Monte Carlo methods mathematical reasoning tasks rollouts