title: Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM
    Reasoning via Autoregressive Search
summary: Large language models (LLMs) have demonstrated remarkable reasoning
  capabilities across diverse domains. Recent studies have shown that increasing
  test-time computation enhances LLMs' reasoning capabilities. This typically
  involves extensive sampling at inference time guided by an external LLM
  verifier, resulting in a two-player system. Despite external guidance, the
  effectiveness of this system demonstrates the potential of a single LLM to
  tackle complex tasks. Thus, we pose a new research problem: Can we internalize
  the searching capabilities to fundamentally enhance the reasoning abilities of
  a single LLM? This work explores an orthogonal direction focusing on
  post-training LLMs for autoregressive searching (i.e., an extended reasoning
  process with self-reflection and self-exploration of new strategies). To
  achieve this, we propose the Chain-of-Action-Thought (COAT) reasoning and a
  two-stage training paradigm: 1) a small-scale format tuning stage to
  internalize the COAT reasoning format and 2) a large-scale self-improvement
  stage leveraging reinforcement learning. Our approach results in Satori, a 7B
  LLM trained on open-source models and data. Extensive empirical evaluations
  demonstrate that Satori achieves state-of-the-art performance on mathematical
  reasoning benchmarks while exhibits strong generalization to out-of-domain
  tasks. Code, data, and models will be fully open-sourced.
paper_authors._id: 67a2d1f9bc9d072d9459e857
paper_authors.name: Maohao Shen
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-02-05T03:00:33.470Z
paper_authors._id: 6553c985a7aded0380b5f928
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6553c985a7aded0380b5f928/tAJnmYvmaAse4fbHq0CjA.jpeg
paper_authors.fullname: Maohao Shen
paper_authors.type: user
paper_authors.user: maohaos2
paper_authors._id: 67a2d1f9bc9d072d9459e858
paper_authors.name: Guangtao Zeng
paper_authors._id: 67a2d1f9bc9d072d9459e859
paper_authors.name: Zhenting Qi
paper_authors._id: 67a2d1f9bc9d072d9459e85a
paper_authors.name: Zhang-Wei Hong
paper_authors._id: 67a2d1f9bc9d072d9459e85b
paper_authors.name: Zhenfang Chen
paper_authors._id: 67a2d1f9bc9d072d9459e85c
paper_authors.name: Wei Lu
paper_authors._id: 67a2d1f9bc9d072d9459e85d
paper_authors.name: Gregory Wornell
paper_authors._id: 67a2d1f9bc9d072d9459e85e
paper_authors.name: Subhro Das
paper_authors._id: 67a2d1f9bc9d072d9459e85f
paper_authors.name: David Cox
paper_authors._id: 67a2d1f9bc9d072d9459e860
paper_authors.name: Chuang Gan
paper_title: Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM
    Reasoning via Autoregressive Search
paper_summary: Large language models (LLMs) have demonstrated remarkable reasoning
  capabilities across diverse domains. Recent studies have shown that increasing
  test-time computation enhances LLMs' reasoning capabilities. This typically
  involves extensive sampling at inference time guided by an external LLM
  verifier, resulting in a two-player system. Despite external guidance, the
  effectiveness of this system demonstrates the potential of a single LLM to
  tackle complex tasks. Thus, we pose a new research problem: Can we internalize
  the searching capabilities to fundamentally enhance the reasoning abilities of
  a single LLM? This work explores an orthogonal direction focusing on
  post-training LLMs for autoregressive searching (i.e., an extended reasoning
  process with self-reflection and self-exploration of new strategies). To
  achieve this, we propose the Chain-of-Action-Thought (COAT) reasoning and a
  two-stage training paradigm: 1) a small-scale format tuning stage to
  internalize the COAT reasoning format and 2) a large-scale self-improvement
  stage leveraging reinforcement learning. Our approach results in Satori, a 7B
  LLM trained on open-source models and data. Extensive empirical evaluations
  demonstrate that Satori achieves state-of-the-art performance on mathematical
  reasoning benchmarks while exhibits strong generalization to out-of-domain
  tasks. Code, data, and models will be fully open-sourced.
paper_ai_summary: Satori, a 7B LLM enhanced with COAT reasoning and two-stage training, achieves top performance in math benchmarks and generalizes well to unrelated tasks.
paper_ai_keywords: LLMs reasoning capabilities test-time computation LLM verifier autoregressive searching COAT reasoning format tuning self-improvement stage reinforcement learning mathematical reasoning benchmarks out-of-domain tasks