Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM
  Reasoning via Autoregressive Search
Large language models (LLMs) have demonstrated remarkable reasoning
capabilities across diverse domains. Recent studies have shown that increasing
test-time computation enhances LLMs' reasoning capabilities. This typically
involves extensive sampling at inference time guided by an external LLM
verifier, resulting in a two-player system. Despite external guidance, the
effectiveness of this system demonstrates the potential of a single LLM to
tackle complex tasks. Thus, we pose a new research problem: Can we internalize
the searching capabilities to fundamentally enhance the reasoning abilities of
a single LLM? This work explores an orthogonal direction focusing on
post-training LLMs for autoregressive searching (i.e., an extended reasoning
process with self-reflection and self-exploration of new strategies). To
achieve this, we propose the Chain-of-Action-Thought (COAT) reasoning and a
two-stage training paradigm: 1) a small-scale format tuning stage to
internalize the COAT reasoning format and 2) a large-scale self-improvement
stage leveraging reinforcement learning. Our approach results in Satori, a 7B
LLM trained on open-source models and data. Extensive empirical evaluations
demonstrate that Satori achieves state-of-the-art performance on mathematical
reasoning benchmarks while exhibits strong generalization to out-of-domain
tasks. Code, data, and models will be fully open-sourced.
67a2d1f9bc9d072d9459e857
Maohao Shen
extracted_confirmed
2025-02-05T03:00:33.470Z
6553c985a7aded0380b5f928
https://cdn-avatars.huggingface.co/v1/production/uploads/6553c985a7aded0380b5f928/tAJnmYvmaAse4fbHq0CjA.jpeg
Maohao Shen
user
maohaos2
67a2d1f9bc9d072d9459e858
Guangtao Zeng
67a2d1f9bc9d072d9459e859
Zhenting Qi
67a2d1f9bc9d072d9459e85a
Zhang-Wei Hong
67a2d1f9bc9d072d9459e85b
Zhenfang Chen
67a2d1f9bc9d072d9459e85c
Wei Lu
67a2d1f9bc9d072d9459e85d
Gregory Wornell
67a2d1f9bc9d072d9459e85e
Subhro Das
67a2d1f9bc9d072d9459e85f
David Cox
67a2d1f9bc9d072d9459e860
Chuang Gan
Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM
  Reasoning via Autoregressive Search
Large language models (LLMs) have demonstrated remarkable reasoning
capabilities across diverse domains. Recent studies have shown that increasing
test-time computation enhances LLMs' reasoning capabilities. This typically
involves extensive sampling at inference time guided by an external LLM
verifier, resulting in a two-player system. Despite external guidance, the
effectiveness of this system demonstrates the potential of a single LLM to
tackle complex tasks. Thus, we pose a new research problem: Can we internalize
the searching capabilities to fundamentally enhance the reasoning abilities of
a single LLM? This work explores an orthogonal direction focusing on
post-training LLMs for autoregressive searching (i.e., an extended reasoning
process with self-reflection and self-exploration of new strategies). To
achieve this, we propose the Chain-of-Action-Thought (COAT) reasoning and a
two-stage training paradigm: 1) a small-scale format tuning stage to
internalize the COAT reasoning format and 2) a large-scale self-improvement
stage leveraging reinforcement learning. Our approach results in Satori, a 7B
LLM trained on open-source models and data. Extensive empirical evaluations
demonstrate that Satori achieves state-of-the-art performance on mathematical
reasoning benchmarks while exhibits strong generalization to out-of-domain
tasks. Code, data, and models will be fully open-sourced.
Satori, a 7B LLM enhanced with COAT reasoning and two-stage training, achieves top performance in math benchmarks and generalizes well to unrelated tasks.
LLMs reasoning capabilities test-time computation LLM verifier autoregressive searching COAT reasoning format tuning self-improvement stage reinforcement learning mathematical reasoning benchmarks out-of-domain tasks