rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep
  Thinking
We present rStar-Math to demonstrate that small language models (SLMs) can
rival or even surpass the math reasoning capability of OpenAI o1, without
distillation from superior models. rStar-Math achieves this by exercising "deep
thinking" through Monte Carlo Tree Search (MCTS), where a math policy SLM
performs test-time search guided by an SLM-based process reward model.
rStar-Math introduces three innovations to tackle the challenges in training
the two SLMs: (1) a novel code-augmented CoT data sythesis method, which
performs extensive MCTS rollouts to generate step-by-step verified reasoning
trajectories used to train the policy SLM; (2) a novel process reward model
training method that avoids na\"ive step-level score annotation, yielding a
more effective process preference model (PPM); (3) a self-evolution recipe in
which the policy SLM and PPM are built from scratch and iteratively evolved to
improve reasoning capabilities. Through 4 rounds of self-evolution with
millions of synthesized solutions for 747k math problems, rStar-Math boosts
SLMs' math reasoning to state-of-the-art levels. On the MATH benchmark, it
improves Qwen2.5-Math-7B from 58.8% to 90.0% and Phi3-mini-3.8B from 41.4% to
86.4%, surpassing o1-preview by +4.5% and +0.9%. On the USA Math Olympiad
(AIME), rStar-Math solves an average of 53.3% (8/15) of problems, ranking among
the top 20% the brightest high school math students. Code and data will be
available at https://github.com/microsoft/rStar.
677f3f364be6eaf5077001f6
Xinyu Guan
claimed_verified
2025-01-16T19:44:17.716Z
6562b2bead639c6ab9c3aa3e
https://cdn-avatars.huggingface.co/v1/production/uploads/6562b2bead639c6ab9c3aa3e/LPjzhlE-NDLd9z19PwIrf.png
Gxy
user
ElonTusk2001
677f3f364be6eaf5077001f7
Li Lyna Zhang
admin_assigned
2025-01-09T10:14:40.859Z
62b0009c72043b05d29492b2
https://cdn-avatars.huggingface.co/v1/production/uploads/62b0009c72043b05d29492b2/NqRkX2YLhlfOLvYysa7dD.png
Li Lyna Zhang
user
lynazhang
677f3f364be6eaf5077001f8
Yifei Liu
claimed_verified
2025-01-09T19:58:00.452Z
662d015a2d4c0e85da85ff0c
/avatars/ff38e82d1371fe9e69bacb9b04cfe444.svg
Yifei Liu
user
YF-L
677f3f364be6eaf5077001f9
Ning Shang
admin_assigned
2025-01-09T10:16:39.284Z
632bc663eafe8eca5e9bfdbc
/avatars/787553c73e9a96adc5219e67acd29c00.svg
Ning Shang
user
J-shang
677f3f364be6eaf5077001fa
Youran Sun
claimed_verified
2025-01-12T19:57:37.560Z
672513010cb712d4a556540d
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/400Fs_eHiiBNtXRrXurnE.png
Youran
user
YouranSun
677f3f364be6eaf5077001fb
Yi Zhu
677f3f364be6eaf5077001fc
Fan Yang
677f3f364be6eaf5077001fd
Mao Yang
rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep
  Thinking
We present rStar-Math to demonstrate that small language models (SLMs) can
rival or even surpass the math reasoning capability of OpenAI o1, without
distillation from superior models. rStar-Math achieves this by exercising "deep
thinking" through Monte Carlo Tree Search (MCTS), where a math policy SLM
performs test-time search guided by an SLM-based process reward model.
rStar-Math introduces three innovations to tackle the challenges in training
the two SLMs: (1) a novel code-augmented CoT data sythesis method, which
performs extensive MCTS rollouts to generate step-by-step verified reasoning
trajectories used to train the policy SLM; (2) a novel process reward model
training method that avoids na\"ive step-level score annotation, yielding a
more effective process preference model (PPM); (3) a self-evolution recipe in
which the policy SLM and PPM are built from scratch and iteratively evolved to
improve reasoning capabilities. Through 4 rounds of self-evolution with
millions of synthesized solutions for 747k math problems, rStar-Math boosts
SLMs' math reasoning to state-of-the-art levels. On the MATH benchmark, it
improves Qwen2.5-Math-7B from 58.8% to 90.0% and Phi3-mini-3.8B from 41.4% to
86.4%, surpassing o1-preview by +4.5% and +0.9%. On the USA Math Olympiad
(AIME), rStar-Math solves an average of 53.3% (8/15) of problems, ranking among
the top 20% the brightest high school math students. Code and data will be
available at https://github.com/microsoft/rStar.
rStar-Math enhances small language models' math reasoning capabilities through Monte Carlo Tree Search and self-evolution, achieving state-of-the-art performance on various benchmarks without distillation from larger models.
Monte Carlo Tree Search policy SLM process reward model code-augmented CoT data synthesis process preference model PPM self-evolution MATH benchmark USA Math Olympiad AIME