title: rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep
    Thinking
summary: We present rStar-Math to demonstrate that small language models (SLMs) can
  rival or even surpass the math reasoning capability of OpenAI o1, without
  distillation from superior models. rStar-Math achieves this by exercising "deep
  thinking" through Monte Carlo Tree Search (MCTS), where a math policy SLM
  performs test-time search guided by an SLM-based process reward model.
  rStar-Math introduces three innovations to tackle the challenges in training
  the two SLMs: (1) a novel code-augmented CoT data sythesis method, which
  performs extensive MCTS rollouts to generate step-by-step verified reasoning
  trajectories used to train the policy SLM; (2) a novel process reward model
  training method that avoids na\"ive step-level score annotation, yielding a
  more effective process preference model (PPM); (3) a self-evolution recipe in
  which the policy SLM and PPM are built from scratch and iteratively evolved to
  improve reasoning capabilities. Through 4 rounds of self-evolution with
  millions of synthesized solutions for 747k math problems, rStar-Math boosts
  SLMs' math reasoning to state-of-the-art levels. On the MATH benchmark, it
  improves Qwen2.5-Math-7B from 58.8% to 90.0% and Phi3-mini-3.8B from 41.4% to
  86.4%, surpassing o1-preview by +4.5% and +0.9%. On the USA Math Olympiad
  (AIME), rStar-Math solves an average of 53.3% (8/15) of problems, ranking among
  the top 20% the brightest high school math students. Code and data will be
  available at https://github.com/microsoft/rStar.
paper_authors._id: 677f3f364be6eaf5077001f6
paper_authors.name: Xinyu Guan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-16T19:44:17.716Z
paper_authors._id: 6562b2bead639c6ab9c3aa3e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6562b2bead639c6ab9c3aa3e/LPjzhlE-NDLd9z19PwIrf.png
paper_authors.fullname: Gxy
paper_authors.type: user
paper_authors.user: ElonTusk2001
paper_authors._id: 677f3f364be6eaf5077001f7
paper_authors.name: Li Lyna Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-09T10:14:40.859Z
paper_authors._id: 62b0009c72043b05d29492b2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b0009c72043b05d29492b2/NqRkX2YLhlfOLvYysa7dD.png
paper_authors.fullname: Li Lyna Zhang
paper_authors.type: user
paper_authors.user: lynazhang
paper_authors._id: 677f3f364be6eaf5077001f8
paper_authors.name: Yifei Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-09T19:58:00.452Z
paper_authors._id: 662d015a2d4c0e85da85ff0c
paper_authors.avatarUrl: /avatars/ff38e82d1371fe9e69bacb9b04cfe444.svg
paper_authors.fullname: Yifei Liu
paper_authors.type: user
paper_authors.user: YF-L
paper_authors._id: 677f3f364be6eaf5077001f9
paper_authors.name: Ning Shang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-09T10:16:39.284Z
paper_authors._id: 632bc663eafe8eca5e9bfdbc
paper_authors.avatarUrl: /avatars/787553c73e9a96adc5219e67acd29c00.svg
paper_authors.fullname: Ning Shang
paper_authors.type: user
paper_authors.user: J-shang
paper_authors._id: 677f3f364be6eaf5077001fa
paper_authors.name: Youran Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-12T19:57:37.560Z
paper_authors._id: 672513010cb712d4a556540d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/400Fs_eHiiBNtXRrXurnE.png
paper_authors.fullname: Youran
paper_authors.type: user
paper_authors.user: YouranSun
paper_authors._id: 677f3f364be6eaf5077001fb
paper_authors.name: Yi Zhu
paper_authors._id: 677f3f364be6eaf5077001fc
paper_authors.name: Fan Yang
paper_authors._id: 677f3f364be6eaf5077001fd
paper_authors.name: Mao Yang
paper_title: rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep
    Thinking
paper_summary: We present rStar-Math to demonstrate that small language models (SLMs) can
  rival or even surpass the math reasoning capability of OpenAI o1, without
  distillation from superior models. rStar-Math achieves this by exercising "deep
  thinking" through Monte Carlo Tree Search (MCTS), where a math policy SLM
  performs test-time search guided by an SLM-based process reward model.
  rStar-Math introduces three innovations to tackle the challenges in training
  the two SLMs: (1) a novel code-augmented CoT data sythesis method, which
  performs extensive MCTS rollouts to generate step-by-step verified reasoning
  trajectories used to train the policy SLM; (2) a novel process reward model
  training method that avoids na\"ive step-level score annotation, yielding a
  more effective process preference model (PPM); (3) a self-evolution recipe in
  which the policy SLM and PPM are built from scratch and iteratively evolved to
  improve reasoning capabilities. Through 4 rounds of self-evolution with
  millions of synthesized solutions for 747k math problems, rStar-Math boosts
  SLMs' math reasoning to state-of-the-art levels. On the MATH benchmark, it
  improves Qwen2.5-Math-7B from 58.8% to 90.0% and Phi3-mini-3.8B from 41.4% to
  86.4%, surpassing o1-preview by +4.5% and +0.9%. On the USA Math Olympiad
  (AIME), rStar-Math solves an average of 53.3% (8/15) of problems, ranking among
  the top 20% the brightest high school math students. Code and data will be
  available at https://github.com/microsoft/rStar.
paper_ai_summary: rStar-Math enhances small language models' math reasoning capabilities through Monte Carlo Tree Search and self-evolution, achieving state-of-the-art performance on various benchmarks without distillation from larger models.
paper_ai_keywords: Monte Carlo Tree Search policy SLM process reward model code-augmented CoT data synthesis process preference model PPM self-evolution MATH benchmark USA Math Olympiad AIME