title: AceReason-Nemotron: Advancing Math and Code Reasoning through
    Reinforcement Learning
summary: Despite recent progress in large-scale reinforcement learning (RL) for
  reasoning, the training recipe for building high-performing reasoning models
  remains elusive. Key implementation details of frontier models, such as
  DeepSeek-R1, including data curation strategies and RL training recipe, are
  often omitted. Moreover, recent research indicates distillation remains more
  effective than RL for smaller models. In this work, we demonstrate that
  large-scale RL can significantly enhance the reasoning capabilities of strong,
  small- and mid-sized models, achieving results that surpass those of
  state-of-the-art distillation-based models. We systematically study the RL
  training process through extensive ablations and propose a simple yet effective
  approach: first training on math-only prompts, then on code-only prompts.
  Notably, we find that math-only RL not only significantly enhances the
  performance of strong distilled models on math benchmarks (e.g., +14.6% /
  +17.2% on AIME 2025 for the 7B / 14B models), but also code reasoning tasks
  (e.g., +6.8% / +5.8% on LiveCodeBench for the 7B / 14B models). In addition,
  extended code-only RL iterations further improve performance on code benchmarks
  with minimal or no degradation in math results. We develop a robust data
  curation pipeline to collect challenging prompts with high-quality, verifiable
  answers and test cases to enable verification-based RL across both domains.
  Finally, we identify key experimental insights, including curriculum learning
  with progressively increasing response lengths and the stabilizing effect of
  on-policy parameter updates. We find that RL not only elicits the foundational
  reasoning capabilities acquired during pretraining and supervised fine-tuning
  (e.g., distillation), but also pushes the limits of the model's reasoning
  ability, enabling it to solve problems that were previously unsolvable.
paper_authors._id: 682fe6644640a9db4d1f31d9
paper_authors.name: Yang Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:16:02.553Z
paper_authors._id: 62bc9d90e81dfd65cced9316
paper_authors.avatarUrl: /avatars/05df14cd1fdbc7d6a80d2960a05a94f0.svg
paper_authors.fullname: Yang Chen
paper_authors.type: user
paper_authors.user: ychenNLP
paper_authors._id: 682fe6644640a9db4d1f31da
paper_authors.name: Zhuolin Yang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-05-23T03:07:17.257Z
paper_authors._id: 67d75b0117c2acac528f47b6
paper_authors.avatarUrl: /avatars/619aacd1a619aab64de3499ac3ee2229.svg
paper_authors.fullname: Zhuolin Yang
paper_authors.type: user
paper_authors.user: zhuoliny
paper_authors._id: 682fe6644640a9db4d1f31db
paper_authors.name: Zihan Liu
paper_authors._id: 682fe6644640a9db4d1f31dc
paper_authors.name: Chankyu Lee
paper_authors._id: 682fe6644640a9db4d1f31dd
paper_authors.name: Peng Xu
paper_authors._id: 682fe6644640a9db4d1f31de
paper_authors.name: Mohammad Shoeybi
paper_authors._id: 682fe6644640a9db4d1f31df
paper_authors.name: Bryan Catanzaro
paper_authors._id: 682fe6644640a9db4d1f31e0
paper_authors.name: Wei Ping
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-05-23T03:07:17.257Z
paper_authors._id: 663ee43bfeeb49803537da98
paper_authors.avatarUrl: /avatars/17c3e9c435cc36fb04b4589e6176a243.svg
paper_authors.fullname: Wei Ping
paper_authors.type: user
paper_authors.user: wping
paper_title: AceReason-Nemotron: Advancing Math and Code Reasoning through
    Reinforcement Learning
paper_summary: Despite recent progress in large-scale reinforcement learning (RL) for
  reasoning, the training recipe for building high-performing reasoning models
  remains elusive. Key implementation details of frontier models, such as
  DeepSeek-R1, including data curation strategies and RL training recipe, are
  often omitted. Moreover, recent research indicates distillation remains more
  effective than RL for smaller models. In this work, we demonstrate that
  large-scale RL can significantly enhance the reasoning capabilities of strong,
  small- and mid-sized models, achieving results that surpass those of
  state-of-the-art distillation-based models. We systematically study the RL
  training process through extensive ablations and propose a simple yet effective
  approach: first training on math-only prompts, then on code-only prompts.
  Notably, we find that math-only RL not only significantly enhances the
  performance of strong distilled models on math benchmarks (e.g., +14.6% /
  +17.2% on AIME 2025 for the 7B / 14B models), but also code reasoning tasks
  (e.g., +6.8% / +5.8% on LiveCodeBench for the 7B / 14B models). In addition,
  extended code-only RL iterations further improve performance on code benchmarks
  with minimal or no degradation in math results. We develop a robust data
  curation pipeline to collect challenging prompts with high-quality, verifiable
  answers and test cases to enable verification-based RL across both domains.
  Finally, we identify key experimental insights, including curriculum learning
  with progressively increasing response lengths and the stabilizing effect of
  on-policy parameter updates. We find that RL not only elicits the foundational
  reasoning capabilities acquired during pretraining and supervised fine-tuning
  (e.g., distillation), but also pushes the limits of the model's reasoning
  ability, enabling it to solve problems that were previously unsolvable.
paper_ai_summary: Large-scale reinforcement learning enhances reasoning capabilities in small and mid-sized models more effectively than distillation, achieving superior results in both math and code benchmarks.
paper_ai_keywords: reinforcement learning RL DeepSeek-R1 data curation distillation math-only prompts code-only prompts AIME 2025 LiveCodeBench curriculum learning on-policy parameter updates