AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT
  and RL Synergy
In this work, we investigate the synergy between supervised fine-tuning (SFT)
and reinforcement learning (RL) in developing strong reasoning models. We begin
by curating the SFT training data through two scaling strategies: increasing
the number of collected prompts and the number of generated responses per
prompt. Both approaches yield notable improvements in reasoning performance,
with scaling the number of prompts resulting in more substantial gains. We then
explore the following questions regarding the synergy between SFT and RL: (i)
Does a stronger SFT model consistently lead to better final performance after
large-scale RL training? (ii) How can we determine an appropriate sampling
temperature during RL training to effectively balance exploration and
exploitation for a given SFT initialization? Our findings suggest that (i)
holds true, provided effective RL training is conducted, particularly when the
sampling temperature is carefully chosen to maintain the temperature-adjusted
entropy around 0.3, a setting that strikes a good balance between exploration
and exploitation. Notably, the performance gap between initial SFT models
narrows significantly throughout the RL process. Leveraging a strong SFT
foundation and insights into the synergistic interplay between SFT and RL, our
AceReason-Nemotron-1.1 7B model significantly outperforms
AceReason-Nemotron-1.0 and achieves new state-of-the-art performance among
Qwen2.5-7B-based reasoning models on challenging math and code benchmarks,
thereby demonstrating the effectiveness of our post-training recipe. We release
the model and data at: https://huggingface.co/nvidia/AceReason-Nemotron-1.1-7B
6850d4585e07650ecce890a8
Zihan Liu
extracted_pending
2025-06-17T02:35:05.073Z
65f33b1c9f7970ccc0234cbf
/avatars/99fbab303912e3674663251c04279907.svg
Zihan Liu
user
zihanliu
6850d4585e07650ecce890a9
Zhuolin Yang
claimed_verified
2025-06-17T21:13:22.533Z
67d75b0117c2acac528f47b6
/avatars/619aacd1a619aab64de3499ac3ee2229.svg
Zhuolin Yang
user
zhuoliny
6850d4585e07650ecce890aa
Yang Chen
claimed_verified
2025-06-17T21:13:20.767Z
62bc9d90e81dfd65cced9316
/avatars/05df14cd1fdbc7d6a80d2960a05a94f0.svg
Yang Chen
user
ychenNLP
6850d4585e07650ecce890ab
Chankyu Lee
6850d4585e07650ecce890ac
Mohammad Shoeybi
6850d4585e07650ecce890ad
Bryan Catanzaro
6850d4585e07650ecce890ae
Wei Ping
extracted_pending
2025-06-17T02:35:05.073Z
663ee43bfeeb49803537da98
/avatars/17c3e9c435cc36fb04b4589e6176a243.svg
Wei Ping
user
wping
AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT
  and RL Synergy
In this work, we investigate the synergy between supervised fine-tuning (SFT)
and reinforcement learning (RL) in developing strong reasoning models. We begin
by curating the SFT training data through two scaling strategies: increasing
the number of collected prompts and the number of generated responses per
prompt. Both approaches yield notable improvements in reasoning performance,
with scaling the number of prompts resulting in more substantial gains. We then
explore the following questions regarding the synergy between SFT and RL: (i)
Does a stronger SFT model consistently lead to better final performance after
large-scale RL training? (ii) How can we determine an appropriate sampling
temperature during RL training to effectively balance exploration and
exploitation for a given SFT initialization? Our findings suggest that (i)
holds true, provided effective RL training is conducted, particularly when the
sampling temperature is carefully chosen to maintain the temperature-adjusted
entropy around 0.3, a setting that strikes a good balance between exploration
and exploitation. Notably, the performance gap between initial SFT models
narrows significantly throughout the RL process. Leveraging a strong SFT
foundation and insights into the synergistic interplay between SFT and RL, our
AceReason-Nemotron-1.1 7B model significantly outperforms
AceReason-Nemotron-1.0 and achieves new state-of-the-art performance among
Qwen2.5-7B-based reasoning models on challenging math and code benchmarks,
thereby demonstrating the effectiveness of our post-training recipe. We release
the model and data at: https://huggingface.co/nvidia/AceReason-Nemotron-1.1-7B
Combining supervised fine-tuning and reinforcement learning enhances reasoning models, especially when optimizing sampling temperature and leveraging strong initial fine-tuning, as demonstrated by the improved AceReason-Nemotron-1.1 model.
supervised fine-tuning reinforcement learning reasoning models sampling temperature entropy