title: Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization
    in Rejection Sampling and RL
summary: Chain-of-thought (CoT) reasoning in large language models (LLMs) can be
  formalized as a latent variable problem, where the model needs to generate
  intermediate reasoning steps. While prior approaches such as iterative
  reward-ranked fine-tuning (RAFT) have relied on such formulations, they
  typically apply uniform inference budgets across prompts, which fails to
  account for variability in difficulty and convergence behavior. This work
  identifies the main bottleneck in CoT training as inefficient stochastic
  gradient estimation due to static sampling strategies. We propose GVM-RAFT, a
  prompt-specific Dynamic Sample Allocation Strategy designed to minimize
  stochastic gradient variance under a computational budget constraint. The
  method dynamically allocates computational resources by monitoring prompt
  acceptance rates and stochastic gradient norms, ensuring that the resulting
  gradient variance is minimized. Our theoretical analysis shows that the
  proposed dynamic sampling strategy leads to accelerated convergence guarantees
  under suitable conditions. Experiments on mathematical reasoning show that
  GVM-RAFT achieves a 2-4x speedup and considerable accuracy improvements over
  vanilla RAFT. The proposed dynamic sampling strategy is general and can be
  incorporated into other reinforcement learning algorithms, such as GRPO,
  leading to similar improvements in convergence and test accuracy. Our code is
  available at https://github.com/RLHFlow/GVM.
paper_authors._id: 6819a63c64ae18f1b60a5c43
paper_authors.name: Jiarui Yao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-06T08:32:24.344Z
paper_authors._id: 66f8689725464a7989b75845
paper_authors.avatarUrl: /avatars/43a61a528c5779103eaf5687ba44ee14.svg
paper_authors.fullname: Jiarui Yao
paper_authors.type: user
paper_authors.user: FlippyDora
paper_authors._id: 6819a63c64ae18f1b60a5c44
paper_authors.name: Yifan Hao
paper_authors._id: 6819a63c64ae18f1b60a5c45
paper_authors.name: Hanning Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T08:59:20.459Z
paper_authors._id: 6470e0f1cfd57849519033a5
paper_authors.avatarUrl: /avatars/7ffefee3e36a4e37b9f4510bc6b689d1.svg
paper_authors.fullname: Hanning Zhang
paper_authors.type: user
paper_authors.user: HanningZhang
paper_authors._id: 6819a63c64ae18f1b60a5c46
paper_authors.name: Hanze Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T08:59:27.200Z
paper_authors._id: 63a3ff69f91ad3ea5703841d
paper_authors.avatarUrl: /avatars/69227c4bce01d33747c1377b6f9672db.svg
paper_authors.fullname: Hanze Dong
paper_authors.type: user
paper_authors.user: hendrydong
paper_authors._id: 6819a63c64ae18f1b60a5c47
paper_authors.name: Wei Xiong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T08:59:34.381Z
paper_authors._id: 6319b29809baf858241f05de
paper_authors.avatarUrl: /avatars/29eef2c52814abea82e2aa9bf37a7f9c.svg
paper_authors.fullname: Xiong
paper_authors.type: user
paper_authors.user: WeiXiong
paper_authors._id: 6819a63c64ae18f1b60a5c48
paper_authors.name: Nan Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T08:59:47.400Z
paper_authors._id: 64b8922ca1827cc8d04ae919
paper_authors.avatarUrl: /avatars/0aaa83e3d09a82434e1d6af724aaa485.svg
paper_authors.fullname: Nan Jiang
paper_authors.type: user
paper_authors.user: nanjiang
paper_authors._id: 6819a63c64ae18f1b60a5c49
paper_authors.name: Tong Zhang
paper_title: Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization
    in Rejection Sampling and RL
paper_summary: Chain-of-thought (CoT) reasoning in large language models (LLMs) can be
  formalized as a latent variable problem, where the model needs to generate
  intermediate reasoning steps. While prior approaches such as iterative
  reward-ranked fine-tuning (RAFT) have relied on such formulations, they
  typically apply uniform inference budgets across prompts, which fails to
  account for variability in difficulty and convergence behavior. This work
  identifies the main bottleneck in CoT training as inefficient stochastic
  gradient estimation due to static sampling strategies. We propose GVM-RAFT, a
  prompt-specific Dynamic Sample Allocation Strategy designed to minimize
  stochastic gradient variance under a computational budget constraint. The
  method dynamically allocates computational resources by monitoring prompt
  acceptance rates and stochastic gradient norms, ensuring that the resulting
  gradient variance is minimized. Our theoretical analysis shows that the
  proposed dynamic sampling strategy leads to accelerated convergence guarantees
  under suitable conditions. Experiments on mathematical reasoning show that
  GVM-RAFT achieves a 2-4x speedup and considerable accuracy improvements over
  vanilla RAFT. The proposed dynamic sampling strategy is general and can be
  incorporated into other reinforcement learning algorithms, such as GRPO,
  leading to similar improvements in convergence and test accuracy. Our code is
  available at https://github.com/RLHFlow/GVM.
paper_ai_summary: GVM-RAFT, a dynamic sampling strategy for chain-of-thought reasoning in large language models, improves convergence and accuracy by adaptively allocating computational resources.
paper_ai_keywords: chain-of-thought large language models latent variable problem iterative reward-ranked fine-tuning stochastic gradient estimation dynamic sample allocation prompt-specific computational budget constraint prompt acceptance rates stochastic gradient norms GRPO reinforcement learning