title: Segment Policy Optimization: Effective Segment-Level Credit Assignment
    in RL for Large Language Models
summary: Enhancing the reasoning capabilities of large language models effectively
  using reinforcement learning (RL) remains a crucial challenge. Existing
  approaches primarily adopt two contrasting advantage estimation granularities:
  Token-level methods (e.g., PPO) aim to provide the fine-grained advantage
  signals but suffer from inaccurate estimation due to difficulties in training
  an accurate critic model. On the other extreme, trajectory-level methods (e.g.,
  GRPO) solely rely on a coarse-grained advantage signal from the final reward,
  leading to imprecise credit assignment. To address these limitations, we
  propose Segment Policy Optimization (SPO), a novel RL framework that leverages
  segment-level advantage estimation at an intermediate granularity, achieving a
  better balance by offering more precise credit assignment than trajectory-level
  methods and requiring fewer estimation points than token-level methods,
  enabling accurate advantage estimation based on Monte Carlo (MC) without a
  critic model. SPO features three components with novel strategies: (1) flexible
  segment partition; (2) accurate segment advantage estimation; and (3) policy
  optimization using segment advantages, including a novel probability-mask
  strategy. We further instantiate SPO for two specific scenarios: (1) SPO-chain
  for short chain-of-thought (CoT), featuring novel cutpoint-based partition and
  chain-based advantage estimation, achieving 6-12 percentage point
  improvements in accuracy over PPO and GRPO on GSM8K. (2) SPO-tree for long CoT,
  featuring novel tree-based advantage estimation, which significantly reduces
  the cost of MC estimation, achieving 7-11 percentage point improvements
  over GRPO on MATH500 under 2K and 4K context evaluation. We make our code
  publicly available at https://github.com/AIFrameResearch/SPO.
paper_authors._id: 6841771545662bb7d33ec3c7
paper_authors.name: Yiran Guo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-05T12:41:51.856Z
paper_authors._id: 652be5ba703b3743c2675448
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/AOZ0lE57OYGd7r1ujZTr0.jpeg
paper_authors.fullname: Yiran Guo
paper_authors.type: user
paper_authors.user: gyr66
paper_authors._id: 6841771545662bb7d33ec3c8
paper_authors.name: Lijie Xu
paper_authors._id: 6841771545662bb7d33ec3c9
paper_authors.name: Jie Liu
paper_authors._id: 6841771545662bb7d33ec3ca
paper_authors.name: Dan Ye
paper_authors._id: 6841771545662bb7d33ec3cb
paper_authors.name: Shuang Qiu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-05T12:41:49.841Z
paper_authors._id: 6841784d883860ad472631c5
paper_authors.avatarUrl: /avatars/290e1b9785f79cbbaccffdef24354282.svg
paper_authors.fullname: Shuang Qiu
paper_authors.type: user
paper_authors.user: shq-ml
paper_title: Segment Policy Optimization: Effective Segment-Level Credit Assignment
    in RL for Large Language Models
paper_summary: Enhancing the reasoning capabilities of large language models effectively
  using reinforcement learning (RL) remains a crucial challenge. Existing
  approaches primarily adopt two contrasting advantage estimation granularities:
  Token-level methods (e.g., PPO) aim to provide the fine-grained advantage
  signals but suffer from inaccurate estimation due to difficulties in training
  an accurate critic model. On the other extreme, trajectory-level methods (e.g.,
  GRPO) solely rely on a coarse-grained advantage signal from the final reward,
  leading to imprecise credit assignment. To address these limitations, we
  propose Segment Policy Optimization (SPO), a novel RL framework that leverages
  segment-level advantage estimation at an intermediate granularity, achieving a
  better balance by offering more precise credit assignment than trajectory-level
  methods and requiring fewer estimation points than token-level methods,
  enabling accurate advantage estimation based on Monte Carlo (MC) without a
  critic model. SPO features three components with novel strategies: (1) flexible
  segment partition; (2) accurate segment advantage estimation; and (3) policy
  optimization using segment advantages, including a novel probability-mask
  strategy. We further instantiate SPO for two specific scenarios: (1) SPO-chain
  for short chain-of-thought (CoT), featuring novel cutpoint-based partition and
  chain-based advantage estimation, achieving 6-12 percentage point
  improvements in accuracy over PPO and GRPO on GSM8K. (2) SPO-tree for long CoT,
  featuring novel tree-based advantage estimation, which significantly reduces
  the cost of MC estimation, achieving 7-11 percentage point improvements
  over GRPO on MATH500 under 2K and 4K context evaluation. We make our code
  publicly available at https://github.com/AIFrameResearch/SPO.
paper_ai_summary: The Segment Policy Optimization (SPO) framework improves large language model reasoning via reinforcement learning by offering intermediate granularity advantage estimation, balancing precision and computational efficiency.
paper_ai_keywords: Segment Policy Optimization SPO token-level methods PPO trajectory-level methods GRPO segment-level advantage estimation Monte Carlo (MC) estimation flexible segment partition segment advantage estimation probability-mask strategy short chain-of-thought (CoT) cutpoint-based partition tree-based advantage estimation GSM8K MATH500