title: SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy
    Optimization
summary: Large language models (LLMs) exhibit varying levels of confidence across
  input prompts (questions): some lead to consistent, semantically similar
  answers, while others yield diverse or contradictory outputs. This variation
  reflects LLM's uncertainty about the input prompt, a signal of how confidently
  the model understands a given problem. However, vanilla Group Relative Policy
  Optimization (GRPO) treats all prompts equally during policy updates, ignoring
  this important information about the model's knowledge boundaries. To address
  this limitation, we propose SEED-GRPO (Semantic Entropy EnhanceD GRPO), which
  explicitly measures LLMs' uncertainty of the input prompts semantic entropy.
  Semantic entropy measures the diversity of meaning in multiple generated
  answers given a prompt and uses this to modulate the magnitude of policy
  updates. This uncertainty-aware training mechanism enables dynamic adjustment
  of policy update magnitudes based on question uncertainty. It allows more
  conservative updates on high-uncertainty questions while maintaining the
  original learning signal on confident ones. Experimental results on five
  mathematical reasoning benchmarks (AIME24 56.7, AMC 68.7, MATH 83.4, Minerva
  34.2, and OlympiadBench 48.0) demonstrate that SEED-GRPO achieves new
  state-of-the-art performance in average accuracy, validating the effectiveness
  of uncertainty-aware policy optimization.
paper_authors._id: 682be9d44182ec69eba2a1f1
paper_authors.name: Minghan Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-20T07:20:47.411Z
paper_authors._id: 6253fb19d3e43081e102af4f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6253fb19d3e43081e102af4f/0M0Ksd8rzm_g4k-hl0xP6.jpeg
paper_authors.fullname: MC
paper_authors.type: user
paper_authors.user: Dreamer312
paper_authors._id: 682be9d44182ec69eba2a1f2
paper_authors.name: Guikun Chen
paper_authors._id: 682be9d44182ec69eba2a1f3
paper_authors.name: Wenguan Wang
paper_authors._id: 682be9d44182ec69eba2a1f4
paper_authors.name: Yi Yang
paper_title: SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy
    Optimization
paper_summary: Large language models (LLMs) exhibit varying levels of confidence across
  input prompts (questions): some lead to consistent, semantically similar
  answers, while others yield diverse or contradictory outputs. This variation
  reflects LLM's uncertainty about the input prompt, a signal of how confidently
  the model understands a given problem. However, vanilla Group Relative Policy
  Optimization (GRPO) treats all prompts equally during policy updates, ignoring
  this important information about the model's knowledge boundaries. To address
  this limitation, we propose SEED-GRPO (Semantic Entropy EnhanceD GRPO), which
  explicitly measures LLMs' uncertainty of the input prompts semantic entropy.
  Semantic entropy measures the diversity of meaning in multiple generated
  answers given a prompt and uses this to modulate the magnitude of policy
  updates. This uncertainty-aware training mechanism enables dynamic adjustment
  of policy update magnitudes based on question uncertainty. It allows more
  conservative updates on high-uncertainty questions while maintaining the
  original learning signal on confident ones. Experimental results on five
  mathematical reasoning benchmarks (AIME24 56.7, AMC 68.7, MATH 83.4, Minerva
  34.2, and OlympiadBench 48.0) demonstrate that SEED-GRPO achieves new
  state-of-the-art performance in average accuracy, validating the effectiveness
  of uncertainty-aware policy optimization.
paper_ai_summary: SEED-GRPO enhances Group Relative Policy Optimization by adjusting policy updates based on the uncertainty of input prompts, achieving state-of-the-art performance in mathematical reasoning benchmarks.
paper_ai_keywords: Large language models (LLMs) Group Relative Policy Optimization (GRPO) Semantic Entropy EnhanceD GRPO (SEED-GRPO) semantic entropy uncertainty-aware training