title: Learning to Reason without External Rewards
summary: Training large language models (LLMs) for complex reasoning via Reinforcement
  Learning with Verifiable Rewards (RLVR) is effective but limited by reliance on
  costly, domain-specific supervision. We explore Reinforcement Learning from
  Internal Feedback (RLIF), a framework that enables LLMs to learn from intrinsic
  signals without external rewards or labeled data. We propose Intuitor, an RLIF
  method that uses a model's own confidence, termed self-certainty, as its sole
  reward signal. Intuitor replaces external rewards in Group Relative Policy
  Optimization (GRPO) with self-certainty scores, enabling fully unsupervised
  learning. Experiments demonstrate that Intuitor matches GRPO's performance on
  mathematical benchmarks while achieving superior generalization to
  out-of-domain tasks like code generation, without requiring gold solutions or
  test cases. Our findings show that intrinsic model signals can drive effective
  learning across domains, offering a scalable alternative to RLVR for autonomous
  AI systems where verifiable rewards are unavailable. Code is available at
  https://github.com/sunblaze-ucb/Intuitor
paper_authors._id: 683523bcb0f9c65224abd710
paper_authors.name: Xuandong Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T07:51:51.514Z
paper_authors._id: 6275a465597c70eb8949fce5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6275a465597c70eb8949fce5/ph4UogqMurMB0hSXZC38w.png
paper_authors.fullname: Xuandong Zhao
paper_authors.type: user
paper_authors.user: Xuandong
paper_authors._id: 683523bcb0f9c65224abd711
paper_authors.name: Zhewei Kang
paper_authors._id: 683523bcb0f9c65224abd712
paper_authors.name: Aosong Feng
paper_authors._id: 683523bcb0f9c65224abd713
paper_authors.name: Sergey Levine
paper_authors._id: 683523bcb0f9c65224abd714
paper_authors.name: Dawn Song
paper_title: Learning to Reason without External Rewards
paper_summary: Training large language models (LLMs) for complex reasoning via Reinforcement
  Learning with Verifiable Rewards (RLVR) is effective but limited by reliance on
  costly, domain-specific supervision. We explore Reinforcement Learning from
  Internal Feedback (RLIF), a framework that enables LLMs to learn from intrinsic
  signals without external rewards or labeled data. We propose Intuitor, an RLIF
  method that uses a model's own confidence, termed self-certainty, as its sole
  reward signal. Intuitor replaces external rewards in Group Relative Policy
  Optimization (GRPO) with self-certainty scores, enabling fully unsupervised
  learning. Experiments demonstrate that Intuitor matches GRPO's performance on
  mathematical benchmarks while achieving superior generalization to
  out-of-domain tasks like code generation, without requiring gold solutions or
  test cases. Our findings show that intrinsic model signals can drive effective
  learning across domains, offering a scalable alternative to RLVR for autonomous
  AI systems where verifiable rewards are unavailable. Code is available at
  https://github.com/sunblaze-ucb/Intuitor
paper_ai_summary: Intuitor, a Reinforcement Learning from Internal Feedback method, uses self-certainty as a reward signal to enable unsupervised learning of large language models, achieving performance comparable to GRPO on benchmarks and superior generalization.
paper_ai_keywords: Reinforcement Learning with Verifiable Rewards Reinforcement Learning from Internal Feedback Group Relative Policy Optimization self-certainty unsupervised learning