title: Expanding RL with Verifiable Rewards Across Diverse Domains
summary: Reinforcement learning (RL) with verifiable rewards (RLVR) has shown
  promising results in mathematical reasoning and coding tasks where
  well-structured reference answers are available. However, its applicability to
  broader domains remains underexplored. In this work, we study the extension of
  RLVR to more diverse domains such as medicine, chemistry, psychology, and
  economics. We observe high agreement in binary judgments across different large
  language models (LLMs) when objective reference answers exist, which challenges
  the necessity of large-scale annotation for training domain-specific reward
  models. To address the limitations of binary rewards when handling unstructured
  reference answers, we further incorporate model-based soft scoring into RLVR to
  improve its flexibility. Our experiments show that a distilled generative
  reward model can serve as an effective cross-domain verifier, providing
  reliable reward signals for RL without requiring domain-specific annotations.
  By fine-tuning a base 7B model using various RL algorithms against our reward
  model, we obtain policies that outperform state-of-the-art open-source aligned
  LLMs such as Qwen2.5-72B-Instruct and DeepSeek-R1-Distill-Qwen-32B by a large
  margin, across domains in free-form answer settings. This also strengthens
  RLVR's robustness and scalability, highlighting its potential for real-world
  applications with noisy or weak labels.
paper_authors._id: 67eb759cb9fa8908e1934f21
paper_authors.name: Yi Su
paper_authors._id: 67eb759cb9fa8908e1934f22
paper_authors.name: Dian Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-01T08:13:47.119Z
paper_authors._id: 62d58fd53bf5e059f7cc3245
paper_authors.avatarUrl: /avatars/7a4f3ee4a37245f67efd26749d66a706.svg
paper_authors.fullname: Dian Yu
paper_authors.type: user
paper_authors.user: yudian
paper_authors._id: 67eb759cb9fa8908e1934f23
paper_authors.name: Linfeng Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-01T08:13:54.065Z
paper_authors._id: 64c94eddcb2f1bf0e7db5a4d
paper_authors.avatarUrl: /avatars/f7e2532d3c85d5e5b5a02c579ea68c3a.svg
paper_authors.fullname: Linfeng Song
paper_authors.type: user
paper_authors.user: freesunshine0316
paper_authors._id: 67eb759cb9fa8908e1934f24
paper_authors.name: Juntao Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-01T08:14:03.120Z
paper_authors._id: 6670e285b0c03c4e9d6e0985
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/uCZHm4gKSHZ2b0hpHWgZv.jpeg
paper_authors.fullname: Juntao Li
paper_authors.type: user
paper_authors.user: douvleplus
paper_authors._id: 67eb759cb9fa8908e1934f25
paper_authors.name: Haitao Mi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-01T08:14:10.594Z
paper_authors._id: 65147a1426fbd558dbd08f1b
paper_authors.avatarUrl: /avatars/86574ee2d5c22e940be1c4e50be88675.svg
paper_authors.fullname: Haitao Mi
paper_authors.type: user
paper_authors.user: haitaominlp
paper_authors._id: 67eb759cb9fa8908e1934f26
paper_authors.name: Zhaopeng Tu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-01T08:14:16.978Z
paper_authors._id: 67485743561b1e6f9579389f
paper_authors.avatarUrl: /avatars/8a4cc63bd7be388010bc329bb74582a1.svg
paper_authors.fullname: Zhaopeng Tu
paper_authors.type: user
paper_authors.user: zptu
paper_authors._id: 67eb759cb9fa8908e1934f27
paper_authors.name: Min Zhang
paper_authors._id: 67eb759cb9fa8908e1934f28
paper_authors.name: Dong Yu
paper_title: Expanding RL with Verifiable Rewards Across Diverse Domains
paper_summary: Reinforcement learning (RL) with verifiable rewards (RLVR) has shown
  promising results in mathematical reasoning and coding tasks where
  well-structured reference answers are available. However, its applicability to
  broader domains remains underexplored. In this work, we study the extension of
  RLVR to more diverse domains such as medicine, chemistry, psychology, and
  economics. We observe high agreement in binary judgments across different large
  language models (LLMs) when objective reference answers exist, which challenges
  the necessity of large-scale annotation for training domain-specific reward
  models. To address the limitations of binary rewards when handling unstructured
  reference answers, we further incorporate model-based soft scoring into RLVR to
  improve its flexibility. Our experiments show that a distilled generative
  reward model can serve as an effective cross-domain verifier, providing
  reliable reward signals for RL without requiring domain-specific annotations.
  By fine-tuning a base 7B model using various RL algorithms against our reward
  model, we obtain policies that outperform state-of-the-art open-source aligned
  LLMs such as Qwen2.5-72B-Instruct and DeepSeek-R1-Distill-Qwen-32B by a large
  margin, across domains in free-form answer settings. This also strengthens
  RLVR's robustness and scalability, highlighting its potential for real-world
  applications with noisy or weak labels.
paper_ai_summary: Reinforcement learning with verifiable rewards extended to diverse domains using model-based soft scoring outperforms existing LLMs in free-form answer settings with reliable reward signals.
paper_ai_keywords: reinforcement learning with verifiable rewards large language models binary judgments objective reference answers model-based soft scoring distilled generative reward model fine-tuning RL algorithms Qwen2.5-72B-Instruct DeepSeek-R1-Distill-Qwen-32B