Expanding RL with Verifiable Rewards Across Diverse Domains
Reinforcement learning (RL) with verifiable rewards (RLVR) has shown
promising results in mathematical reasoning and coding tasks where
well-structured reference answers are available. However, its applicability to
broader domains remains underexplored. In this work, we study the extension of
RLVR to more diverse domains such as medicine, chemistry, psychology, and
economics. We observe high agreement in binary judgments across different large
language models (LLMs) when objective reference answers exist, which challenges
the necessity of large-scale annotation for training domain-specific reward
models. To address the limitations of binary rewards when handling unstructured
reference answers, we further incorporate model-based soft scoring into RLVR to
improve its flexibility. Our experiments show that a distilled generative
reward model can serve as an effective cross-domain verifier, providing
reliable reward signals for RL without requiring domain-specific annotations.
By fine-tuning a base 7B model using various RL algorithms against our reward
model, we obtain policies that outperform state-of-the-art open-source aligned
LLMs such as Qwen2.5-72B-Instruct and DeepSeek-R1-Distill-Qwen-32B by a large
margin, across domains in free-form answer settings. This also strengthens
RLVR's robustness and scalability, highlighting its potential for real-world
applications with noisy or weak labels.
67eb759cb9fa8908e1934f21
Yi Su
67eb759cb9fa8908e1934f22
Dian Yu
admin_assigned
2025-04-01T08:13:47.119Z
62d58fd53bf5e059f7cc3245
/avatars/7a4f3ee4a37245f67efd26749d66a706.svg
Dian Yu
user
yudian
67eb759cb9fa8908e1934f23
Linfeng Song
admin_assigned
2025-04-01T08:13:54.065Z
64c94eddcb2f1bf0e7db5a4d
/avatars/f7e2532d3c85d5e5b5a02c579ea68c3a.svg
Linfeng Song
user
freesunshine0316
67eb759cb9fa8908e1934f24
Juntao Li
admin_assigned
2025-04-01T08:14:03.120Z
6670e285b0c03c4e9d6e0985
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/uCZHm4gKSHZ2b0hpHWgZv.jpeg
Juntao Li
user
douvleplus
67eb759cb9fa8908e1934f25
Haitao Mi
admin_assigned
2025-04-01T08:14:10.594Z
65147a1426fbd558dbd08f1b
/avatars/86574ee2d5c22e940be1c4e50be88675.svg
Haitao Mi
user
haitaominlp
67eb759cb9fa8908e1934f26
Zhaopeng Tu
admin_assigned
2025-04-01T08:14:16.978Z
67485743561b1e6f9579389f
/avatars/8a4cc63bd7be388010bc329bb74582a1.svg
Zhaopeng Tu
user
zptu
67eb759cb9fa8908e1934f27
Min Zhang
67eb759cb9fa8908e1934f28
Dong Yu
Expanding RL with Verifiable Rewards Across Diverse Domains
Reinforcement learning (RL) with verifiable rewards (RLVR) has shown
promising results in mathematical reasoning and coding tasks where
well-structured reference answers are available. However, its applicability to
broader domains remains underexplored. In this work, we study the extension of
RLVR to more diverse domains such as medicine, chemistry, psychology, and
economics. We observe high agreement in binary judgments across different large
language models (LLMs) when objective reference answers exist, which challenges
the necessity of large-scale annotation for training domain-specific reward
models. To address the limitations of binary rewards when handling unstructured
reference answers, we further incorporate model-based soft scoring into RLVR to
improve its flexibility. Our experiments show that a distilled generative
reward model can serve as an effective cross-domain verifier, providing
reliable reward signals for RL without requiring domain-specific annotations.
By fine-tuning a base 7B model using various RL algorithms against our reward
model, we obtain policies that outperform state-of-the-art open-source aligned
LLMs such as Qwen2.5-72B-Instruct and DeepSeek-R1-Distill-Qwen-32B by a large
margin, across domains in free-form answer settings. This also strengthens
RLVR's robustness and scalability, highlighting its potential for real-world
applications with noisy or weak labels.
Reinforcement learning with verifiable rewards extended to diverse domains using model-based soft scoring outperforms existing LLMs in free-form answer settings with reliable reward signals.
reinforcement learning with verifiable rewards large language models binary judgments objective reference answers model-based soft scoring distilled generative reward model fine-tuning RL algorithms Qwen2.5-72B-Instruct DeepSeek-R1-Distill-Qwen-32B