title: Reinforcing General Reasoning without Verifiers
summary: The recent paradigm shift towards training large language models (LLMs) using
  DeepSeek-R1-Zero-style reinforcement learning (RL) on verifiable rewards has
  led to impressive advancements in code and mathematical reasoning. However,
  this methodology is limited to tasks where rule-based answer verification is
  possible and does not naturally extend to real-world domains such as chemistry,
  healthcare, engineering, law, biology, business, and economics. Current
  practical workarounds use an additional LLM as a model-based verifier; however,
  this introduces issues such as reliance on a strong verifier LLM,
  susceptibility to reward hacking, and the practical burden of maintaining the
  verifier model in memory during training. To address this and extend
  DeepSeek-R1-Zero-style training to general reasoning domains, we propose a
  verifier-free method (VeriFree) that bypasses answer verification and instead
  uses RL to directly maximize the probability of generating the reference
  answer. We compare VeriFree with verifier-based methods and demonstrate that,
  in addition to its significant practical benefits and reduced compute
  requirements, VeriFree matches and even surpasses verifier-based methods on
  extensive evaluations across MMLU-Pro, GPQA, SuperGPQA, and math-related
  benchmarks. Moreover, we provide insights into this method from multiple
  perspectives: as an elegant integration of training both the policy and
  implicit verifier in a unified model, and as a variational optimization
  approach. Code is available at https://github.com/sail-sg/VeriFree.
paper_authors._id: 68372228726e45ea3f19fd59
paper_authors.name: Xiangxin Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:48:45.621Z
paper_authors._id: 66129c7b50350afe76757262
paper_authors.avatarUrl: /avatars/a2f4fac076b9d658a0d904ed54960f6f.svg
paper_authors.fullname: Xiangxin Zhou
paper_authors.type: user
paper_authors.user: zhouxiangxin
paper_authors._id: 68372228726e45ea3f19fd5a
paper_authors.name: Zichen Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T19:13:48.854Z
paper_authors._id: 65f5392c68b8e0cb3c9977a2
paper_authors.avatarUrl: /avatars/aa64772475098e8a135c13072fde6744.svg
paper_authors.fullname: Zichen
paper_authors.type: user
paper_authors.user: lkevinzc
paper_authors._id: 68372228726e45ea3f19fd5b
paper_authors.name: Anya Sims
paper_authors._id: 68372228726e45ea3f19fd5c
paper_authors.name: Haonan Wang
paper_authors._id: 68372228726e45ea3f19fd5d
paper_authors.name: Tianyu Pang
paper_authors._id: 68372228726e45ea3f19fd5e
paper_authors.name: Chongxuan Li
paper_authors._id: 68372228726e45ea3f19fd5f
paper_authors.name: Liang Wang
paper_authors._id: 68372228726e45ea3f19fd60
paper_authors.name: Min Lin
paper_authors._id: 68372228726e45ea3f19fd61
paper_authors.name: Chao Du
paper_title: Reinforcing General Reasoning without Verifiers
paper_summary: The recent paradigm shift towards training large language models (LLMs) using
  DeepSeek-R1-Zero-style reinforcement learning (RL) on verifiable rewards has
  led to impressive advancements in code and mathematical reasoning. However,
  this methodology is limited to tasks where rule-based answer verification is
  possible and does not naturally extend to real-world domains such as chemistry,
  healthcare, engineering, law, biology, business, and economics. Current
  practical workarounds use an additional LLM as a model-based verifier; however,
  this introduces issues such as reliance on a strong verifier LLM,
  susceptibility to reward hacking, and the practical burden of maintaining the
  verifier model in memory during training. To address this and extend
  DeepSeek-R1-Zero-style training to general reasoning domains, we propose a
  verifier-free method (VeriFree) that bypasses answer verification and instead
  uses RL to directly maximize the probability of generating the reference
  answer. We compare VeriFree with verifier-based methods and demonstrate that,
  in addition to its significant practical benefits and reduced compute
  requirements, VeriFree matches and even surpasses verifier-based methods on
  extensive evaluations across MMLU-Pro, GPQA, SuperGPQA, and math-related
  benchmarks. Moreover, we provide insights into this method from multiple
  perspectives: as an elegant integration of training both the policy and
  implicit verifier in a unified model, and as a variational optimization
  approach. Code is available at https://github.com/sail-sg/VeriFree.
paper_ai_summary: A verifier-free method (VeriFree) is introduced to extend reinforcement learning training of large language models to general reasoning domains, improving efficiency and performance compared to verifier-based methods.
paper_ai_keywords: DeepSeek-R1-Zero reinforcement learning RL large language models LLMs VeriFree MMLU-Pro GPQA SuperGPQA