title: NOVER: Incentive Training for Language Models via Verifier-Free
    Reinforcement Learning
summary: Recent advances such as DeepSeek R1-Zero highlight the effectiveness of
  incentive training, a reinforcement learning paradigm that computes rewards
  solely based on the final answer part of a language model's output, thereby
  encouraging the generation of intermediate reasoning steps. However, these
  methods fundamentally rely on external verifiers, which limits their
  applicability to domains like mathematics and coding where such verifiers are
  readily available. Although reward models can serve as verifiers, they require
  high-quality annotated data and are costly to train. In this work, we propose
  NOVER, NO-VERifier Reinforcement Learning, a general reinforcement learning
  framework that requires only standard supervised fine-tuning data with no need
  for an external verifier. NOVER enables incentive training across a wide range
  of text-to-text tasks and outperforms the model of the same size distilled from
  large reasoning models such as DeepSeek R1 671B by 7.7 percent. Moreover, the
  flexibility of NOVER enables new possibilities for optimizing large language
  models, such as inverse incentive training.
paper_authors._id: 68342cb2924393051af84722
paper_authors.name: Wei Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T14:07:36.205Z
paper_authors._id: 66e2932e5c100c12aa2def39
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/FiQ5Fap-qVqnXeULGPYs6.png
paper_authors.fullname: weiliu
paper_authors.type: user
paper_authors.user: thinkwee
paper_authors._id: 68342cb2924393051af84723
paper_authors.name: Siya Qi
paper_authors._id: 68342cb2924393051af84724
paper_authors.name: Xinyu Wang
paper_authors._id: 68342cb2924393051af84725
paper_authors.name: Chen Qian
paper_authors._id: 68342cb2924393051af84726
paper_authors.name: Yali Du
paper_authors._id: 68342cb2924393051af84727
paper_authors.name: Yulan He
paper_title: NOVER: Incentive Training for Language Models via Verifier-Free
    Reinforcement Learning
paper_summary: Recent advances such as DeepSeek R1-Zero highlight the effectiveness of
  incentive training, a reinforcement learning paradigm that computes rewards
  solely based on the final answer part of a language model's output, thereby
  encouraging the generation of intermediate reasoning steps. However, these
  methods fundamentally rely on external verifiers, which limits their
  applicability to domains like mathematics and coding where such verifiers are
  readily available. Although reward models can serve as verifiers, they require
  high-quality annotated data and are costly to train. In this work, we propose
  NOVER, NO-VERifier Reinforcement Learning, a general reinforcement learning
  framework that requires only standard supervised fine-tuning data with no need
  for an external verifier. NOVER enables incentive training across a wide range
  of text-to-text tasks and outperforms the model of the same size distilled from
  large reasoning models such as DeepSeek R1 671B by 7.7 percent. Moreover, the
  flexibility of NOVER enables new possibilities for optimizing large language
  models, such as inverse incentive training.
paper_ai_summary: NOVER, a reinforcement learning framework that eliminates the need for external verifiers, enhances language model performance across text-to-text tasks.
paper_ai_keywords: incentive training reinforcement learning NOVER NO-VERifier Reinforcement Learning DeepSeek R1-Zero DeepSeek R1 671B inverse incentive training