title: RL Tango: Reinforcing Generator and Verifier Together for Language
    Reasoning
summary: Reinforcement learning (RL) has recently emerged as a compelling approach for
  enhancing the reasoning capabilities of large language models (LLMs), where an
  LLM generator serves as a policy guided by a verifier (reward model). However,
  current RL post-training methods for LLMs typically use verifiers that are
  fixed (rule-based or frozen pretrained) or trained discriminatively via
  supervised fine-tuning (SFT). Such designs are susceptible to reward hacking
  and generalize poorly beyond their training distributions. To overcome these
  limitations, we propose Tango, a novel framework that uses RL to concurrently
  train both an LLM generator and a verifier in an interleaved manner. A central
  innovation of Tango is its generative, process-level LLM verifier, which is
  trained via RL and co-evolves with the generator. Importantly, the verifier is
  trained solely based on outcome-level verification correctness rewards without
  requiring explicit process-level annotations. This generative RL-trained
  verifier exhibits improved robustness and superior generalization compared to
  deterministic or SFT-trained verifiers, fostering effective mutual
  reinforcement with the generator. Extensive experiments demonstrate that both
  components of Tango achieve state-of-the-art results among 7B/8B-scale models:
  the generator attains best-in-class performance across five competition-level
  math benchmarks and four challenging out-of-domain reasoning tasks, while the
  verifier leads on the ProcessBench dataset. Remarkably, both components exhibit
  particularly substantial improvements on the most difficult mathematical
  reasoning problems. Code is at: https://github.com/kaiwenzha/rl-tango.
paper_authors._id: 682ecc348573bac82a974c8b
paper_authors.name: Kaiwen Zha
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-22T07:15:06.379Z
paper_authors._id: 6801fbe4cae7a6623f66ef64
paper_authors.avatarUrl: /avatars/383a72cc890946d89a185b8482d1b828.svg
paper_authors.fullname: Kaiwen Zha
paper_authors.type: user
paper_authors.user: sunshinekevin
paper_authors._id: 682ecc348573bac82a974c8c
paper_authors.name: Zhengqi Gao
paper_authors._id: 682ecc348573bac82a974c8d
paper_authors.name: Maohao Shen
paper_authors._id: 682ecc348573bac82a974c8e
paper_authors.name: Zhang-Wei Hong
paper_authors._id: 682ecc348573bac82a974c8f
paper_authors.name: Duane S. Boning
paper_authors._id: 682ecc348573bac82a974c90
paper_authors.name: Dina Katabi
paper_title: RL Tango: Reinforcing Generator and Verifier Together for Language
    Reasoning
paper_summary: Reinforcement learning (RL) has recently emerged as a compelling approach for
  enhancing the reasoning capabilities of large language models (LLMs), where an
  LLM generator serves as a policy guided by a verifier (reward model). However,
  current RL post-training methods for LLMs typically use verifiers that are
  fixed (rule-based or frozen pretrained) or trained discriminatively via
  supervised fine-tuning (SFT). Such designs are susceptible to reward hacking
  and generalize poorly beyond their training distributions. To overcome these
  limitations, we propose Tango, a novel framework that uses RL to concurrently
  train both an LLM generator and a verifier in an interleaved manner. A central
  innovation of Tango is its generative, process-level LLM verifier, which is
  trained via RL and co-evolves with the generator. Importantly, the verifier is
  trained solely based on outcome-level verification correctness rewards without
  requiring explicit process-level annotations. This generative RL-trained
  verifier exhibits improved robustness and superior generalization compared to
  deterministic or SFT-trained verifiers, fostering effective mutual
  reinforcement with the generator. Extensive experiments demonstrate that both
  components of Tango achieve state-of-the-art results among 7B/8B-scale models:
  the generator attains best-in-class performance across five competition-level
  math benchmarks and four challenging out-of-domain reasoning tasks, while the
  verifier leads on the ProcessBench dataset. Remarkably, both components exhibit
  particularly substantial improvements on the most difficult mathematical
  reasoning problems. Code is at: https://github.com/kaiwenzha/rl-tango.
paper_ai_summary: Tango is an RL framework that concurrently trains a generative LLM and a RL-trained verifier, achieving superior robustness and generalization in math benchmarks and out-of-domain reasoning tasks.
paper_ai_keywords: Reinforcement learning (RL) large language models (LLMs) policy verifier reward model reward hacking supervised fine-tuning (SFT) generative process-level verification correctness rewards mutual reinforcement ProcessBench dataset mathematical reasoning