title: Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models
summary: Reinforcement learning from human feedback (RLHF) has become a powerful
  post-training paradigm for aligning large language models with human
  preferences. A core challenge in RLHF is constructing accurate reward signals,
  where the conventional Bradley-Terry reward models (BT RMs) often suffer from
  sensitivity to data size and coverage, as well as vulnerability to reward
  hacking. Generative reward models (GenRMs) offer a more robust alternative by
  generating chain-of-thought (CoT) rationales followed by a final reward.
  However, existing GenRMs rely on shallow, vertically scaled reasoning, limiting
  their capacity to handle nuanced or complex (e.g., reasoning-intensive) tasks.
  Moreover, their pairwise preference outputs are incompatible with standard RLHF
  algorithms that require pointwise reward signals. In this work, we introduce
  Think-RM, a training framework that enables long-horizon reasoning in GenRMs by
  modeling an internal thinking process. Rather than producing structured,
  externally provided rationales, Think-RM generates flexible, self-guided
  reasoning traces that support advanced capabilities such as self-reflection,
  hypothetical reasoning, and divergent reasoning. To elicit these reasoning
  abilities, we first warm-up the models by supervised fine-tuning (SFT) over
  long CoT data. We then further improve the model's long-horizon abilities by
  rule-based reinforcement learning (RL). In addition, we propose a novel
  pairwise RLHF pipeline that directly optimizes policies using pairwise
  preference rewards, eliminating the need for pointwise reward conversion and
  enabling more effective use of Think-RM outputs. Experiments show that Think-RM
  achieves state-of-the-art results on RM-Bench, outperforming both BT RM and
  vertically scaled GenRM by 8%. When combined with our pairwise RLHF pipeline,
  it demonstrates superior end-policy performance compared to traditional
  approaches.
paper_authors._id: 6830a6c602886033c4cc5d7b
paper_authors.name: Ilgee Hong
paper_authors._id: 6830a6c602886033c4cc5d7c
paper_authors.name: Changlong Yu
paper_authors._id: 6830a6c602886033c4cc5d7d
paper_authors.name: Liang Qiu
paper_authors._id: 6830a6c602886033c4cc5d7e
paper_authors.name: Weixiang Yan
paper_authors._id: 6830a6c602886033c4cc5d7f
paper_authors.name: Zhenghao Xu
paper_authors._id: 6830a6c602886033c4cc5d80
paper_authors.name: Haoming Jiang
paper_authors._id: 6830a6c602886033c4cc5d81
paper_authors.name: Qingru Zhang
paper_authors._id: 6830a6c602886033c4cc5d82
paper_authors.name: Qin Lu
paper_authors._id: 6830a6c602886033c4cc5d83
paper_authors.name: Xin Liu
paper_authors._id: 6830a6c602886033c4cc5d84
paper_authors.name: Chao Zhang
paper_authors._id: 6830a6c602886033c4cc5d85
paper_authors.name: Tuo Zhao
paper_title: Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models
paper_summary: Reinforcement learning from human feedback (RLHF) has become a powerful
  post-training paradigm for aligning large language models with human
  preferences. A core challenge in RLHF is constructing accurate reward signals,
  where the conventional Bradley-Terry reward models (BT RMs) often suffer from
  sensitivity to data size and coverage, as well as vulnerability to reward
  hacking. Generative reward models (GenRMs) offer a more robust alternative by
  generating chain-of-thought (CoT) rationales followed by a final reward.
  However, existing GenRMs rely on shallow, vertically scaled reasoning, limiting
  their capacity to handle nuanced or complex (e.g., reasoning-intensive) tasks.
  Moreover, their pairwise preference outputs are incompatible with standard RLHF
  algorithms that require pointwise reward signals. In this work, we introduce
  Think-RM, a training framework that enables long-horizon reasoning in GenRMs by
  modeling an internal thinking process. Rather than producing structured,
  externally provided rationales, Think-RM generates flexible, self-guided
  reasoning traces that support advanced capabilities such as self-reflection,
  hypothetical reasoning, and divergent reasoning. To elicit these reasoning
  abilities, we first warm-up the models by supervised fine-tuning (SFT) over
  long CoT data. We then further improve the model's long-horizon abilities by
  rule-based reinforcement learning (RL). In addition, we propose a novel
  pairwise RLHF pipeline that directly optimizes policies using pairwise
  preference rewards, eliminating the need for pointwise reward conversion and
  enabling more effective use of Think-RM outputs. Experiments show that Think-RM
  achieves state-of-the-art results on RM-Bench, outperforming both BT RM and
  vertically scaled GenRM by 8%. When combined with our pairwise RLHF pipeline,
  it demonstrates superior end-policy performance compared to traditional
  approaches.
paper_ai_summary: Think-RM is a framework that enhances generative reward models with long-horizon reasoning and a novel pairwise RLHF pipeline to improve end-policy performance in aligning large language models with human preferences.
paper_ai_keywords: reinforcement learning from human feedback RLHF reward models Bradley-Terry reward models BT RMs generative reward models GenRMs chain-of-thought CoT rationale long-horizon reasoning internal thinking process reasoning traces self-reflection hypothetical reasoning divergent reasoning supervised fine-tuning SFT rule-based reinforcement learning pairwise RLHF pipeline RM-Bench