title: Reward Reasoning Model
summary: Reward models play a critical role in guiding large language models toward
  outputs that align with human expectations. However, an open challenge remains
  in effectively utilizing test-time compute to enhance reward model performance.
  In this work, we introduce Reward Reasoning Models (RRMs), which are
  specifically designed to execute a deliberate reasoning process before
  generating final rewards. Through chain-of-thought reasoning, RRMs leverage
  additional test-time compute for complex queries where appropriate rewards are
  not immediately apparent. To develop RRMs, we implement a reinforcement
  learning framework that fosters self-evolved reward reasoning capabilities
  without requiring explicit reasoning traces as training data. Experimental
  results demonstrate that RRMs achieve superior performance on reward modeling
  benchmarks across diverse domains. Notably, we show that RRMs can adaptively
  exploit test-time compute to further improve reward accuracy. The pretrained
  reward reasoning models are available at
  https://huggingface.co/Reward-Reasoning.
paper_authors._id: 682d4145514c96fbf03f5f76
paper_authors.name: Jiaxin Guo
paper_authors._id: 682d4145514c96fbf03f5f77
paper_authors.name: Zewen Chi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-22T07:17:51.665Z
paper_authors._id: 60f6d61f89b21b8fd2d471c6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60f6d61f89b21b8fd2d471c6/RmLFf97vUoXMoCT3rWbhm.jpeg
paper_authors.fullname: Zewen Chi
paper_authors.type: user
paper_authors.user: CZWin32768
paper_authors._id: 682d4145514c96fbf03f5f78
paper_authors.name: Li Dong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-21T08:40:52.470Z
paper_authors._id: 5df85abada6d0311fd3d5408
paper_authors.avatarUrl: /avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
paper_authors.fullname: Li Dong
paper_authors.type: user
paper_authors.user: unilm
paper_authors._id: 682d4145514c96fbf03f5f79
paper_authors.name: Qingxiu Dong
paper_authors._id: 682d4145514c96fbf03f5f7a
paper_authors.name: Xun Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-21T08:40:50.085Z
paper_authors._id: 62d1227384bfbee86b6eec56
paper_authors.avatarUrl: /avatars/84435f9768a76c0fe9d404dfc2d70be3.svg
paper_authors.fullname: Xun Wu
paper_authors.type: user
paper_authors.user: YUSHUIWX
paper_authors._id: 682d4145514c96fbf03f5f7b
paper_authors.name: Shaohan Huang
paper_authors._id: 682d4145514c96fbf03f5f7c
paper_authors.name: Furu Wei
paper_title: Reward Reasoning Model
paper_summary: Reward models play a critical role in guiding large language models toward
  outputs that align with human expectations. However, an open challenge remains
  in effectively utilizing test-time compute to enhance reward model performance.
  In this work, we introduce Reward Reasoning Models (RRMs), which are
  specifically designed to execute a deliberate reasoning process before
  generating final rewards. Through chain-of-thought reasoning, RRMs leverage
  additional test-time compute for complex queries where appropriate rewards are
  not immediately apparent. To develop RRMs, we implement a reinforcement
  learning framework that fosters self-evolved reward reasoning capabilities
  without requiring explicit reasoning traces as training data. Experimental
  results demonstrate that RRMs achieve superior performance on reward modeling
  benchmarks across diverse domains. Notably, we show that RRMs can adaptively
  exploit test-time compute to further improve reward accuracy. The pretrained
  reward reasoning models are available at
  https://huggingface.co/Reward-Reasoning.
paper_ai_summary: RRMs, employing chain-of-thought reasoning and reinforcement learning, enhance reward model performance by adaptively utilizing test-time compute.
paper_ai_keywords: reward models large language models chain-of-thought reasoning reinforcement learning test-time compute reward accuracy