title: Inference-Time Scaling for Generalist Reward Modeling
summary: Reinforcement learning (RL) has been widely adopted in post-training for
  large language models (LLMs) at scale. Recently, the incentivization of
  reasoning capabilities in LLMs from RL indicates that proper learning
  methods could enable effective inference-time scalability. A key challenge of
  RL is to obtain accurate reward signals for LLMs in various domains beyond
  verifiable questions or artificial rules. In this work, we investigate how to
  improve reward modeling (RM) with more inference compute for general queries,
  i.e. the inference-time scalability of generalist RM, and further,
  how to improve the effectiveness of performance-compute scaling with proper
  learning methods. For the RM approach, we adopt pointwise generative reward
  modeling (GRM) to enable flexibility for different input types and potential
  for inference-time scaling. For the learning method, we propose Self-Principled
  Critique Tuning (SPCT) to foster scalable reward generation behaviors in GRMs
  through online RL, to generate principles adaptively and critiques accurately,
  resulting in DeepSeek-GRM models. Furthermore, for effective
  inference-time scaling, we use parallel sampling to expand compute usage, and
  introduce a meta RM to guide voting process for better scaling performance.
  Empirically, we show that SPCT significantly improves the quality and
  scalability of GRMs, outperforming existing methods and models in various RM
  benchmarks without severe biases, and could achieve better performance compared
  to training-time scaling. DeepSeek-GRM still meets challenges in some tasks,
  which we believe can be addressed by future efforts in generalist reward
  systems. The models will be released and open-sourced.
paper_authors._id: 67efb66889b2961e6d95eb5b
paper_authors.name: Zijun Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-08T06:54:11.123Z
paper_authors._id: 6468c76bff18750165a64df3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6468c76bff18750165a64df3/dHhE62SHOSJZjyU60vgh7.jpeg
paper_authors.fullname: Zijun Liu
paper_authors.type: user
paper_authors.user: BBQGOD
paper_authors._id: 67efb66889b2961e6d95eb5c
paper_authors.name: Peiyi Wang
paper_authors._id: 67efb66889b2961e6d95eb5d
paper_authors.name: Runxin Xu
paper_authors._id: 67efb66889b2961e6d95eb5e
paper_authors.name: Shirong Ma
paper_authors._id: 67efb66889b2961e6d95eb5f
paper_authors.name: Chong Ruan
paper_authors._id: 67efb66889b2961e6d95eb60
paper_authors.name: Peng Li
paper_authors._id: 67efb66889b2961e6d95eb61
paper_authors.name: Yang Liu
paper_authors._id: 67efb66889b2961e6d95eb62
paper_authors.name: Yu Wu
paper_title: Inference-Time Scaling for Generalist Reward Modeling
paper_summary: Reinforcement learning (RL) has been widely adopted in post-training for
  large language models (LLMs) at scale. Recently, the incentivization of
  reasoning capabilities in LLMs from RL indicates that proper learning
  methods could enable effective inference-time scalability. A key challenge of
  RL is to obtain accurate reward signals for LLMs in various domains beyond
  verifiable questions or artificial rules. In this work, we investigate how to
  improve reward modeling (RM) with more inference compute for general queries,
  i.e. the inference-time scalability of generalist RM, and further,
  how to improve the effectiveness of performance-compute scaling with proper
  learning methods. For the RM approach, we adopt pointwise generative reward
  modeling (GRM) to enable flexibility for different input types and potential
  for inference-time scaling. For the learning method, we propose Self-Principled
  Critique Tuning (SPCT) to foster scalable reward generation behaviors in GRMs
  through online RL, to generate principles adaptively and critiques accurately,
  resulting in DeepSeek-GRM models. Furthermore, for effective
  inference-time scaling, we use parallel sampling to expand compute usage, and
  introduce a meta RM to guide voting process for better scaling performance.
  Empirically, we show that SPCT significantly improves the quality and
  scalability of GRMs, outperforming existing methods and models in various RM
  benchmarks without severe biases, and could achieve better performance compared
  to training-time scaling. DeepSeek-GRM still meets challenges in some tasks,
  which we believe can be addressed by future efforts in generalist reward
  systems. The models will be released and open-sourced.
paper_ai_summary: Self-Principled Critique Tuning enhances pointwise generative reward modeling for large language models, improving scalability and quality compared to existing methods.
paper_ai_keywords: reinforcement learning reward modeling pointwise generative reward modeling Self-Principled Critique Tuning online RL parallel sampling meta RM