title: R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement
    Learning
summary: Multimodal Reward Models (MRMs) play a crucial role in enhancing the
  performance of Multimodal Large Language Models (MLLMs). While recent
  advancements have primarily focused on improving the model structure and
  training data of MRMs, there has been limited exploration into the
  effectiveness of long-term reasoning capabilities for reward modeling and how
  to activate these capabilities in MRMs. In this paper, we explore how
  Reinforcement Learning (RL) can be used to improve reward modeling.
  Specifically, we reformulate the reward modeling problem as a rule-based RL
  task. However, we observe that directly applying existing RL algorithms, such
  as Reinforce++, to reward modeling often leads to training instability or even
  collapse due to the inherent limitations of these algorithms. To address this
  issue, we propose the StableReinforce algorithm, which refines the training
  loss, advantage estimation strategy, and reward design of existing RL methods.
  These refinements result in more stable training dynamics and superior
  performance. To facilitate MRM training, we collect 200K preference data from
  diverse datasets. Our reward model, R1-Reward, trained using the
  StableReinforce algorithm on this dataset, significantly improves performance
  on multimodal reward modeling benchmarks. Compared to previous SOTA models,
  R1-Reward achieves a 8.4% improvement on the VL Reward-Bench and a 14.3%
  improvement on the Multimodal Reward Bench. Moreover, with more inference
  compute, R1-Reward's performance is further enhanced, highlighting the
  potential of RL algorithms in optimizing MRMs.
paper_authors._id: 6819762e64ae18f1b6fde347
paper_authors.name: Yi-Fan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T08:57:15.220Z
paper_authors._id: 623d8ca4c29adf5ef6175615
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/623d8ca4c29adf5ef6175615/q7lHao7UPwU1u7YLSP56m.jpeg
paper_authors.fullname: Yi-Fan Zhang
paper_authors.type: user
paper_authors.user: yifanzhang114
paper_authors._id: 6819762e64ae18f1b6fde348
paper_authors.name: Xingyu Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T08:57:24.963Z
paper_authors._id: 664ba004bfd9b93ba4bfb353
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/UHaEcXmMSKvvFDsY3hCnb.jpeg
paper_authors.fullname: LuXingyu
paper_authors.type: user
paper_authors.user: XingyuLu
paper_authors._id: 6819762e64ae18f1b6fde349
paper_authors.name: Xiao Hu
paper_authors._id: 6819762e64ae18f1b6fde34a
paper_authors.name: Chaoyou Fu
paper_authors._id: 6819762e64ae18f1b6fde34b
paper_authors.name: Bin Wen
paper_authors._id: 6819762e64ae18f1b6fde34c
paper_authors.name: Tianke Zhang
paper_authors._id: 6819762e64ae18f1b6fde34d
paper_authors.name: Changyi Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T08:58:28.151Z
paper_authors._id: 673421bf18caf8e877861cc6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/a8UfIZTUTFaCnWmJ_Bztr.png
paper_authors.fullname: Changyi Liu
paper_authors.type: user
paper_authors.user: bhsc24
paper_authors._id: 6819762e64ae18f1b6fde34e
paper_authors.name: Kaiyu Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T08:58:21.743Z
paper_authors._id: 63774c47455f6ad89ac41be1
paper_authors.avatarUrl: /avatars/e7d6048155cdf4497d58aa18523e745e.svg
paper_authors.fullname: Kaiyu Jiang
paper_authors.type: user
paper_authors.user: KaiyuValley
paper_authors._id: 6819762e64ae18f1b6fde34f
paper_authors.name: Kaibing Chen
paper_authors._id: 6819762e64ae18f1b6fde350
paper_authors.name: Kaiyu Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T08:58:06.587Z
paper_authors._id: 66c605e808fee728d0dd94f5
paper_authors.avatarUrl: /avatars/d2ff37fedc5ac1b5b817543b80bf5256.svg
paper_authors.fullname: Kaiyu Tang
paper_authors.type: user
paper_authors.user: KevinTowne
paper_authors._id: 6819762e64ae18f1b6fde351
paper_authors.name: Haojie Ding
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T08:57:59.222Z
paper_authors._id: 6610f64ee94d9046b71e19c8
paper_authors.avatarUrl: /avatars/11cc11199669129a740956d12c7214e8.svg
paper_authors.fullname: Haojie Ding
paper_authors.type: user
paper_authors.user: haojieding
paper_authors._id: 6819762e64ae18f1b6fde352
paper_authors.name: Jiankang Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T08:57:51.408Z
paper_authors._id: 6433abff546e16f17a0f1cd8
paper_authors.avatarUrl: /avatars/7c9bbcba69b823834eb0232da12cc7a9.svg
paper_authors.fullname: chen
paper_authors.type: user
paper_authors.user: jiankang
paper_authors._id: 6819762e64ae18f1b6fde353
paper_authors.name: Fan Yang
paper_authors._id: 6819762e64ae18f1b6fde354
paper_authors.name: Zhang Zhang
paper_authors._id: 6819762e64ae18f1b6fde355
paper_authors.name: Tingting Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T08:57:35.059Z
paper_authors._id: 656453832bdaccfcd5379431
paper_authors.avatarUrl: /avatars/a0d764ce6b3fd05532c7a9cb2f263e33.svg
paper_authors.fullname: Gao Ting
paper_authors.type: user
paper_authors.user: TingTingGao
paper_authors._id: 6819762e64ae18f1b6fde356
paper_authors.name: Liang Wang
paper_title: R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement
    Learning
paper_summary: Multimodal Reward Models (MRMs) play a crucial role in enhancing the
  performance of Multimodal Large Language Models (MLLMs). While recent
  advancements have primarily focused on improving the model structure and
  training data of MRMs, there has been limited exploration into the
  effectiveness of long-term reasoning capabilities for reward modeling and how
  to activate these capabilities in MRMs. In this paper, we explore how
  Reinforcement Learning (RL) can be used to improve reward modeling.
  Specifically, we reformulate the reward modeling problem as a rule-based RL
  task. However, we observe that directly applying existing RL algorithms, such
  as Reinforce++, to reward modeling often leads to training instability or even
  collapse due to the inherent limitations of these algorithms. To address this
  issue, we propose the StableReinforce algorithm, which refines the training
  loss, advantage estimation strategy, and reward design of existing RL methods.
  These refinements result in more stable training dynamics and superior
  performance. To facilitate MRM training, we collect 200K preference data from
  diverse datasets. Our reward model, R1-Reward, trained using the
  StableReinforce algorithm on this dataset, significantly improves performance
  on multimodal reward modeling benchmarks. Compared to previous SOTA models,
  R1-Reward achieves a 8.4% improvement on the VL Reward-Bench and a 14.3%
  improvement on the Multimodal Reward Bench. Moreover, with more inference
  compute, R1-Reward's performance is further enhanced, highlighting the
  potential of RL algorithms in optimizing MRMs.
paper_ai_summary: Using Reinforcement Learning to improve reward modeling in Multimodal Reward Models leads to more stable training and superior performance.
paper_ai_keywords: Reinforcement Learning RL StableReinforce training loss advantage estimation reward design Multimodal Reward Models preference data VL Reward-Bench Multimodal Reward Bench