title: Unified Multimodal Chain-of-Thought Reward Model through Reinforcement
    Fine-Tuning
summary: Recent advances in multimodal Reward Models (RMs) have shown significant
  promise in delivering reward signals to align vision models with human
  preferences. However, current RMs are generally restricted to providing direct
  responses or engaging in shallow reasoning processes with limited depth, often
  leading to inaccurate reward signals. We posit that incorporating explicit long
  chains of thought (CoT) into the reward reasoning process can significantly
  strengthen their reliability and robustness. Furthermore, we believe that once
  RMs internalize CoT reasoning, their direct response accuracy can also be
  improved through implicit reasoning capabilities. To this end, this paper
  proposes UnifiedReward-Think, the first unified multimodal CoT-based reward
  model, capable of multi-dimensional, step-by-step long-chain reasoning for both
  visual understanding and generation reward tasks. Specifically, we adopt an
  exploration-driven reinforcement fine-tuning approach to elicit and incentivize
  the model's latent complex reasoning ability: (1) We first use a small amount
  of image generation preference data to distill the reasoning process of GPT-4o,
  which is then used for the model's cold start to learn the format and structure
  of CoT reasoning. (2) Subsequently, by leveraging the model's prior knowledge
  and generalization capabilities, we prepare large-scale unified multimodal
  preference data to elicit the model's reasoning process across various vision
  tasks. During this phase, correct reasoning outputs are retained for rejection
  sampling to refine the model (3) while incorrect predicted samples are finally
  used for Group Relative Policy Optimization (GRPO) based reinforcement
  fine-tuning, enabling the model to explore diverse reasoning paths and optimize
  for correct and robust solutions. Extensive experiments across various vision
  reward tasks demonstrate the superiority of our model.
paper_authors._id: 681aac4fd31f567552f0cc0e
paper_authors.name: Yibin Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-08T08:59:41.170Z
paper_authors._id: 654c6845bac6e6e49895a5b5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/QwUOUujUAyiV4tAhB_moO.png
paper_authors.fullname: SII-Yibin Wang
paper_authors.type: user
paper_authors.user: CodeGoat24
paper_authors._id: 681aac4fd31f567552f0cc0f
paper_authors.name: Zhimin Li
paper_authors._id: 681aac4fd31f567552f0cc10
paper_authors.name: Yuhang Zang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-07T08:09:33.194Z
paper_authors._id: 63859cf3b2906edaf83af9f0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/mCgynEtoXdILvzyoPexii.png
paper_authors.fullname: Yuhang Zang
paper_authors.type: user
paper_authors.user: yuhangzang
paper_authors._id: 681aac4fd31f567552f0cc11
paper_authors.name: Chunyu Wang
paper_authors._id: 681aac4fd31f567552f0cc12
paper_authors.name: Qinglin Lu
paper_authors._id: 681aac4fd31f567552f0cc13
paper_authors.name: Cheng Jin
paper_authors._id: 681aac4fd31f567552f0cc14
paper_authors.name: Jiaqi Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-21T09:04:44.098Z
paper_authors._id: 64b4eec4faa3181a5eab9c46
paper_authors.avatarUrl: /avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
paper_authors.fullname: Jiaqi Wang
paper_authors.type: user
paper_authors.user: myownskyW7
paper_title: Unified Multimodal Chain-of-Thought Reward Model through Reinforcement
    Fine-Tuning
paper_summary: Recent advances in multimodal Reward Models (RMs) have shown significant
  promise in delivering reward signals to align vision models with human
  preferences. However, current RMs are generally restricted to providing direct
  responses or engaging in shallow reasoning processes with limited depth, often
  leading to inaccurate reward signals. We posit that incorporating explicit long
  chains of thought (CoT) into the reward reasoning process can significantly
  strengthen their reliability and robustness. Furthermore, we believe that once
  RMs internalize CoT reasoning, their direct response accuracy can also be
  improved through implicit reasoning capabilities. To this end, this paper
  proposes UnifiedReward-Think, the first unified multimodal CoT-based reward
  model, capable of multi-dimensional, step-by-step long-chain reasoning for both
  visual understanding and generation reward tasks. Specifically, we adopt an
  exploration-driven reinforcement fine-tuning approach to elicit and incentivize
  the model's latent complex reasoning ability: (1) We first use a small amount
  of image generation preference data to distill the reasoning process of GPT-4o,
  which is then used for the model's cold start to learn the format and structure
  of CoT reasoning. (2) Subsequently, by leveraging the model's prior knowledge
  and generalization capabilities, we prepare large-scale unified multimodal
  preference data to elicit the model's reasoning process across various vision
  tasks. During this phase, correct reasoning outputs are retained for rejection
  sampling to refine the model (3) while incorrect predicted samples are finally
  used for Group Relative Policy Optimization (GRPO) based reinforcement
  fine-tuning, enabling the model to explore diverse reasoning paths and optimize
  for correct and robust solutions. Extensive experiments across various vision
  reward tasks demonstrate the superiority of our model.
paper_ai_summary: A unified multimodal reward model that incorporates long chains of thought reasoning improves reliability and accuracy in vision reward tasks through exploration-driven reinforcement learning.
paper_ai_keywords: Reward Models multimodal Reward Models long chains of thought reasoning process UnifiedReward-Think exploration-driven reinforcement fine-tuning GPT-4o cold start large-scale unified multimodal preference data Group Relative Policy Optimization GRPO vision reward tasks