title: Unified Reward Model for Multimodal Understanding and Generation
summary: Recent advances in human preference alignment have significantly enhanced
  multimodal generation and understanding. A key approach is training reward
  models to guide preference optimization. However, existing models are often
  task-specific, limiting their adaptability across diverse visual applications.
  We also argue that jointly learning to assess multiple tasks may foster a
  synergistic effect, where improved image understanding enhances image
  generation assessment, and refined image evaluation benefits video assessment
  through better frame analysis. To this end, this paper proposes UnifiedReward,
  the first unified reward model for multimodal understanding and generation
  assessment, enabling both pairwise ranking and pointwise scoring, which can be
  employed for vision model preference alignment. Specifically, (1) we first
  develop UnifiedReward on our constructed large-scale human preference dataset,
  including both image and video generation/understanding tasks. (2) Then, it is
  utilized to automatically construct high-quality preference pair data based on
  the vision models, fine-gradually filtering their outputs through pair ranking
  and point sifting. (3) Finally, these data are used for their preference
  alignment through Direct Preference Optimization (DPO). Experimental results
  demonstrate that joint learning to assess diverse visual tasks can lead to
  substantial mutual benefits and we apply our pipeline to both image and video
  understanding/generation tasks, significantly improving the performance in each
  domain.
paper_authors._id: 67ce37239f9aaaae837f3894
paper_authors.name: Yibin Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-10T09:37:51.835Z
paper_authors._id: 654c6845bac6e6e49895a5b5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/QwUOUujUAyiV4tAhB_moO.png
paper_authors.fullname: SII-Yibin Wang
paper_authors.type: user
paper_authors.user: CodeGoat24
paper_authors._id: 67ce37239f9aaaae837f3895
paper_authors.name: Yuhang Zang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-10T08:01:24.660Z
paper_authors._id: 63859cf3b2906edaf83af9f0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/mCgynEtoXdILvzyoPexii.png
paper_authors.fullname: Yuhang Zang
paper_authors.type: user
paper_authors.user: yuhangzang
paper_authors._id: 67ce37239f9aaaae837f3896
paper_authors.name: Hao Li
paper_authors._id: 67ce37239f9aaaae837f3897
paper_authors.name: Cheng Jin
paper_authors._id: 67ce37239f9aaaae837f3898
paper_authors.name: Jiaqi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-10T09:38:17.938Z
paper_authors._id: 64638c4d51fa6e63060521b5
paper_authors.avatarUrl: /avatars/c863ace5b1dc788a341bcf4ddbdfaec1.svg
paper_authors.fullname: JIaqi
paper_authors.type: user
paper_authors.user: Jiaqiwang
paper_title: Unified Reward Model for Multimodal Understanding and Generation
paper_summary: Recent advances in human preference alignment have significantly enhanced
  multimodal generation and understanding. A key approach is training reward
  models to guide preference optimization. However, existing models are often
  task-specific, limiting their adaptability across diverse visual applications.
  We also argue that jointly learning to assess multiple tasks may foster a
  synergistic effect, where improved image understanding enhances image
  generation assessment, and refined image evaluation benefits video assessment
  through better frame analysis. To this end, this paper proposes UnifiedReward,
  the first unified reward model for multimodal understanding and generation
  assessment, enabling both pairwise ranking and pointwise scoring, which can be
  employed for vision model preference alignment. Specifically, (1) we first
  develop UnifiedReward on our constructed large-scale human preference dataset,
  including both image and video generation/understanding tasks. (2) Then, it is
  utilized to automatically construct high-quality preference pair data based on
  the vision models, fine-gradually filtering their outputs through pair ranking
  and point sifting. (3) Finally, these data are used for their preference
  alignment through Direct Preference Optimization (DPO). Experimental results
  demonstrate that joint learning to assess diverse visual tasks can lead to
  substantial mutual benefits and we apply our pipeline to both image and video
  understanding/generation tasks, significantly improving the performance in each
  domain.
paper_ai_summary: A unified reward model for multimodal understanding and generation improves the assessment and preference alignment in image and video tasks through joint learning and direct preference optimization.
paper_ai_keywords: UnifiedReward reward models preference optimization pairwise ranking pointwise scoring vision model preference alignment Direct Preference Optimization (DPO)