title: InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward
    Model
summary: Despite the promising performance of Large Vision Language Models (LVLMs) in
  visual understanding, they occasionally generate incorrect outputs. While
  reward models (RMs) with reinforcement learning or test-time scaling offer the
  potential for improving generation quality, a critical gap remains: publicly
  available multi-modal RMs for LVLMs are scarce, and the implementation details
  of proprietary models are often unclear. We bridge this gap with
  InternLM-XComposer2.5-Reward (IXC-2.5-Reward), a simple yet effective
  multi-modal reward model that aligns LVLMs with human preferences. To ensure
  the robustness and versatility of IXC-2.5-Reward, we set up a high-quality
  multi-modal preference corpus spanning text, image, and video inputs across
  diverse domains, such as instruction following, general understanding,
  text-rich documents, mathematical reasoning, and video understanding.
  IXC-2.5-Reward achieves excellent results on the latest multi-modal reward
  model benchmark and shows competitive performance on text-only reward model
  benchmarks. We further demonstrate three key applications of IXC-2.5-Reward:
  (1) Providing a supervisory signal for RL training. We integrate IXC-2.5-Reward
  with Proximal Policy Optimization (PPO) yields IXC-2.5-Chat, which shows
  consistent improvements in instruction following and multi-modal open-ended
  dialogue; (2) Selecting the best response from candidate responses for
  test-time scaling; and (3) Filtering outlier or noisy samples from existing
  image and video instruction tuning training data. To ensure reproducibility and
  facilitate further research, we have open-sourced all model weights and
  training recipes at https://github.com/InternLM/InternLM-XComposer
paper_authors._id: 67907fd7d37463df976acaa7
paper_authors.name: Yuhang Zang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:44:47.687Z
paper_authors._id: 63859cf3b2906edaf83af9f0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/iUQm5FAomzqYi6fkqIn9F.jpeg
paper_authors.fullname: Yuhang Zang
paper_authors.type: user
paper_authors.user: yuhangzang
paper_authors._id: 67907fd7d37463df976acaa8
paper_authors.name: Xiaoyi Dong
paper_authors._id: 67907fd7d37463df976acaa9
paper_authors.name: Pan Zhang
paper_authors._id: 67907fd7d37463df976acaaa
paper_authors.name: Yuhang Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:45:22.947Z
paper_authors._id: 65000bef18830fabea469fdd
paper_authors.avatarUrl: /avatars/b320c77dfad039d9f9c54127f610d44f.svg
paper_authors.fullname: Cao Yuhang
paper_authors.type: user
paper_authors.user: yhcao
paper_authors._id: 67907fd7d37463df976acaab
paper_authors.name: Ziyu Liu
paper_authors._id: 67907fd7d37463df976acaac
paper_authors.name: Shengyuan Ding
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:45:51.536Z
paper_authors._id: 646cd947da8e99940b6e55cf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646cd947da8e99940b6e55cf/9c0P0WppFqNW9pdo8LgOS.jpeg
paper_authors.fullname: Shengyuan Ding
paper_authors.type: user
paper_authors.user: ChrisDing1105
paper_authors._id: 67907fd7d37463df976acaad
paper_authors.name: Shenxi Wu
paper_authors._id: 67907fd7d37463df976acaae
paper_authors.name: Yubo Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:46:21.440Z
paper_authors._id: 63883d1cad6d6d6e93574705
paper_authors.avatarUrl: /avatars/19f17abdcfae6ff17e16d0a478c6b87f.svg
paper_authors.fullname: Yubo Ma
paper_authors.type: user
paper_authors.user: yubo2333
paper_authors._id: 67907fd7d37463df976acaaf
paper_authors.name: Haodong Duan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:46:30.500Z
paper_authors._id: 63ee1379190ddd6214efd73a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
paper_authors.fullname: HAODONG DUAN
paper_authors.type: user
paper_authors.user: KennyUTC
paper_authors._id: 67907fd7d37463df976acab0
paper_authors.name: Wenwei Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:46:37.430Z
paper_authors._id: 64e8505321540e1da3226b54
paper_authors.avatarUrl: /avatars/18958b8406d1ce492b54c1c839f18c54.svg
paper_authors.fullname: Wenwei Zhang
paper_authors.type: user
paper_authors.user: ZwwWayne
paper_authors._id: 67907fd7d37463df976acab1
paper_authors.name: Kai Chen
paper_authors._id: 67907fd7d37463df976acab2
paper_authors.name: Dahua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:46:44.848Z
paper_authors._id: 636317ed80c1a705a6eff396
paper_authors.avatarUrl: /avatars/3db090e101b916d9256d0d3e043db71d.svg
paper_authors.fullname: Dahua Lin
paper_authors.type: user
paper_authors.user: lindahua
paper_authors._id: 67907fd7d37463df976acab3
paper_authors.name: Jiaqi Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-22T14:04:07.315Z
paper_authors._id: 64b4eec4faa3181a5eab9c46
paper_authors.avatarUrl: /avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
paper_authors.fullname: Jiaqi Wang
paper_authors.type: user
paper_authors.user: myownskyW7
paper_title: InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward
    Model
paper_summary: Despite the promising performance of Large Vision Language Models (LVLMs) in
  visual understanding, they occasionally generate incorrect outputs. While
  reward models (RMs) with reinforcement learning or test-time scaling offer the
  potential for improving generation quality, a critical gap remains: publicly
  available multi-modal RMs for LVLMs are scarce, and the implementation details
  of proprietary models are often unclear. We bridge this gap with
  InternLM-XComposer2.5-Reward (IXC-2.5-Reward), a simple yet effective
  multi-modal reward model that aligns LVLMs with human preferences. To ensure
  the robustness and versatility of IXC-2.5-Reward, we set up a high-quality
  multi-modal preference corpus spanning text, image, and video inputs across
  diverse domains, such as instruction following, general understanding,
  text-rich documents, mathematical reasoning, and video understanding.
  IXC-2.5-Reward achieves excellent results on the latest multi-modal reward
  model benchmark and shows competitive performance on text-only reward model
  benchmarks. We further demonstrate three key applications of IXC-2.5-Reward:
  (1) Providing a supervisory signal for RL training. We integrate IXC-2.5-Reward
  with Proximal Policy Optimization (PPO) yields IXC-2.5-Chat, which shows
  consistent improvements in instruction following and multi-modal open-ended
  dialogue; (2) Selecting the best response from candidate responses for
  test-time scaling; and (3) Filtering outlier or noisy samples from existing
  image and video instruction tuning training data. To ensure reproducibility and
  facilitate further research, we have open-sourced all model weights and
  training recipes at https://github.com/InternLM/InternLM-XComposer
paper_ai_keywords: Large Vision Language Models (LVLMs) reward models (RMs) reinforcement learning test-time scaling multi-modal reward model human preferences multi-modal preference corpus Proximal Policy Optimization (PPO) instruction following multi-modal open-ended dialogue candidate responses outlier or noisy samples image and video instruction tuning