InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward
  Model
Despite the promising performance of Large Vision Language Models (LVLMs) in
visual understanding, they occasionally generate incorrect outputs. While
reward models (RMs) with reinforcement learning or test-time scaling offer the
potential for improving generation quality, a critical gap remains: publicly
available multi-modal RMs for LVLMs are scarce, and the implementation details
of proprietary models are often unclear. We bridge this gap with
InternLM-XComposer2.5-Reward (IXC-2.5-Reward), a simple yet effective
multi-modal reward model that aligns LVLMs with human preferences. To ensure
the robustness and versatility of IXC-2.5-Reward, we set up a high-quality
multi-modal preference corpus spanning text, image, and video inputs across
diverse domains, such as instruction following, general understanding,
text-rich documents, mathematical reasoning, and video understanding.
IXC-2.5-Reward achieves excellent results on the latest multi-modal reward
model benchmark and shows competitive performance on text-only reward model
benchmarks. We further demonstrate three key applications of IXC-2.5-Reward:
(1) Providing a supervisory signal for RL training. We integrate IXC-2.5-Reward
with Proximal Policy Optimization (PPO) yields IXC-2.5-Chat, which shows
consistent improvements in instruction following and multi-modal open-ended
dialogue; (2) Selecting the best response from candidate responses for
test-time scaling; and (3) Filtering outlier or noisy samples from existing
image and video instruction tuning training data. To ensure reproducibility and
facilitate further research, we have open-sourced all model weights and
training recipes at https://github.com/InternLM/InternLM-XComposer
67907fd7d37463df976acaa7
Yuhang Zang
admin_assigned
2025-01-22T15:44:47.687Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/mCgynEtoXdILvzyoPexii.png
Yuhang Zang
user
yuhangzang
67907fd7d37463df976acaa8
Xiaoyi Dong
67907fd7d37463df976acaa9
Pan Zhang
67907fd7d37463df976acaaa
Yuhang Cao
admin_assigned
2025-01-22T15:45:22.947Z
65000bef18830fabea469fdd
/avatars/b320c77dfad039d9f9c54127f610d44f.svg
Cao Yuhang
user
yhcao
67907fd7d37463df976acaab
Ziyu Liu
67907fd7d37463df976acaac
Shengyuan Ding
admin_assigned
2025-01-22T15:45:51.536Z
646cd947da8e99940b6e55cf
https://cdn-avatars.huggingface.co/v1/production/uploads/646cd947da8e99940b6e55cf/9c0P0WppFqNW9pdo8LgOS.jpeg
Shengyuan Ding
user
ChrisDing1105
67907fd7d37463df976acaad
Shenxi Wu
67907fd7d37463df976acaae
Yubo Ma
admin_assigned
2025-01-22T15:46:21.440Z
63883d1cad6d6d6e93574705
/avatars/19f17abdcfae6ff17e16d0a478c6b87f.svg
Yubo Ma
user
yubo2333
67907fd7d37463df976acaaf
Haodong Duan
admin_assigned
2025-01-22T15:46:30.500Z
63ee1379190ddd6214efd73a
https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
HAODONG DUAN
user
KennyUTC
67907fd7d37463df976acab0
Wenwei Zhang
admin_assigned
2025-01-22T15:46:37.430Z
64e8505321540e1da3226b54
/avatars/18958b8406d1ce492b54c1c839f18c54.svg
Wenwei Zhang
user
ZwwWayne
67907fd7d37463df976acab1
Kai Chen
67907fd7d37463df976acab2
Dahua Lin
admin_assigned
2025-01-22T15:46:44.848Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
67907fd7d37463df976acab3
Jiaqi Wang
claimed_verified
2025-01-22T14:04:07.315Z
64b4eec4faa3181a5eab9c46
/avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
Jiaqi Wang
user
myownskyW7
InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward
  Model
Despite the promising performance of Large Vision Language Models (LVLMs) in
visual understanding, they occasionally generate incorrect outputs. While
reward models (RMs) with reinforcement learning or test-time scaling offer the
potential for improving generation quality, a critical gap remains: publicly
available multi-modal RMs for LVLMs are scarce, and the implementation details
of proprietary models are often unclear. We bridge this gap with
InternLM-XComposer2.5-Reward (IXC-2.5-Reward), a simple yet effective
multi-modal reward model that aligns LVLMs with human preferences. To ensure
the robustness and versatility of IXC-2.5-Reward, we set up a high-quality
multi-modal preference corpus spanning text, image, and video inputs across
diverse domains, such as instruction following, general understanding,
text-rich documents, mathematical reasoning, and video understanding.
IXC-2.5-Reward achieves excellent results on the latest multi-modal reward
model benchmark and shows competitive performance on text-only reward model
benchmarks. We further demonstrate three key applications of IXC-2.5-Reward:
(1) Providing a supervisory signal for RL training. We integrate IXC-2.5-Reward
with Proximal Policy Optimization (PPO) yields IXC-2.5-Chat, which shows
consistent improvements in instruction following and multi-modal open-ended
dialogue; (2) Selecting the best response from candidate responses for
test-time scaling; and (3) Filtering outlier or noisy samples from existing
image and video instruction tuning training data. To ensure reproducibility and
facilitate further research, we have open-sourced all model weights and
training recipes at https://github.com/InternLM/InternLM-XComposer
IXC-2.5-Reward is a multi-modal reward model that enhances Large Vision Language Models by aligning them with human preferences, improving performance across various tasks and applications.
Large Vision Language Models LVLMs reward models RMs reinforcement learning test-time scaling multi-modal reward model preference corpus text image video instruction following general understanding text-rich documents mathematical reasoning video understanding multi-modal reward model benchmark text-only reward model benchmarks Proximal Policy Optimization PPO IXC-2.5-Chat multi-modal open-ended dialogue candidate responses outlier filtering noisy samples image and video instruction tuning model weights training recipes