MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
Despite notable advancements in Multimodal Large Language Models (MLLMs),
most state-of-the-art models have not undergone thorough alignment with human
preferences. This gap exists because current alignment research has primarily
achieved progress in specific areas (e.g., hallucination reduction), while the
broader question of whether aligning models with human preferences can
systematically enhance MLLM capability remains largely unexplored. To this end,
we introduce MM-RLHF, a dataset containing 120k fine-grained,
human-annotated preference comparison pairs. This dataset represents a
substantial advancement over existing resources, offering superior size,
diversity, annotation granularity, and quality. Leveraging this dataset, we
propose several key innovations to improve both the quality of reward models
and the efficiency of alignment algorithms. Notably, we introduce a
Critique-Based Reward Model, which generates critiques of model outputs before
assigning scores, offering enhanced interpretability and more informative
feedback compared to traditional scalar reward mechanisms. Additionally, we
propose Dynamic Reward Scaling, a method that adjusts the loss weight of each
sample according to the reward signal, thereby optimizing the use of
high-quality comparison pairs. Our approach is rigorously evaluated across
10 distinct dimensions and 27 benchmarks, with results
demonstrating significant and consistent improvements in model performance.
Specifically, fine-tuning LLaVA-ov-7B with MM-RLHF and our alignment algorithm
leads to a 19.5% increase in conversational abilities and a
60% improvement in safety.
  We have open-sourced the preference dataset, reward model, training and
evaluation code, as well as reward modeling and safety benchmarks. For more
details, please visit our project page: https://mm-rlhf.github.io.
67b2ab548191c180b9c4eb83
Yi-Fan Zhang
claimed_verified
2025-03-20T10:54:09.856Z
623d8ca4c29adf5ef6175615
https://cdn-avatars.huggingface.co/v1/production/uploads/623d8ca4c29adf5ef6175615/q7lHao7UPwU1u7YLSP56m.jpeg
Yi-Fan Zhang
user
yifanzhang114
67b2ab548191c180b9c4eb84
Tao Yu
67b2ab548191c180b9c4eb85
Haochen Tian
claimed_verified
2025-02-19T14:37:54.641Z
6448baa3e780dbfc89058bc3
/avatars/44c243c8eb4714cdcdaa7cd04e5a9716.svg
Micheal Tian
user
StarBurger
67b2ab548191c180b9c4eb86
Chaoyou Fu
67b2ab548191c180b9c4eb87
Peiyan Li
claimed_verified
2025-06-16T07:18:05.690Z
6337e04b171879571956212f
/avatars/aa69142bf92e4c50b192463490251ed9.svg
Li Peiyan
user
LPY
67b2ab548191c180b9c4eb88
Jianshu Zeng
67b2ab548191c180b9c4eb89
Wulin Xie
67b2ab548191c180b9c4eb8a
Yang Shi
claimed_verified
2025-02-18T09:32:59.588Z
673c7319d11b1c2e246ead9c
https://cdn-avatars.huggingface.co/v1/production/uploads/673c7319d11b1c2e246ead9c/IjFIO--N7Hm_BOEafhEQv.jpeg
Yang Shi
user
DogNeverSleep
67b2ab548191c180b9c4eb8b
Huanyu Zhang
67b2ab548191c180b9c4eb8c
Junkang Wu
claimed_verified
2025-03-11T08:23:36.787Z
664764e5d834283e7ff96d37
https://cdn-avatars.huggingface.co/v1/production/uploads/664764e5d834283e7ff96d37/JBi3dvAUKqqGf_DjVORK7.jpeg
Junkang Wu
user
junkang0909
67b2ab548191c180b9c4eb8d
Xue Wang
67b2ab548191c180b9c4eb8e
Yibo Hu
67b2ab548191c180b9c4eb8f
Bin Wen
67b2ab548191c180b9c4eb90
Fan Yang
67b2ab548191c180b9c4eb91
Zhang Zhang
67b2ab548191c180b9c4eb92
Tingting Gao
67b2ab548191c180b9c4eb93
Di Zhang
67b2ab548191c180b9c4eb94
Liang Wang
67b2ab548191c180b9c4eb95
Rong Jin
67b2ab548191c180b9c4eb96
Tieniu Tan
MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
Despite notable advancements in Multimodal Large Language Models (MLLMs),
most state-of-the-art models have not undergone thorough alignment with human
preferences. This gap exists because current alignment research has primarily
achieved progress in specific areas (e.g., hallucination reduction), while the
broader question of whether aligning models with human preferences can
systematically enhance MLLM capability remains largely unexplored. To this end,
we introduce MM-RLHF, a dataset containing 120k fine-grained,
human-annotated preference comparison pairs. This dataset represents a
substantial advancement over existing resources, offering superior size,
diversity, annotation granularity, and quality. Leveraging this dataset, we
propose several key innovations to improve both the quality of reward models
and the efficiency of alignment algorithms. Notably, we introduce a
Critique-Based Reward Model, which generates critiques of model outputs before
assigning scores, offering enhanced interpretability and more informative
feedback compared to traditional scalar reward mechanisms. Additionally, we
propose Dynamic Reward Scaling, a method that adjusts the loss weight of each
sample according to the reward signal, thereby optimizing the use of
high-quality comparison pairs. Our approach is rigorously evaluated across
10 distinct dimensions and 27 benchmarks, with results
demonstrating significant and consistent improvements in model performance.
Specifically, fine-tuning LLaVA-ov-7B with MM-RLHF and our alignment algorithm
leads to a 19.5% increase in conversational abilities and a
60% improvement in safety.
  We have open-sourced the preference dataset, reward model, training and
evaluation code, as well as reward modeling and safety benchmarks. For more
details, please visit our project page: https://mm-rlhf.github.io.
MM-RLHF improves Multimodal Large Language Models' alignment with human preferences through a large preference dataset and innovative reward model techniques.
Multimodal Large Language Models MM-RLHF preference comparison pairs reward models Critique-Based Reward Model Dynamic Reward Scaling conversational abilities safety