title: MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
summary: Despite notable advancements in Multimodal Large Language Models (MLLMs),
  most state-of-the-art models have not undergone thorough alignment with human
  preferences. This gap exists because current alignment research has primarily
  achieved progress in specific areas (e.g., hallucination reduction), while the
  broader question of whether aligning models with human preferences can
  systematically enhance MLLM capability remains largely unexplored. To this end,
  we introduce MM-RLHF, a dataset containing 120k fine-grained,
  human-annotated preference comparison pairs. This dataset represents a
  substantial advancement over existing resources, offering superior size,
  diversity, annotation granularity, and quality. Leveraging this dataset, we
  propose several key innovations to improve both the quality of reward models
  and the efficiency of alignment algorithms. Notably, we introduce a
  Critique-Based Reward Model, which generates critiques of model outputs before
  assigning scores, offering enhanced interpretability and more informative
  feedback compared to traditional scalar reward mechanisms. Additionally, we
  propose Dynamic Reward Scaling, a method that adjusts the loss weight of each
  sample according to the reward signal, thereby optimizing the use of
  high-quality comparison pairs. Our approach is rigorously evaluated across
  10 distinct dimensions and 27 benchmarks, with results
  demonstrating significant and consistent improvements in model performance.
  Specifically, fine-tuning LLaVA-ov-7B with MM-RLHF and our alignment algorithm
  leads to a 19.5% increase in conversational abilities and a
  60% improvement in safety.
    We have open-sourced the preference dataset, reward model, training and
  evaluation code, as well as reward modeling and safety benchmarks. For more
  details, please visit our project page: https://mm-rlhf.github.io.
paper_authors._id: 67b2ab548191c180b9c4eb83
paper_authors.name: Yi-Fan Zhang
paper_authors._id: 67b2ab548191c180b9c4eb84
paper_authors.name: Tao Yu
paper_authors._id: 67b2ab548191c180b9c4eb85
paper_authors.name: Haochen Tian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-19T14:37:54.641Z
paper_authors._id: 6448baa3e780dbfc89058bc3
paper_authors.avatarUrl: /avatars/44c243c8eb4714cdcdaa7cd04e5a9716.svg
paper_authors.fullname: Micheal Tian
paper_authors.type: user
paper_authors.user: StarBurger
paper_authors._id: 67b2ab548191c180b9c4eb86
paper_authors.name: Chaoyou Fu
paper_authors._id: 67b2ab548191c180b9c4eb87
paper_authors.name: Peiyan Li
paper_authors._id: 67b2ab548191c180b9c4eb88
paper_authors.name: Jianshu Zeng
paper_authors._id: 67b2ab548191c180b9c4eb89
paper_authors.name: Wulin Xie
paper_authors._id: 67b2ab548191c180b9c4eb8a
paper_authors.name: Yang Shi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-18T09:32:59.588Z
paper_authors._id: 673c7319d11b1c2e246ead9c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/673c7319d11b1c2e246ead9c/IjFIO--N7Hm_BOEafhEQv.jpeg
paper_authors.fullname: Yang Shi
paper_authors.type: user
paper_authors.user: DogNeverSleep
paper_authors._id: 67b2ab548191c180b9c4eb8b
paper_authors.name: Huanyu Zhang
paper_authors._id: 67b2ab548191c180b9c4eb8c
paper_authors.name: Junkang Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-11T08:23:36.787Z
paper_authors._id: 664764e5d834283e7ff96d37
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/664764e5d834283e7ff96d37/JBi3dvAUKqqGf_DjVORK7.jpeg
paper_authors.fullname: Junkang Wu
paper_authors.type: user
paper_authors.user: junkang0909
paper_authors._id: 67b2ab548191c180b9c4eb8d
paper_authors.name: Xue Wang
paper_authors._id: 67b2ab548191c180b9c4eb8e
paper_authors.name: Yibo Hu
paper_authors._id: 67b2ab548191c180b9c4eb8f
paper_authors.name: Bin Wen
paper_authors._id: 67b2ab548191c180b9c4eb90
paper_authors.name: Fan Yang
paper_authors._id: 67b2ab548191c180b9c4eb91
paper_authors.name: Zhang Zhang
paper_authors._id: 67b2ab548191c180b9c4eb92
paper_authors.name: Tingting Gao
paper_authors._id: 67b2ab548191c180b9c4eb93
paper_authors.name: Di Zhang
paper_authors._id: 67b2ab548191c180b9c4eb94
paper_authors.name: Liang Wang
paper_authors._id: 67b2ab548191c180b9c4eb95
paper_authors.name: Rong Jin
paper_authors._id: 67b2ab548191c180b9c4eb96
paper_authors.name: Tieniu Tan
paper_title: MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
paper_summary: Despite notable advancements in Multimodal Large Language Models (MLLMs),
  most state-of-the-art models have not undergone thorough alignment with human
  preferences. This gap exists because current alignment research has primarily
  achieved progress in specific areas (e.g., hallucination reduction), while the
  broader question of whether aligning models with human preferences can
  systematically enhance MLLM capability remains largely unexplored. To this end,
  we introduce MM-RLHF, a dataset containing 120k fine-grained,
  human-annotated preference comparison pairs. This dataset represents a
  substantial advancement over existing resources, offering superior size,
  diversity, annotation granularity, and quality. Leveraging this dataset, we
  propose several key innovations to improve both the quality of reward models
  and the efficiency of alignment algorithms. Notably, we introduce a
  Critique-Based Reward Model, which generates critiques of model outputs before
  assigning scores, offering enhanced interpretability and more informative
  feedback compared to traditional scalar reward mechanisms. Additionally, we
  propose Dynamic Reward Scaling, a method that adjusts the loss weight of each
  sample according to the reward signal, thereby optimizing the use of
  high-quality comparison pairs. Our approach is rigorously evaluated across
  10 distinct dimensions and 27 benchmarks, with results
  demonstrating significant and consistent improvements in model performance.
  Specifically, fine-tuning LLaVA-ov-7B with MM-RLHF and our alignment algorithm
  leads to a 19.5% increase in conversational abilities and a
  60% improvement in safety.
    We have open-sourced the preference dataset, reward model, training and
  evaluation code, as well as reward modeling and safety benchmarks. For more
  details, please visit our project page: https://mm-rlhf.github.io.
paper_ai_keywords: Multimodal Large Language Models (MLLMs) MM-RLHF Critique-Based Reward Model Dynamic Reward Scaling Reward models Alignment algorithms Conversational abilities Safety