title: RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from
    Fine-grained Correctional Human Feedback
summary: Multimodal Large Language Models (MLLMs) have recently demonstrated
  impressive capabilities in multimodal understanding, reasoning, and
  interaction. However, existing MLLMs prevalently suffer from serious
  hallucination problems, generating text that is not factually grounded in
  associated images. The problem makes existing MLLMs untrustworthy and thus
  impractical in real-world (especially high-stakes) applications. To address the
  challenge, we present RLHF-V, which enhances MLLM trustworthiness via behavior
  alignment from fine-grained correctional human feedback. Specifically, RLHF-V
  collects human preference in the form of segment-level corrections on
  hallucinations, and performs dense direct preference optimization over the
  human feedback. Comprehensive experiments on five benchmarks in both automatic
  and human evaluation show that, RLHF-V can enable substantially more
  trustworthy MLLM behaviors with promising data and computation efficiency.
  Remarkably, using 1.4k annotated data samples, RLHF-V significantly reduces the
  hallucination rate of the base MLLM by 34.8%, outperforming the concurrent
  LLaVA-RLHF trained on 10k annotated data. The final model achieves
  state-of-the-art performance in trustworthiness among open-source MLLMs, and
  shows better robustness than GPT-4V in preventing hallucinations aroused from
  over-generalization. We open-source our code, model, and data at
  https://github.com/RLHF-V/RLHF-V.
paper_authors._id: 656e8a8c2040818de9d4fb4f
paper_authors.name: Tianyu Yu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-12-05T03:00:46.507Z
paper_authors._id: 64abc4aa6cadc7aca585dddf
paper_authors.avatarUrl: /avatars/736afea979cd0021c7a37f68731524ea.svg
paper_authors.fullname: Tianyu Yu
paper_authors.type: user
paper_authors.user: Yirany
paper_authors._id: 656e8a8c2040818de9d4fb50
paper_authors.name: Yuan Yao
paper_authors._id: 656e8a8c2040818de9d4fb51
paper_authors.name: Haoye Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T09:10:38.404Z
paper_authors._id: 6566e0c493e30c8a60048eb3
paper_authors.avatarUrl: /avatars/78a021f1b205dabef3d060d1edc33bb7.svg
paper_authors.fullname: Haoye Zhang
paper_authors.type: user
paper_authors.user: HaoyeZhang
paper_authors._id: 656e8a8c2040818de9d4fb52
paper_authors.name: Taiwen He
paper_authors._id: 656e8a8c2040818de9d4fb53
paper_authors.name: Yifeng Han
paper_authors._id: 656e8a8c2040818de9d4fb54
paper_authors.name: Ganqu Cui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T09:11:10.353Z
paper_authors._id: 650eba9555dc1e841746f132
paper_authors.avatarUrl: /avatars/af6f5ee78f161d25ec0afc45d2def8eb.svg
paper_authors.fullname: Ganqu Cui
paper_authors.type: user
paper_authors.user: ganqu
paper_authors._id: 656e8a8c2040818de9d4fb55
paper_authors.name: Jinyi Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T09:11:27.504Z
paper_authors._id: 637a06580a77f602dc4ac922
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637a06580a77f602dc4ac922/JR2VHWhG1tj3TKKNzD8GD.jpeg
paper_authors.fullname: Jinyi Hu
paper_authors.type: user
paper_authors.user: JamesHujy
paper_authors._id: 656e8a8c2040818de9d4fb56
paper_authors.name: Zhiyuan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T09:11:52.320Z
paper_authors._id: 6310a3cd531cc21f9e06de6a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6310a3cd531cc21f9e06de6a/aTGMx3O41lUARK9s3dAik.jpeg
paper_authors.fullname: Zhiyuan Liu
paper_authors.type: user
paper_authors.user: acharkq
paper_authors._id: 656e8a8c2040818de9d4fb57
paper_authors.name: Hai-Tao Zheng
paper_authors._id: 656e8a8c2040818de9d4fb58
paper_authors.name: Maosong Sun
paper_authors._id: 656e8a8c2040818de9d4fb59
paper_authors.name: Tat-Seng Chua
paper_title: RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from
    Fine-grained Correctional Human Feedback
paper_summary: Multimodal Large Language Models (MLLMs) have recently demonstrated
  impressive capabilities in multimodal understanding, reasoning, and
  interaction. However, existing MLLMs prevalently suffer from serious
  hallucination problems, generating text that is not factually grounded in
  associated images. The problem makes existing MLLMs untrustworthy and thus
  impractical in real-world (especially high-stakes) applications. To address the
  challenge, we present RLHF-V, which enhances MLLM trustworthiness via behavior
  alignment from fine-grained correctional human feedback. Specifically, RLHF-V
  collects human preference in the form of segment-level corrections on
  hallucinations, and performs dense direct preference optimization over the
  human feedback. Comprehensive experiments on five benchmarks in both automatic
  and human evaluation show that, RLHF-V can enable substantially more
  trustworthy MLLM behaviors with promising data and computation efficiency.
  Remarkably, using 1.4k annotated data samples, RLHF-V significantly reduces the
  hallucination rate of the base MLLM by 34.8%, outperforming the concurrent
  LLaVA-RLHF trained on 10k annotated data. The final model achieves
  state-of-the-art performance in trustworthiness among open-source MLLMs, and
  shows better robustness than GPT-4V in preventing hallucinations aroused from
  over-generalization. We open-source our code, model, and data at
  https://github.com/RLHF-V/RLHF-V.
paper_ai_keywords: Multimodal Large Language Models (MLLMs) multimodal evaluation multimodal reasoning multimodal interaction hallucination problems behavior alignment fine-grained correctional human feedback segment-level corrections dense direct preference optimization human evaluation automatic evaluation hallucination rate robustness over-generalization LLaVA-RLHF GPT-4V