title: Aligning Large Multimodal Models with Factually Augmented RLHF
summary: Large Multimodal Models (LMM) are built across modalities and the
  misalignment between two modalities can result in "hallucination", generating
  textual outputs that are not grounded by the multimodal information in context.
  To address the multimodal misalignment issue, we adapt the Reinforcement
  Learning from Human Feedback (RLHF) from the text domain to the task of
  vision-language alignment, where human annotators are asked to compare two
  responses and pinpoint the more hallucinated one, and the vision-language model
  is trained to maximize the simulated human rewards. We propose a new alignment
  algorithm called Factually Augmented RLHF that augments the reward model with
  additional factual information such as image captions and ground-truth
  multi-choice options, which alleviates the reward hacking phenomenon in RLHF
  and further improves the performance. We also enhance the GPT-4-generated
  training data (for vision instruction tuning) with previously available
  human-written image-text pairs to improve the general capabilities of our
  model. To evaluate the proposed approach in real-world scenarios, we develop a
  new evaluation benchmark MMHAL-BENCH with a special focus on penalizing
  hallucinations. As the first LMM trained with RLHF, our approach achieves
  remarkable improvement on the LLaVA-Bench dataset with the 94% performance
  level of the text-only GPT-4 (while previous best methods can only achieve the
  87% level), and an improvement by 60% on MMHAL-BENCH over other baselines. We
  opensource our code, model, data at https://llava-rlhf.github.io.
paper_authors._id: 65137ef5b7994cff61e638ed
paper_authors.name: Zhiqing Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-27T10:15:45.367Z
paper_authors._id: 5ee24411636bdb3834e2da24
paper_authors.avatarUrl: /avatars/0e12a0b19ada4138ddf808501ba68423.svg
paper_authors.fullname: Zhiqing Sun
paper_authors.type: user
paper_authors.user: zhiqings
paper_authors._id: 65137ef5b7994cff61e638ee
paper_authors.name: Sheng Shen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-27T07:26:56.830Z
paper_authors._id: 614b532936c340687d05a585
paper_authors.avatarUrl: /avatars/23914c474bb737bbdf2f49acacad6ef7.svg
paper_authors.fullname: Sheng Shen
paper_authors.type: user
paper_authors.user: shengs
paper_authors._id: 65137ef5b7994cff61e638ef
paper_authors.name: Shengcao Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-27T10:15:52.900Z
paper_authors._id: 64a84e1dcc73827ce666e2f8
paper_authors.avatarUrl: /avatars/45ab4a97d3300643b57bf263aefa4cd4.svg
paper_authors.fullname: Shengcao Cao
paper_authors.type: user
paper_authors.user: Shengcao1006
paper_authors._id: 65137ef5b7994cff61e638f0
paper_authors.name: Haotian Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-27T10:16:20.668Z
paper_authors._id: 63898b61ec1f539adc0f4da2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1674248167280-63898b61ec1f539adc0f4da2.jpeg
paper_authors.fullname: Haotian Liu
paper_authors.type: user
paper_authors.user: liuhaotian
paper_authors._id: 65137ef5b7994cff61e638f1
paper_authors.name: Chunyuan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-27T10:16:28.536Z
paper_authors._id: 62aba526cae4462c0c6caa0f
paper_authors.avatarUrl: /avatars/430560ec2c2547f819225769ab432f30.svg
paper_authors.fullname: Chunyuan Li
paper_authors.type: user
paper_authors.user: Chunyuan24
paper_authors._id: 65137ef5b7994cff61e638f2
paper_authors.name: Yikang Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-27T10:17:02.945Z
paper_authors._id: 6463995939359568c63d740c
paper_authors.avatarUrl: /avatars/11caacefc68c1259dbc8764cc4340481.svg
paper_authors.fullname: Yikang Shen
paper_authors.type: user
paper_authors.user: YikangS
paper_authors._id: 65137ef5b7994cff61e638f3
paper_authors.name: Chuang Gan
paper_authors._id: 65137ef5b7994cff61e638f4
paper_authors.name: Liang-Yan Gui
paper_authors._id: 65137ef5b7994cff61e638f5
paper_authors.name: Yu-Xiong Wang
paper_authors._id: 65137ef5b7994cff61e638f6
paper_authors.name: Yiming Yang
paper_authors._id: 65137ef5b7994cff61e638f7
paper_authors.name: Kurt Keutzer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-27T10:18:49.901Z
paper_authors._id: 6251bf4b183aa4266924ad91
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678041834400-6251bf4b183aa4266924ad91.jpeg
paper_authors.fullname: Kurt Keutzer
paper_authors.type: user
paper_authors.user: kurtkeutzer
paper_authors._id: 65137ef5b7994cff61e638f8
paper_authors.name: Trevor Darrell
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-27T10:18:57.133Z
paper_authors._id: 64cbdf02f103036e23d1c7f3
paper_authors.avatarUrl: /avatars/496069463900dea20929b57381182d39.svg
paper_authors.fullname: Trevor Darrell
paper_authors.type: user
paper_authors.user: trevordarrell
paper_title: Aligning Large Multimodal Models with Factually Augmented RLHF
paper_summary: Large Multimodal Models (LMM) are built across modalities and the
  misalignment between two modalities can result in "hallucination", generating
  textual outputs that are not grounded by the multimodal information in context.
  To address the multimodal misalignment issue, we adapt the Reinforcement
  Learning from Human Feedback (RLHF) from the text domain to the task of
  vision-language alignment, where human annotators are asked to compare two
  responses and pinpoint the more hallucinated one, and the vision-language model
  is trained to maximize the simulated human rewards. We propose a new alignment
  algorithm called Factually Augmented RLHF that augments the reward model with
  additional factual information such as image captions and ground-truth
  multi-choice options, which alleviates the reward hacking phenomenon in RLHF
  and further improves the performance. We also enhance the GPT-4-generated
  training data (for vision instruction tuning) with previously available
  human-written image-text pairs to improve the general capabilities of our
  model. To evaluate the proposed approach in real-world scenarios, we develop a
  new evaluation benchmark MMHAL-BENCH with a special focus on penalizing
  hallucinations. As the first LMM trained with RLHF, our approach achieves
  remarkable improvement on the LLaVA-Bench dataset with the 94% performance
  level of the text-only GPT-4 (while previous best methods can only achieve the
  87% level), and an improvement by 60% on MMHAL-BENCH over other baselines. We
  opensource our code, model, data at https://llava-rlhf.github.io.
paper_ai_summary: The paper presents Factually Augmented RLHF to address multimodal misalignment in large multimodal models, significantly improving hallucination reduction and overall performance on vision-language tasks compared to existing methods.
paper_ai_keywords: Reinforcement Learning from Human Feedback (RLHF) vision-language alignment hallucination reward model image captions ground-truth multi-choice options reward hacking vision instruction tuning image-text pairs evaluation benchmark MMHAL-BENCH LLaVA-Bench GPT-4