Aligning Large Multimodal Models with Factually Augmented RLHF
Large Multimodal Models (LMM) are built across modalities and the
misalignment between two modalities can result in "hallucination", generating
textual outputs that are not grounded by the multimodal information in context.
To address the multimodal misalignment issue, we adapt the Reinforcement
Learning from Human Feedback (RLHF) from the text domain to the task of
vision-language alignment, where human annotators are asked to compare two
responses and pinpoint the more hallucinated one, and the vision-language model
is trained to maximize the simulated human rewards. We propose a new alignment
algorithm called Factually Augmented RLHF that augments the reward model with
additional factual information such as image captions and ground-truth
multi-choice options, which alleviates the reward hacking phenomenon in RLHF
and further improves the performance. We also enhance the GPT-4-generated
training data (for vision instruction tuning) with previously available
human-written image-text pairs to improve the general capabilities of our
model. To evaluate the proposed approach in real-world scenarios, we develop a
new evaluation benchmark MMHAL-BENCH with a special focus on penalizing
hallucinations. As the first LMM trained with RLHF, our approach achieves
remarkable improvement on the LLaVA-Bench dataset with the 94% performance
level of the text-only GPT-4 (while previous best methods can only achieve the
87% level), and an improvement by 60% on MMHAL-BENCH over other baselines. We
opensource our code, model, data at https://llava-rlhf.github.io.
65137ef5b7994cff61e638ed
Zhiqing Sun
admin_assigned
2023-09-27T10:15:45.367Z
5ee24411636bdb3834e2da24
/avatars/0e12a0b19ada4138ddf808501ba68423.svg
Zhiqing Sun
user
zhiqings
65137ef5b7994cff61e638ee
Sheng Shen
claimed_verified
2023-09-27T07:26:56.830Z
614b532936c340687d05a585
/avatars/23914c474bb737bbdf2f49acacad6ef7.svg
Sheng Shen
user
shengs
65137ef5b7994cff61e638ef
Shengcao Cao
admin_assigned
2023-09-27T10:15:52.900Z
64a84e1dcc73827ce666e2f8
/avatars/45ab4a97d3300643b57bf263aefa4cd4.svg
Shengcao Cao
user
Shengcao1006
65137ef5b7994cff61e638f0
Haotian Liu
admin_assigned
2023-09-27T10:16:20.668Z
63898b61ec1f539adc0f4da2
https://cdn-avatars.huggingface.co/v1/production/uploads/1674248167280-63898b61ec1f539adc0f4da2.jpeg
Haotian Liu
user
liuhaotian
65137ef5b7994cff61e638f1
Chunyuan Li
admin_assigned
2023-09-27T10:16:28.536Z
62aba526cae4462c0c6caa0f
/avatars/430560ec2c2547f819225769ab432f30.svg
Chunyuan Li
user
Chunyuan24
65137ef5b7994cff61e638f2
Yikang Shen
admin_assigned
2023-09-27T10:17:02.945Z
6463995939359568c63d740c
/avatars/11caacefc68c1259dbc8764cc4340481.svg
Yikang Shen
user
YikangS
65137ef5b7994cff61e638f3
Chuang Gan
65137ef5b7994cff61e638f4
Liang-Yan Gui
65137ef5b7994cff61e638f5
Yu-Xiong Wang
65137ef5b7994cff61e638f6
Yiming Yang
65137ef5b7994cff61e638f7
Kurt Keutzer
admin_assigned
2023-09-27T10:18:49.901Z
6251bf4b183aa4266924ad91
https://cdn-avatars.huggingface.co/v1/production/uploads/1678041834400-6251bf4b183aa4266924ad91.jpeg
Kurt Keutzer
user
kurtkeutzer
65137ef5b7994cff61e638f8
Trevor Darrell
admin_assigned
2023-09-27T10:18:57.133Z
64cbdf02f103036e23d1c7f3
/avatars/496069463900dea20929b57381182d39.svg
Trevor Darrell
user
trevordarrell
Aligning Large Multimodal Models with Factually Augmented RLHF
Large Multimodal Models (LMM) are built across modalities and the
misalignment between two modalities can result in "hallucination", generating
textual outputs that are not grounded by the multimodal information in context.
To address the multimodal misalignment issue, we adapt the Reinforcement
Learning from Human Feedback (RLHF) from the text domain to the task of
vision-language alignment, where human annotators are asked to compare two
responses and pinpoint the more hallucinated one, and the vision-language model
is trained to maximize the simulated human rewards. We propose a new alignment
algorithm called Factually Augmented RLHF that augments the reward model with
additional factual information such as image captions and ground-truth
multi-choice options, which alleviates the reward hacking phenomenon in RLHF
and further improves the performance. We also enhance the GPT-4-generated
training data (for vision instruction tuning) with previously available
human-written image-text pairs to improve the general capabilities of our
model. To evaluate the proposed approach in real-world scenarios, we develop a
new evaluation benchmark MMHAL-BENCH with a special focus on penalizing
hallucinations. As the first LMM trained with RLHF, our approach achieves
remarkable improvement on the LLaVA-Bench dataset with the 94% performance
level of the text-only GPT-4 (while previous best methods can only achieve the
87% level), and an improvement by 60% on MMHAL-BENCH over other baselines. We
opensource our code, model, data at https://llava-rlhf.github.io.
The paper presents Factually Augmented RLHF to address multimodal misalignment in large multimodal models, significantly improving hallucination reduction and overall performance on vision-language tasks compared to existing methods.
Reinforcement Learning from Human Feedback (RLHF) vision-language alignment hallucination reward model image captions ground-truth multi-choice options reward hacking vision instruction tuning image-text pairs evaluation benchmark MMHAL-BENCH LLaVA-Bench GPT-4