Understanding Alignment in Multimodal LLMs: A Comprehensive Study
Preference alignment has become a crucial component in enhancing the
performance of Large Language Models (LLMs), yet its impact in Multimodal Large
Language Models (MLLMs) remains comparatively underexplored. Similar to
language models, MLLMs for image understanding tasks encounter challenges like
hallucination. In MLLMs, hallucination can occur not only by stating incorrect
facts but also by producing responses that are inconsistent with the image
content. A primary objective of alignment for MLLMs is to encourage these
models to align responses more closely with image information. Recently,
multiple works have introduced preference datasets for MLLMs and examined
different alignment methods, including Direct Preference Optimization (DPO) and
Proximal Policy Optimization (PPO). However, due to variations in datasets,
base model types, and alignment methods, it remains unclear which specific
elements contribute most significantly to the reported improvements in these
works. In this paper, we independently analyze each aspect of preference
alignment in MLLMs. We start by categorizing the alignment algorithms into two
groups, offline (such as DPO), and online (such as online-DPO), and show that
combining offline and online methods can improve the performance of the model
in certain scenarios. We review a variety of published multimodal preference
datasets and discuss how the details of their construction impact model
performance. Based on these insights, we introduce a novel way of creating
multimodal preference data called Bias-Driven Hallucination Sampling (BDHS)
that needs neither additional annotation nor external models, and show that it
can achieve competitive performance to previously published alignment work for
multimodal models across a range of benchmarks.
668557da6c0ca62168747f72
Elmira Amirloo
668557da6c0ca62168747f73
Jean-Philippe Fauconnier
admin_assigned
2024-07-03T15:01:16.900Z
62cd5c43299c0c2e0e437842
/avatars/8fa8511baf2d9bd95d3ba4535a5b3d69.svg
Jean-Philippe Fauconnier
user
j4kn
668557da6c0ca62168747f74
Christoph Roesmann
claimed_verified
2024-07-03T14:44:39.303Z
668561a93468473e1930cad5
/avatars/8ec5f7216ebd5c45291d0f0f722978c0.svg
Christoph Roesmann
user
croesmann
668557da6c0ca62168747f75
Christian Kerl
668557da6c0ca62168747f76
Rinu Boney
668557da6c0ca62168747f77
Yusu Qian
admin_assigned
2024-07-03T15:01:38.374Z
65d5d6e590119106db895670
https://cdn-avatars.huggingface.co/v1/production/uploads/65d5d6e590119106db895670/SphTxh2E0MtT8T_bH0_zb.png
Yusu Qian
user
YusuQian
668557da6c0ca62168747f78
Zirui Wang
668557da6c0ca62168747f79
Afshin Dehghan
668557da6c0ca62168747f7a
Yinfei Yang
admin_assigned
2024-07-03T15:02:18.572Z
64b762568c632fbca942a405
/avatars/1eb737ec169967872f1ebf5ff29f1e6b.svg
Yinfei Yang
user
yinfeiy
668557da6c0ca62168747f7b
Zhe Gan
admin_assigned
2024-07-03T15:02:24.628Z
644022db21ec7c165d848b6e
/avatars/d9ed37ec77f2682878b2976a04d2fcd8.svg
Zhe Gan
user
zhegan27
668557da6c0ca62168747f7c
Peter Grasch
admin_assigned
2024-07-03T15:02:30.934Z
6649e0b6f50d471119285ab8
/avatars/1c93d0e15f2b68281a1a2ea052450a45.svg
Peter Grasch
user
petergrasch
Understanding Alignment in Multimodal LLMs: A Comprehensive Study
Preference alignment has become a crucial component in enhancing the
performance of Large Language Models (LLMs), yet its impact in Multimodal Large
Language Models (MLLMs) remains comparatively underexplored. Similar to
language models, MLLMs for image understanding tasks encounter challenges like
hallucination. In MLLMs, hallucination can occur not only by stating incorrect
facts but also by producing responses that are inconsistent with the image
content. A primary objective of alignment for MLLMs is to encourage these
models to align responses more closely with image information. Recently,
multiple works have introduced preference datasets for MLLMs and examined
different alignment methods, including Direct Preference Optimization (DPO) and
Proximal Policy Optimization (PPO). However, due to variations in datasets,
base model types, and alignment methods, it remains unclear which specific
elements contribute most significantly to the reported improvements in these
works. In this paper, we independently analyze each aspect of preference
alignment in MLLMs. We start by categorizing the alignment algorithms into two
groups, offline (such as DPO), and online (such as online-DPO), and show that
combining offline and online methods can improve the performance of the model
in certain scenarios. We review a variety of published multimodal preference
datasets and discuss how the details of their construction impact model
performance. Based on these insights, we introduce a novel way of creating
multimodal preference data called Bias-Driven Hallucination Sampling (BDHS)
that needs neither additional annotation nor external models, and show that it
can achieve competitive performance to previously published alignment work for
multimodal models across a range of benchmarks.
Preference alignment in Multimodal Large Language Models (MLLMs) is analyzed to understand which elements contribute to improved performance, leading to the introduction of Bias-Driven Hallucination Sampling (BDHS) for creating multimodal preference data.
Large Language Models (LLMs) Multimodal Large Language Models (MLLMs) preference alignment hallucination Direct Preference Optimization (DPO) Proximal Policy Optimization (PPO) offline methods online methods multimodal preference datasets Bias-Driven Hallucination Sampling (BDHS)