Visual Instruction Bottleneck Tuning
Despite widespread adoption, multimodal large language models (MLLMs) suffer
performance degradation when encountering unfamiliar queries under distribution
shifts. Existing methods to improve MLLM generalization typically require
either more instruction data or larger advanced model architectures, both of
which incur non-trivial human labor or computational costs. In this work, we
take an alternative approach to enhance the robustness of MLLMs under
distribution shifts, from a representation learning perspective. Inspired by
the information bottleneck (IB) principle, we derive a variational lower bound
of the IB for MLLMs and devise a practical implementation, Visual Instruction
Bottleneck Tuning (Vittle). We then provide a theoretical justification of
Vittle by revealing its connection to an information-theoretic robustness
metric of MLLM. Empirical validation of three MLLMs on open-ended and
closed-form question answering and object hallucination detection tasks over 45
datasets, including 30 shift scenarios, demonstrates that Vittle consistently
improves the MLLM's robustness under shifts by pursuing the learning of a
minimal sufficient representation.
682e33fa7a4bdea890eb25d0
Changdae Oh
claimed_verified
2025-05-22T07:17:24.158Z
672fc8ede7c89e44c9757259
/avatars/caa0d0de519ea96992c81328c89b3843.svg
Changdae Oh
user
changdae
682e33fa7a4bdea890eb25d1
Jiatong Li
682e33fa7a4bdea890eb25d2
Shawn Im
682e33fa7a4bdea890eb25d3
Yixuan Li
Visual Instruction Bottleneck Tuning
Despite widespread adoption, multimodal large language models (MLLMs) suffer
performance degradation when encountering unfamiliar queries under distribution
shifts. Existing methods to improve MLLM generalization typically require
either more instruction data or larger advanced model architectures, both of
which incur non-trivial human labor or computational costs. In this work, we
take an alternative approach to enhance the robustness of MLLMs under
distribution shifts, from a representation learning perspective. Inspired by
the information bottleneck (IB) principle, we derive a variational lower bound
of the IB for MLLMs and devise a practical implementation, Visual Instruction
Bottleneck Tuning (Vittle). We then provide a theoretical justification of
Vittle by revealing its connection to an information-theoretic robustness
metric of MLLM. Empirical validation of three MLLMs on open-ended and
closed-form question answering and object hallucination detection tasks over 45
datasets, including 30 shift scenarios, demonstrates that Vittle consistently
improves the MLLM's robustness under shifts by pursuing the learning of a
minimal sufficient representation.
A variational lower bound of the information bottleneck principle is implemented to enhance the robustness of multimodal large language models under distribution shifts.
multimodal large language models MLLMs distribution shifts information bottleneck principle variational lower bound Visual Instruction Bottleneck Tuning Vittle information-theoretic robustness metric minimal sufficient representation