title: Visual Instruction Bottleneck Tuning
summary: Despite widespread adoption, multimodal large language models (MLLMs) suffer
  performance degradation when encountering unfamiliar queries under distribution
  shifts. Existing methods to improve MLLM generalization typically require
  either more instruction data or larger advanced model architectures, both of
  which incur non-trivial human labor or computational costs. In this work, we
  take an alternative approach to enhance the robustness of MLLMs under
  distribution shifts, from a representation learning perspective. Inspired by
  the information bottleneck (IB) principle, we derive a variational lower bound
  of the IB for MLLMs and devise a practical implementation, Visual Instruction
  Bottleneck Tuning (Vittle). We then provide a theoretical justification of
  Vittle by revealing its connection to an information-theoretic robustness
  metric of MLLM. Empirical validation of three MLLMs on open-ended and
  closed-form question answering and object hallucination detection tasks over 45
  datasets, including 30 shift scenarios, demonstrates that Vittle consistently
  improves the MLLM's robustness under shifts by pursuing the learning of a
  minimal sufficient representation.
paper_authors._id: 682e33fa7a4bdea890eb25d0
paper_authors.name: Changdae Oh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-22T07:17:24.158Z
paper_authors._id: 672fc8ede7c89e44c9757259
paper_authors.avatarUrl: /avatars/caa0d0de519ea96992c81328c89b3843.svg
paper_authors.fullname: Changdae Oh
paper_authors.type: user
paper_authors.user: changdae
paper_authors._id: 682e33fa7a4bdea890eb25d1
paper_authors.name: Jiatong Li
paper_authors._id: 682e33fa7a4bdea890eb25d2
paper_authors.name: Shawn Im
paper_authors._id: 682e33fa7a4bdea890eb25d3
paper_authors.name: Yixuan Li
paper_title: Visual Instruction Bottleneck Tuning
paper_summary: Despite widespread adoption, multimodal large language models (MLLMs) suffer
  performance degradation when encountering unfamiliar queries under distribution
  shifts. Existing methods to improve MLLM generalization typically require
  either more instruction data or larger advanced model architectures, both of
  which incur non-trivial human labor or computational costs. In this work, we
  take an alternative approach to enhance the robustness of MLLMs under
  distribution shifts, from a representation learning perspective. Inspired by
  the information bottleneck (IB) principle, we derive a variational lower bound
  of the IB for MLLMs and devise a practical implementation, Visual Instruction
  Bottleneck Tuning (Vittle). We then provide a theoretical justification of
  Vittle by revealing its connection to an information-theoretic robustness
  metric of MLLM. Empirical validation of three MLLMs on open-ended and
  closed-form question answering and object hallucination detection tasks over 45
  datasets, including 30 shift scenarios, demonstrates that Vittle consistently
  improves the MLLM's robustness under shifts by pursuing the learning of a
  minimal sufficient representation.
paper_ai_summary: A variational lower bound of the information bottleneck principle is implemented to enhance the robustness of multimodal large language models under distribution shifts.
paper_ai_keywords: multimodal large language models MLLMs distribution shifts information bottleneck principle variational lower bound Visual Instruction Bottleneck Tuning Vittle information-theoretic robustness metric minimal sufficient representation