title: Honeybee: Locality-enhanced Projector for Multimodal LLM
summary: In Multimodal Large Language Models (MLLMs), a visual projector plays a
  crucial role in bridging pre-trained vision encoders with LLMs, enabling
  profound visual understanding while harnessing the LLMs' robust capabilities.
  Despite the importance of the visual projector, it has been relatively less
  explored. In this study, we first identify two essential projector properties:
  (i) flexibility in managing the number of visual tokens, crucial for MLLMs'
  overall efficiency, and (ii) preservation of local context from visual
  features, vital for spatial understanding. Based on these findings, we propose
  a novel projector design that is both flexible and locality-enhanced,
  effectively satisfying the two desirable properties. Additionally, we present
  comprehensive strategies to effectively utilize multiple and multifaceted
  instruction datasets. Through extensive experiments, we examine the impact of
  individual design choices. Finally, our proposed MLLM, Honeybee, remarkably
  outperforms previous state-of-the-art methods across various benchmarks,
  including MME, MMBench, SEED-Bench, and LLaVA-Bench, achieving significantly
  higher efficiency. Code and models are available at
  https://github.com/kakaobrain/honeybee.
paper_authors._id: 65791a18c17736d75f7d6e42
paper_authors.name: Junbum Cha
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-13T09:40:07.151Z
paper_authors._id: 641fee15b286e8c464f14f10
paper_authors.avatarUrl: /avatars/dbfb6ac17156086df2408703f325881e.svg
paper_authors.fullname: Junbum Cha
paper_authors.type: user
paper_authors.user: khanrc
paper_authors._id: 65791a18c17736d75f7d6e43
paper_authors.name: Wooyoung Kang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-13T09:40:16.303Z
paper_authors._id: 64b785c84df206a3ed1443ba
paper_authors.avatarUrl: /avatars/141d4ae55f604c54eadd4aacb4e29514.svg
paper_authors.fullname: Woo-young Kang
paper_authors.type: user
paper_authors.user: Edwin-kang
paper_authors._id: 65791a18c17736d75f7d6e44
paper_authors.name: Jonghwan Mun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-13T09:40:23.331Z
paper_authors._id: 64c9cc003a3f428da6927587
paper_authors.avatarUrl: /avatars/05daf4190b3c96b7ef4c5c9fe78697c3.svg
paper_authors.fullname: Jonghwan Mun
paper_authors.type: user
paper_authors.user: jhmun
paper_authors._id: 65791a18c17736d75f7d6e45
paper_authors.name: Byungseok Roh
paper_title: Honeybee: Locality-enhanced Projector for Multimodal LLM
paper_summary: In Multimodal Large Language Models (MLLMs), a visual projector plays a
  crucial role in bridging pre-trained vision encoders with LLMs, enabling
  profound visual understanding while harnessing the LLMs' robust capabilities.
  Despite the importance of the visual projector, it has been relatively less
  explored. In this study, we first identify two essential projector properties:
  (i) flexibility in managing the number of visual tokens, crucial for MLLMs'
  overall efficiency, and (ii) preservation of local context from visual
  features, vital for spatial understanding. Based on these findings, we propose
  a novel projector design that is both flexible and locality-enhanced,
  effectively satisfying the two desirable properties. Additionally, we present
  comprehensive strategies to effectively utilize multiple and multifaceted
  instruction datasets. Through extensive experiments, we examine the impact of
  individual design choices. Finally, our proposed MLLM, Honeybee, remarkably
  outperforms previous state-of-the-art methods across various benchmarks,
  including MME, MMBench, SEED-Bench, and LLaVA-Bench, achieving significantly
  higher efficiency. Code and models are available at
  https://github.com/kakaobrain/honeybee.
paper_ai_keywords: visual projector pre-trained vision encoders Multimodal Large Language Models (MLLMs) visual tokens local context spatial understanding locality-enhanced instruction datasets MME MMBench SEED-Bench LLaVA-Bench