title: Dense Connector for MLLMs
summary: Do we fully leverage the potential of visual encoder in Multimodal Large
  Language Models (MLLMs)? The recent outstanding performance of MLLMs in
  multimodal understanding has garnered broad attention from both academia and
  industry. In the current MLLM rat race, the focus seems to be predominantly on
  the linguistic side. We witness the rise of larger and higher-quality
  instruction datasets, as well as the involvement of larger-sized LLMs. Yet,
  scant attention has been directed towards the visual signals utilized by MLLMs,
  often assumed to be the final high-level features extracted by a frozen visual
  encoder. In this paper, we introduce the Dense Connector - a simple, effective,
  and plug-and-play vision-language connector that significantly enhances
  existing MLLMs by leveraging multi-layer visual features, with minimal
  additional computational overhead. Furthermore, our model, trained solely on
  images, showcases remarkable zero-shot capabilities in video understanding as
  well. Experimental results across various vision encoders, image resolutions,
  training dataset scales, varying sizes of LLMs (2.7B->70B), and diverse
  architectures of MLLMs (e.g., LLaVA and Mini-Gemini) validate the versatility
  and scalability of our approach, achieving state-of-the-art performance on
  across 19 image and video benchmarks. We hope that this work will provide
  valuable experience and serve as a basic module for future MLLM development.
paper_authors._id: 6650187ca120d855a7f00a10
paper_authors.name: Huanjin Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-24T10:59:37.410Z
paper_authors._id: 6590e03454f8826173ed5ee6
paper_authors.avatarUrl: /avatars/f5e59d3e58c28a99f2ff39267ca51cdb.svg
paper_authors.fullname: Huanjin Yao
paper_authors.type: user
paper_authors.user: HuanjinYao
paper_authors._id: 6650187ca120d855a7f00a11
paper_authors.name: Wenhao Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-24T10:59:44.675Z
paper_authors._id: 64b66ddbd5a02e636b577dcc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b66ddbd5a02e636b577dcc/61YPkyef3m3k5dpVI1d86.jpeg
paper_authors.fullname: Wenhao Wu
paper_authors.type: user
paper_authors.user: whwu-me
paper_authors._id: 6650187ca120d855a7f00a12
paper_authors.name: Taojiannan Yang
paper_authors._id: 6650187ca120d855a7f00a13
paper_authors.name: YuXin Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-24T11:00:03.632Z
paper_authors._id: 63ad98c142fd3b8dbae78f51
paper_authors.avatarUrl: /avatars/d7a96f0ad47a729027757b850b5e9712.svg
paper_authors.fullname: Yuxin Song
paper_authors.type: user
paper_authors.user: syxbb
paper_authors._id: 6650187ca120d855a7f00a14
paper_authors.name: Mengxi Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-24T11:00:09.604Z
paper_authors._id: 65f2a7267d5cdbce8882548b
paper_authors.avatarUrl: /avatars/f0244fa4e71403d05ea8bfd25986a02b.svg
paper_authors.fullname: mengxizhang
paper_authors.type: user
paper_authors.user: mengxizhang
paper_authors._id: 6650187ca120d855a7f00a15
paper_authors.name: Haocheng Feng
paper_authors._id: 6650187ca120d855a7f00a16
paper_authors.name: Yifan Sun
paper_authors._id: 6650187ca120d855a7f00a17
paper_authors.name: Zhiheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-24T11:01:22.958Z
paper_authors._id: 60f86f2085dbbb185d2e9a5b
paper_authors.avatarUrl: /avatars/52b0a8471115a8aa400e315224530e8a.svg
paper_authors.fullname: Zhiheng Li
paper_authors.type: user
paper_authors.user: hubertlee915
paper_authors._id: 6650187ca120d855a7f00a18
paper_authors.name: Wanli Ouyang
paper_authors._id: 6650187ca120d855a7f00a19
paper_authors.name: Jingdong Wang
paper_title: Dense Connector for MLLMs
paper_summary: Do we fully leverage the potential of visual encoder in Multimodal Large
  Language Models (MLLMs)? The recent outstanding performance of MLLMs in
  multimodal understanding has garnered broad attention from both academia and
  industry. In the current MLLM rat race, the focus seems to be predominantly on
  the linguistic side. We witness the rise of larger and higher-quality
  instruction datasets, as well as the involvement of larger-sized LLMs. Yet,
  scant attention has been directed towards the visual signals utilized by MLLMs,
  often assumed to be the final high-level features extracted by a frozen visual
  encoder. In this paper, we introduce the Dense Connector - a simple, effective,
  and plug-and-play vision-language connector that significantly enhances
  existing MLLMs by leveraging multi-layer visual features, with minimal
  additional computational overhead. Furthermore, our model, trained solely on
  images, showcases remarkable zero-shot capabilities in video understanding as
  well. Experimental results across various vision encoders, image resolutions,
  training dataset scales, varying sizes of LLMs (2.7B->70B), and diverse
  architectures of MLLMs (e.g., LLaVA and Mini-Gemini) validate the versatility
  and scalability of our approach, achieving state-of-the-art performance on
  across 19 image and video benchmarks. We hope that this work will provide
  valuable experience and serve as a basic module for future MLLM development.
paper_ai_summary: The Dense Connector enhances Multimodal Large Language Models by integrating multi-layer visual features, improving performance across image and video benchmarks.
paper_ai_keywords: Dense Connector vision-language connector multi-layer visual features Multimodal Large Language Models MLLMs zero-shot capabilities video understanding image and video benchmarks