title: TokenPacker: Efficient Visual Projector for Multimodal LLM
summary: The visual projector serves as an essential bridge between the visual encoder
  and the Large Language Model (LLM) in a Multimodal LLM (MLLM). Typically, MLLMs
  adopt a simple MLP to preserve all visual contexts via one-to-one
  transformation. However, the visual tokens are redundant and can be
  considerably increased when dealing with high-resolution images, impairing the
  efficiency of MLLMs significantly. Some recent works have introduced resampler
  or abstractor to reduce the number of resulting visual tokens. Unfortunately,
  they fail to capture finer details and undermine the visual reasoning
  capabilities of MLLMs. In this work, we propose a novel visual projector, which
  adopts a coarse-to-fine scheme to inject the enriched characteristics to
  generate the condensed visual tokens. In specific, we first interpolate the
  visual features as a low-resolution point query, providing the overall visual
  representation as the foundation. Then, we introduce a region-to-point
  injection module that utilizes high-resolution, multi-level region-based cues
  as fine-grained reference keys and values, allowing them to be fully absorbed
  within the corresponding local context region. This step effectively updates
  the coarse point query, transforming it into an enriched one for the subsequent
  LLM reasoning. Extensive experiments demonstrate that our approach compresses
  the visual tokens by 75%~89%, while achieves comparable or even better
  performance across diverse benchmarks with significantly higher efficiency. The
  source codes can be found at https://github.com/CircleRadon/TokenPacker.
paper_authors._id: 668618e247f2a33570ed316e
paper_authors.name: Wentong Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-22T07:12:20.412Z
paper_authors._id: 64c48a78d07620bdc99777d4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c48a78d07620bdc99777d4/NJC4Ot0a7YSdU5RC6dgga.jpeg
paper_authors.fullname: LI WENTONG
paper_authors.type: user
paper_authors.user: sunshine-lwt
paper_authors._id: 668618e247f2a33570ed316f
paper_authors.name: Yuqian Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-04T08:03:26.166Z
paper_authors._id: 64a3fe3dde901eb01df12398
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64a3fe3dde901eb01df12398/Js2bEx4rxKuEKVt5z9I2D.jpeg
paper_authors.fullname: YuqianYuan
paper_authors.type: user
paper_authors.user: CircleRadon
paper_authors._id: 668618e247f2a33570ed3170
paper_authors.name: Jian Liu
paper_authors._id: 668618e247f2a33570ed3171
paper_authors.name: Dongqi Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-04T12:13:51.284Z
paper_authors._id: 6353b31206d707b332426678
paper_authors.avatarUrl: /avatars/901a91f8270d3363f89a26785e471586.svg
paper_authors.fullname: Dongqi Tang
paper_authors.type: user
paper_authors.user: coura
paper_authors._id: 668618e247f2a33570ed3172
paper_authors.name: Song Wang
paper_authors._id: 668618e247f2a33570ed3173
paper_authors.name: Jianke Zhu
paper_authors._id: 668618e247f2a33570ed3174
paper_authors.name: Lei Zhang
paper_title: TokenPacker: Efficient Visual Projector for Multimodal LLM
paper_summary: The visual projector serves as an essential bridge between the visual encoder
  and the Large Language Model (LLM) in a Multimodal LLM (MLLM). Typically, MLLMs
  adopt a simple MLP to preserve all visual contexts via one-to-one
  transformation. However, the visual tokens are redundant and can be
  considerably increased when dealing with high-resolution images, impairing the
  efficiency of MLLMs significantly. Some recent works have introduced resampler
  or abstractor to reduce the number of resulting visual tokens. Unfortunately,
  they fail to capture finer details and undermine the visual reasoning
  capabilities of MLLMs. In this work, we propose a novel visual projector, which
  adopts a coarse-to-fine scheme to inject the enriched characteristics to
  generate the condensed visual tokens. In specific, we first interpolate the
  visual features as a low-resolution point query, providing the overall visual
  representation as the foundation. Then, we introduce a region-to-point
  injection module that utilizes high-resolution, multi-level region-based cues
  as fine-grained reference keys and values, allowing them to be fully absorbed
  within the corresponding local context region. This step effectively updates
  the coarse point query, transforming it into an enriched one for the subsequent
  LLM reasoning. Extensive experiments demonstrate that our approach compresses
  the visual tokens by 75%~89%, while achieves comparable or even better
  performance across diverse benchmarks with significantly higher efficiency. The
  source codes can be found at https://github.com/CircleRadon/TokenPacker.
paper_ai_keywords: visual projector Large Language Model (LLM) Multimodal LLM (MLLM) MLP (Multilayer Perceptron) visual tokens high-resolution images resampler abstractor coarse-to-fine scheme visual features low-resolution point query region-to-point injection module high-resolution, multi-level region-based cues reference keys reference values local context region LLM reasoning visual token compression