TokenPacker: Efficient Visual Projector for Multimodal LLM
The visual projector serves as an essential bridge between the visual encoder
and the Large Language Model (LLM) in a Multimodal LLM (MLLM). Typically, MLLMs
adopt a simple MLP to preserve all visual contexts via one-to-one
transformation. However, the visual tokens are redundant and can be
considerably increased when dealing with high-resolution images, impairing the
efficiency of MLLMs significantly. Some recent works have introduced resampler
or abstractor to reduce the number of resulting visual tokens. Unfortunately,
they fail to capture finer details and undermine the visual reasoning
capabilities of MLLMs. In this work, we propose a novel visual projector, which
adopts a coarse-to-fine scheme to inject the enriched characteristics to
generate the condensed visual tokens. In specific, we first interpolate the
visual features as a low-resolution point query, providing the overall visual
representation as the foundation. Then, we introduce a region-to-point
injection module that utilizes high-resolution, multi-level region-based cues
as fine-grained reference keys and values, allowing them to be fully absorbed
within the corresponding local context region. This step effectively updates
the coarse point query, transforming it into an enriched one for the subsequent
LLM reasoning. Extensive experiments demonstrate that our approach compresses
the visual tokens by 75%~89%, while achieves comparable or even better
performance across diverse benchmarks with significantly higher efficiency. The
source codes can be found at https://github.com/CircleRadon/TokenPacker.
668618e247f2a33570ed316e
Wentong Li
claimed_verified
2024-07-22T07:12:20.412Z
64c48a78d07620bdc99777d4
https://cdn-avatars.huggingface.co/v1/production/uploads/64c48a78d07620bdc99777d4/NJC4Ot0a7YSdU5RC6dgga.jpeg
LI WENTONG
user
sunshine-lwt
668618e247f2a33570ed316f
Yuqian Yuan
claimed_verified
2024-07-04T08:03:26.166Z
64a3fe3dde901eb01df12398
https://cdn-avatars.huggingface.co/v1/production/uploads/64a3fe3dde901eb01df12398/Js2bEx4rxKuEKVt5z9I2D.jpeg
YuqianYuan
user
CircleRadon
668618e247f2a33570ed3170
Jian Liu
668618e247f2a33570ed3171
Dongqi Tang
admin_assigned
2024-07-04T12:13:51.284Z
6353b31206d707b332426678
/avatars/901a91f8270d3363f89a26785e471586.svg
Dongqi Tang
user
coura
668618e247f2a33570ed3172
Song Wang
claimed_verified
2025-05-29T20:18:10.779Z
66863d26e2b71e3d09189ae9
/avatars/3c0e6f30e053f2e622ae75e1dc43edba.svg
Song Wang
user
songw-zju
668618e247f2a33570ed3173
Jianke Zhu
668618e247f2a33570ed3174
Lei Zhang
TokenPacker: Efficient Visual Projector for Multimodal LLM
The visual projector serves as an essential bridge between the visual encoder
and the Large Language Model (LLM) in a Multimodal LLM (MLLM). Typically, MLLMs
adopt a simple MLP to preserve all visual contexts via one-to-one
transformation. However, the visual tokens are redundant and can be
considerably increased when dealing with high-resolution images, impairing the
efficiency of MLLMs significantly. Some recent works have introduced resampler
or abstractor to reduce the number of resulting visual tokens. Unfortunately,
they fail to capture finer details and undermine the visual reasoning
capabilities of MLLMs. In this work, we propose a novel visual projector, which
adopts a coarse-to-fine scheme to inject the enriched characteristics to
generate the condensed visual tokens. In specific, we first interpolate the
visual features as a low-resolution point query, providing the overall visual
representation as the foundation. Then, we introduce a region-to-point
injection module that utilizes high-resolution, multi-level region-based cues
as fine-grained reference keys and values, allowing them to be fully absorbed
within the corresponding local context region. This step effectively updates
the coarse point query, transforming it into an enriched one for the subsequent
LLM reasoning. Extensive experiments demonstrate that our approach compresses
the visual tokens by 75%~89%, while achieves comparable or even better
performance across diverse benchmarks with significantly higher efficiency. The
source codes can be found at https://github.com/CircleRadon/TokenPacker.
A novel visual projector using a coarse-to-fine scheme reduces visual token redundancy and improves visual reasoning in Multimodal LLMs.
MLLM visual projector MLP visual tokens resampler abstractor coarse-to-fine scheme region-to-point injection module visual features low-resolution point query high-resolution multi-level region-based cues fine-grained reference keys local context region LLM reasoning