LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One
  Vision Token
The advent of real-time large multimodal models (LMMs) like GPT-4o has
sparked considerable interest in efficient LMMs. LMM frameworks typically
encode visual inputs into vision tokens (continuous representations) and
integrate them and textual instructions into the context of large language
models (LLMs), where large-scale parameters and numerous context tokens
(predominantly vision tokens) result in substantial computational overhead.
Previous efforts towards efficient LMMs always focus on replacing the LLM
backbone with smaller models, while neglecting the crucial issue of token
quantity. In this paper, we introduce LLaVA-Mini, an efficient LMM with minimal
vision tokens. To achieve a high compression ratio of vision tokens while
preserving visual information, we first analyze how LMMs understand vision
tokens and find that most vision tokens only play a crucial role in the early
layers of LLM backbone, where they mainly fuse visual information into text
tokens. Building on this finding, LLaVA-Mini introduces modality pre-fusion to
fuse visual information into text tokens in advance, thereby facilitating the
extreme compression of vision tokens fed to LLM backbone into one token.
LLaVA-Mini is a unified large multimodal model that can support the
understanding of images, high-resolution images, and videos in an efficient
manner. Experiments across 11 image-based and 7 video-based benchmarks
demonstrate that LLaVA-Mini outperforms LLaVA-v1.5 with just 1 vision token
instead of 576. Efficiency analyses reveal that LLaVA-Mini can reduce FLOPs by
77%, deliver low-latency responses within 40 milliseconds, and process over
10,000 frames of video on the GPU hardware with 24GB of memory.
677ded917e773a03180e90c9
Shaolei Zhang
admin_assigned
2025-01-08T13:27:33.924Z
64803e5dc57f629056c601f1
/avatars/a9e9c97c70714e3a29bef2cf929ee6b3.svg
Shaolei Zhang
user
zhangshaolei
677ded917e773a03180e90ca
Qingkai Fang
admin_assigned
2025-01-08T13:27:41.696Z
65b7573482d384513443875e
/avatars/0f2175e4adf507f5ccb0636c1cb647de.svg
Qingkai Fang
user
poeroz
677ded917e773a03180e90cb
Zhe Yang
677ded917e773a03180e90cc
Yang Feng
admin_assigned
2025-01-08T13:29:03.040Z
63b39f33922f26a27e7e93dd
/avatars/fa1562f8c44270647826b293f49483bb.svg
Yang Feng
user
fengyang0317
LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One
  Vision Token
The advent of real-time large multimodal models (LMMs) like GPT-4o has
sparked considerable interest in efficient LMMs. LMM frameworks typically
encode visual inputs into vision tokens (continuous representations) and
integrate them and textual instructions into the context of large language
models (LLMs), where large-scale parameters and numerous context tokens
(predominantly vision tokens) result in substantial computational overhead.
Previous efforts towards efficient LMMs always focus on replacing the LLM
backbone with smaller models, while neglecting the crucial issue of token
quantity. In this paper, we introduce LLaVA-Mini, an efficient LMM with minimal
vision tokens. To achieve a high compression ratio of vision tokens while
preserving visual information, we first analyze how LMMs understand vision
tokens and find that most vision tokens only play a crucial role in the early
layers of LLM backbone, where they mainly fuse visual information into text
tokens. Building on this finding, LLaVA-Mini introduces modality pre-fusion to
fuse visual information into text tokens in advance, thereby facilitating the
extreme compression of vision tokens fed to LLM backbone into one token.
LLaVA-Mini is a unified large multimodal model that can support the
understanding of images, high-resolution images, and videos in an efficient
manner. Experiments across 11 image-based and 7 video-based benchmarks
demonstrate that LLaVA-Mini outperforms LLaVA-v1.5 with just 1 vision token
instead of 576. Efficiency analyses reveal that LLaVA-Mini can reduce FLOPs by
77%, deliver low-latency responses within 40 milliseconds, and process over
10,000 frames of video on the GPU hardware with 24GB of memory.
LLaVA-Mini, a unified large multimodal model, significantly reduces computational cost and latency by compressing vision tokens to a single token before feeding them to the LLM backbone, outperforming previous models.
real-time large multimodal models LMMs GPT-4o vision tokens text tokens large language models LLM token quantity modality pre-fusion FLOPs low-latency responses