title: LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One
    Vision Token
summary: The advent of real-time large multimodal models (LMMs) like GPT-4o has
  sparked considerable interest in efficient LMMs. LMM frameworks typically
  encode visual inputs into vision tokens (continuous representations) and
  integrate them and textual instructions into the context of large language
  models (LLMs), where large-scale parameters and numerous context tokens
  (predominantly vision tokens) result in substantial computational overhead.
  Previous efforts towards efficient LMMs always focus on replacing the LLM
  backbone with smaller models, while neglecting the crucial issue of token
  quantity. In this paper, we introduce LLaVA-Mini, an efficient LMM with minimal
  vision tokens. To achieve a high compression ratio of vision tokens while
  preserving visual information, we first analyze how LMMs understand vision
  tokens and find that most vision tokens only play a crucial role in the early
  layers of LLM backbone, where they mainly fuse visual information into text
  tokens. Building on this finding, LLaVA-Mini introduces modality pre-fusion to
  fuse visual information into text tokens in advance, thereby facilitating the
  extreme compression of vision tokens fed to LLM backbone into one token.
  LLaVA-Mini is a unified large multimodal model that can support the
  understanding of images, high-resolution images, and videos in an efficient
  manner. Experiments across 11 image-based and 7 video-based benchmarks
  demonstrate that LLaVA-Mini outperforms LLaVA-v1.5 with just 1 vision token
  instead of 576. Efficiency analyses reveal that LLaVA-Mini can reduce FLOPs by
  77%, deliver low-latency responses within 40 milliseconds, and process over
  10,000 frames of video on the GPU hardware with 24GB of memory.
paper_authors._id: 677ded917e773a03180e90c9
paper_authors.name: Shaolei Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-08T13:27:33.924Z
paper_authors._id: 64803e5dc57f629056c601f1
paper_authors.avatarUrl: /avatars/a9e9c97c70714e3a29bef2cf929ee6b3.svg
paper_authors.fullname: Shaolei Zhang
paper_authors.type: user
paper_authors.user: zhangshaolei
paper_authors._id: 677ded917e773a03180e90ca
paper_authors.name: Qingkai Fang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-08T13:27:41.696Z
paper_authors._id: 65b7573482d384513443875e
paper_authors.avatarUrl: /avatars/0f2175e4adf507f5ccb0636c1cb647de.svg
paper_authors.fullname: Qingkai Fang
paper_authors.type: user
paper_authors.user: poeroz
paper_authors._id: 677ded917e773a03180e90cb
paper_authors.name: Zhe Yang
paper_authors._id: 677ded917e773a03180e90cc
paper_authors.name: Yang Feng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-08T13:29:03.040Z
paper_authors._id: 63b39f33922f26a27e7e93dd
paper_authors.avatarUrl: /avatars/fa1562f8c44270647826b293f49483bb.svg
paper_authors.fullname: Yang Feng
paper_authors.type: user
paper_authors.user: fengyang0317
paper_title: LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One
    Vision Token
paper_summary: The advent of real-time large multimodal models (LMMs) like GPT-4o has
  sparked considerable interest in efficient LMMs. LMM frameworks typically
  encode visual inputs into vision tokens (continuous representations) and
  integrate them and textual instructions into the context of large language
  models (LLMs), where large-scale parameters and numerous context tokens
  (predominantly vision tokens) result in substantial computational overhead.
  Previous efforts towards efficient LMMs always focus on replacing the LLM
  backbone with smaller models, while neglecting the crucial issue of token
  quantity. In this paper, we introduce LLaVA-Mini, an efficient LMM with minimal
  vision tokens. To achieve a high compression ratio of vision tokens while
  preserving visual information, we first analyze how LMMs understand vision
  tokens and find that most vision tokens only play a crucial role in the early
  layers of LLM backbone, where they mainly fuse visual information into text
  tokens. Building on this finding, LLaVA-Mini introduces modality pre-fusion to
  fuse visual information into text tokens in advance, thereby facilitating the
  extreme compression of vision tokens fed to LLM backbone into one token.
  LLaVA-Mini is a unified large multimodal model that can support the
  understanding of images, high-resolution images, and videos in an efficient
  manner. Experiments across 11 image-based and 7 video-based benchmarks
  demonstrate that LLaVA-Mini outperforms LLaVA-v1.5 with just 1 vision token
  instead of 576. Efficiency analyses reveal that LLaVA-Mini can reduce FLOPs by
  77%, deliver low-latency responses within 40 milliseconds, and process over
  10,000 frames of video on the GPU hardware with 24GB of memory.
paper_ai_summary: LLaVA-Mini, a unified large multimodal model, significantly reduces computational cost and latency by compressing vision tokens to a single token before feeding them to the LLM backbone, outperforming previous models.
paper_ai_keywords: real-time large multimodal models LMMs GPT-4o vision tokens text tokens large language models LLM token quantity modality pre-fusion FLOPs low-latency responses