title: DyMU: Dynamic Merging and Virtual Unmerging for Efficient VLMs
summary: We present DyMU, an efficient, training-free framework that dynamically
  reduces the computational burden of vision-language models (VLMs) while
  maintaining high task performance. Our approach comprises two key components.
  First, Dynamic Token Merging (DToMe) reduces the number of visual token
  embeddings by merging similar tokens based on image complexity, addressing the
  inherent inefficiency of fixed-length outputs in vision transformers. Second,
  Virtual Token Unmerging (VTU) simulates the expected token sequence for large
  language models (LLMs) by efficiently reconstructing the attention dynamics of
  a full sequence, thus preserving the downstream performance without additional
  fine-tuning. Unlike previous approaches, our method dynamically adapts token
  compression to the content of the image and operates completely training-free,
  making it readily applicable to most state-of-the-art VLM architectures.
  Extensive experiments on image and video understanding tasks demonstrate that
  DyMU can reduce the average visual token count by 32%-85% while achieving
  comparable performance to full-length models across diverse VLM architectures,
  including the recently popularized AnyRes-based visual encoders. Furthermore,
  through qualitative analyses, we demonstrate that DToMe effectively adapts
  token reduction based on image complexity and, unlike existing systems,
  provides users more control over computational costs. Project page:
  https://mikewangwzhl.github.io/dymu/.
paper_authors._id: 680af0c4175842e433ae348e
paper_authors.name: Zhenhailong Wang
paper_authors._id: 680af0c4175842e433ae348f
paper_authors.name: Senthil Purushwalkam
paper_authors._id: 680af0c4175842e433ae3490
paper_authors.name: Caiming Xiong
paper_authors._id: 680af0c4175842e433ae3491
paper_authors.name: Silvio Savarese
paper_authors._id: 680af0c4175842e433ae3492
paper_authors.name: Heng Ji
paper_authors._id: 680af0c4175842e433ae3493
paper_authors.name: Ran Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-15T10:33:19.970Z
paper_authors._id: 6465c4c863e7e09dd02e3e1b
paper_authors.avatarUrl: /avatars/200b029184d2616f98296a2c212f0785.svg
paper_authors.fullname: Ran Xu
paper_authors.type: user
paper_authors.user: xurantju
paper_title: DyMU: Dynamic Merging and Virtual Unmerging for Efficient VLMs
paper_summary: We present DyMU, an efficient, training-free framework that dynamically
  reduces the computational burden of vision-language models (VLMs) while
  maintaining high task performance. Our approach comprises two key components.
  First, Dynamic Token Merging (DToMe) reduces the number of visual token
  embeddings by merging similar tokens based on image complexity, addressing the
  inherent inefficiency of fixed-length outputs in vision transformers. Second,
  Virtual Token Unmerging (VTU) simulates the expected token sequence for large
  language models (LLMs) by efficiently reconstructing the attention dynamics of
  a full sequence, thus preserving the downstream performance without additional
  fine-tuning. Unlike previous approaches, our method dynamically adapts token
  compression to the content of the image and operates completely training-free,
  making it readily applicable to most state-of-the-art VLM architectures.
  Extensive experiments on image and video understanding tasks demonstrate that
  DyMU can reduce the average visual token count by 32%-85% while achieving
  comparable performance to full-length models across diverse VLM architectures,
  including the recently popularized AnyRes-based visual encoders. Furthermore,
  through qualitative analyses, we demonstrate that DToMe effectively adapts
  token reduction based on image complexity and, unlike existing systems,
  provides users more control over computational costs. Project page:
  https://mikewangwzhl.github.io/dymu/.
paper_ai_summary: DyMU, an efficient, training-free framework, reduces the computational load of vision-language models by dynamically merging and unmerging tokens without performance loss.
paper_ai_keywords: Dynamic Token Merging DToMe Virtual Token Unmerging VTU vision-language models VLMs vision transformers large language models LLMs attention dynamics computational burden token compression AnyRes-based visual encoders