title: MMInference: Accelerating Pre-filling for Long-Context VLMs via
    Modality-Aware Permutation Sparse Attention
summary: The integration of long-context capabilities with visual understanding
  unlocks unprecedented potential for Vision Language Models (VLMs). However, the
  quadratic attention complexity during the pre-filling phase remains a
  significant obstacle to real-world deployment. To overcome this limitation, we
  introduce MMInference (Multimodality Million tokens Inference), a dynamic
  sparse attention method that accelerates the prefilling stage for long-context
  multi-modal inputs. First, our analysis reveals that the temporal and spatial
  locality of video input leads to a unique sparse pattern, the Grid pattern.
  Simultaneously, VLMs exhibit markedly different sparse distributions across
  different modalities. We introduce a permutation-based method to leverage the
  unique Grid pattern and handle modality boundary issues. By offline search the
  optimal sparse patterns for each head, MMInference constructs the sparse
  distribution dynamically based on the input. We also provide optimized GPU
  kernels for efficient sparse computations. Notably, MMInference integrates
  seamlessly into existing VLM pipelines without any model modifications or
  fine-tuning. Experiments on multi-modal benchmarks-including Video QA,
  Captioning, VisionNIAH, and Mixed-Modality NIAH-with state-of-the-art
  long-context VLMs (LongVila, LlavaVideo, VideoChat-Flash, Qwen2.5-VL) show that
  MMInference accelerates the pre-filling stage by up to 8.3x at 1M tokens while
  maintaining accuracy. Our code is available at https://aka.ms/MMInference.
paper_authors._id: 68105d8632d635f02bc2976e
paper_authors.name: Yucheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-29T12:26:01.688Z
paper_authors._id: 655ea46bb11e49dd1fd6ce2e
paper_authors.avatarUrl: /avatars/671771dc565cf718f677536f91f1e8a3.svg
paper_authors.fullname: Yuchengli
paper_authors.type: user
paper_authors.user: Yuchengli
paper_authors._id: 68105d8632d635f02bc2976f
paper_authors.name: Huiqiang Jiang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-05-08T06:07:20.459Z
paper_authors._id: 6278bd42541f3d2dfa77ea70
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6278bd42541f3d2dfa77ea70/ejn49eapnB3UXQckAYdTd.jpeg
paper_authors.fullname: Huiqiang Jiang
paper_authors.type: user
paper_authors.user: iofu728
paper_authors._id: 68105d8632d635f02bc29770
paper_authors.name: Chengruidong Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-29T12:26:11.194Z
paper_authors._id: 64646896884f2e3e1ced3cd5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64646896884f2e3e1ced3cd5/86-t8V8LGMNaPQRXnADiD.png
paper_authors.fullname: Zhang
paper_authors.type: user
paper_authors.user: Chengruidong
paper_authors._id: 68105d8632d635f02bc29771
paper_authors.name: Qianhui Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-29T12:26:17.854Z
paper_authors._id: 63ef330b1e695b35aa484e11
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63ef330b1e695b35aa484e11/bXwpGy0dl8JXeJwJ--ilr.jpeg
paper_authors.fullname: Qianhui WU
paper_authors.type: user
paper_authors.user: qianhuiwu
paper_authors._id: 68105d8632d635f02bc29772
paper_authors.name: Xufang Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-29T12:26:38.807Z
paper_authors._id: 64b750a2fdb702b3d8619514
paper_authors.avatarUrl: /avatars/f09181c0825763dff692c4bc65effc4c.svg
paper_authors.fullname: Xufang Luo
paper_authors.type: user
paper_authors.user: luoxufang
paper_authors._id: 68105d8632d635f02bc29773
paper_authors.name: Surin Ahn
paper_authors._id: 68105d8632d635f02bc29774
paper_authors.name: Amir H. Abdi
paper_authors._id: 68105d8632d635f02bc29775
paper_authors.name: Dongsheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-29T12:27:31.251Z
paper_authors._id: 6524f13821af4b07309015aa
paper_authors.avatarUrl: /avatars/f97d6a041428d48305568df7afd83cc8.svg
paper_authors.fullname: dongsheng li
paper_authors.type: user
paper_authors.user: dongshengli
paper_authors._id: 68105d8632d635f02bc29776
paper_authors.name: Jianfeng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-29T12:27:37.335Z
paper_authors._id: 641904caf9d6f1d772ec7af7
paper_authors.avatarUrl: /avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
paper_authors.fullname: Jianfeng Gao
paper_authors.type: user
paper_authors.user: wyngjf
paper_authors._id: 68105d8632d635f02bc29777
paper_authors.name: Yuqing Yang
paper_authors._id: 68105d8632d635f02bc29778
paper_authors.name: Lili Qiu
paper_title: MMInference: Accelerating Pre-filling for Long-Context VLMs via
    Modality-Aware Permutation Sparse Attention
paper_summary: The integration of long-context capabilities with visual understanding
  unlocks unprecedented potential for Vision Language Models (VLMs). However, the
  quadratic attention complexity during the pre-filling phase remains a
  significant obstacle to real-world deployment. To overcome this limitation, we
  introduce MMInference (Multimodality Million tokens Inference), a dynamic
  sparse attention method that accelerates the prefilling stage for long-context
  multi-modal inputs. First, our analysis reveals that the temporal and spatial
  locality of video input leads to a unique sparse pattern, the Grid pattern.
  Simultaneously, VLMs exhibit markedly different sparse distributions across
  different modalities. We introduce a permutation-based method to leverage the
  unique Grid pattern and handle modality boundary issues. By offline search the
  optimal sparse patterns for each head, MMInference constructs the sparse
  distribution dynamically based on the input. We also provide optimized GPU
  kernels for efficient sparse computations. Notably, MMInference integrates
  seamlessly into existing VLM pipelines without any model modifications or
  fine-tuning. Experiments on multi-modal benchmarks-including Video QA,
  Captioning, VisionNIAH, and Mixed-Modality NIAH-with state-of-the-art
  long-context VLMs (LongVila, LlavaVideo, VideoChat-Flash, Qwen2.5-VL) show that
  MMInference accelerates the pre-filling stage by up to 8.3x at 1M tokens while
  maintaining accuracy. Our code is available at https://aka.ms/MMInference.
paper_ai_summary: MMInference, a dynamic sparse attention method, accelerates the pre-filling stage of Vision Language Models by leveraging unique sparse patterns in video input and modality boundary handling, achieving up to 8.3x speedup at 1M tokens.
paper_ai_keywords: long-context capabilities visual understanding Vision Language Models VLMs quadratic attention complexity dynamic sparse attention permuted-based method Grid pattern sparse distribution GPU kernels Video QA Captioning VisionNIAH Mixed-Modality NIAH LongVila LlavaVideo VideoChat-Flash Qwen2.5-VL