title: LongVU: Spatiotemporal Adaptive Compression for Long Video-Language
    Understanding
summary: Multimodal Large Language Models (MLLMs) have shown promising progress in
  understanding and analyzing video content. However, processing long videos
  remains a significant challenge constrained by LLM's context size. To address
  this limitation, we propose LongVU, a spatiotemporal adaptive compression
  mechanism thats reduces the number of video tokens while preserving visual
  details of long videos. Our idea is based on leveraging cross-modal query and
  inter-frame dependencies to adaptively reduce temporal and spatial redundancy
  in videos. Specifically, we leverage DINOv2 features to remove redundant frames
  that exhibit high similarity. Then we utilize text-guided cross-modal query for
  selective frame feature reduction. Further, we perform spatial token reduction
  across frames based on their temporal dependencies. Our adaptive compression
  strategy effectively processes a large number of frames with little visual
  information loss within given context length. Our LongVU consistently surpass
  existing methods across a variety of video understanding benchmarks, especially
  on hour-long video understanding tasks such as VideoMME and MLVU. Given a
  light-weight LLM, our LongVU also scales effectively into a smaller size with
  state-of-the-art video understanding performance.
paper_authors._id: 6719f9b5248e67e1a920a554
paper_authors.name: Xiaoqian Shen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-25T09:31:01.552Z
paper_authors._id: 620c5b20c33ca69c4c82145b
paper_authors.avatarUrl: /avatars/f3b532c47e434be30d1005fe6d0e25fb.svg
paper_authors.fullname: Xiaoqian Shen
paper_authors.type: user
paper_authors.user: shenxq
paper_authors._id: 6719f9b5248e67e1a920a555
paper_authors.name: Yunyang Xiong
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-10-24T08:06:59.406Z
paper_authors._id: 65304b62e7535baecd85d080
paper_authors.avatarUrl: /avatars/6e546c7d1414bd92c5a7c8d8c404de92.svg
paper_authors.fullname: Yunyang Xiong
paper_authors.type: user
paper_authors.user: yunyangx
paper_authors._id: 6719f9b5248e67e1a920a556
paper_authors.name: Changsheng Zhao
paper_authors._id: 6719f9b5248e67e1a920a557
paper_authors.name: Lemeng Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-25T09:30:59.791Z
paper_authors._id: 6364acf56fa33d2f59397c59
paper_authors.avatarUrl: /avatars/f642a7dcc098f50fef865f30f93a25c3.svg
paper_authors.fullname: Lemeng Wu
paper_authors.type: user
paper_authors.user: klight
paper_authors._id: 6719f9b5248e67e1a920a558
paper_authors.name: Jun Chen
paper_authors._id: 6719f9b5248e67e1a920a559
paper_authors.name: Chenchen Zhu
paper_authors._id: 6719f9b5248e67e1a920a55a
paper_authors.name: Zechun Liu
paper_authors._id: 6719f9b5248e67e1a920a55b
paper_authors.name: Fanyi Xiao
paper_authors._id: 6719f9b5248e67e1a920a55c
paper_authors.name: Balakrishnan Varadarajan
paper_authors._id: 6719f9b5248e67e1a920a55d
paper_authors.name: Florian Bordes
paper_authors._id: 6719f9b5248e67e1a920a55e
paper_authors.name: Zhuang Liu
paper_authors._id: 6719f9b5248e67e1a920a55f
paper_authors.name: Hu Xu
paper_authors._id: 6719f9b5248e67e1a920a560
paper_authors.name: Hyunwoo J. Kim
paper_authors._id: 6719f9b5248e67e1a920a561
paper_authors.name: Bilge Soran
paper_authors._id: 6719f9b5248e67e1a920a562
paper_authors.name: Raghuraman Krishnamoorthi
paper_authors._id: 6719f9b5248e67e1a920a563
paper_authors.name: Mohamed Elhoseiny
paper_authors._id: 6719f9b5248e67e1a920a564
paper_authors.name: Vikas Chandra
paper_title: LongVU: Spatiotemporal Adaptive Compression for Long Video-Language
    Understanding
paper_summary: Multimodal Large Language Models (MLLMs) have shown promising progress in
  understanding and analyzing video content. However, processing long videos
  remains a significant challenge constrained by LLM's context size. To address
  this limitation, we propose LongVU, a spatiotemporal adaptive compression
  mechanism thats reduces the number of video tokens while preserving visual
  details of long videos. Our idea is based on leveraging cross-modal query and
  inter-frame dependencies to adaptively reduce temporal and spatial redundancy
  in videos. Specifically, we leverage DINOv2 features to remove redundant frames
  that exhibit high similarity. Then we utilize text-guided cross-modal query for
  selective frame feature reduction. Further, we perform spatial token reduction
  across frames based on their temporal dependencies. Our adaptive compression
  strategy effectively processes a large number of frames with little visual
  information loss within given context length. Our LongVU consistently surpass
  existing methods across a variety of video understanding benchmarks, especially
  on hour-long video understanding tasks such as VideoMME and MLVU. Given a
  light-weight LLM, our LongVU also scales effectively into a smaller size with
  state-of-the-art video understanding performance.
paper_ai_summary: LongVU is a spatiotemporal adaptive compression mechanism that enhances video understanding by reducing token redundancy while preserving visual details, enabling long video processing within the context size constraints of large language models.
paper_ai_keywords: Multimodal Large Language Models video tokens DINOv2 cross-modal query temporal redundancy spatial token reduction VideoMME MLVU