LongVU: Spatiotemporal Adaptive Compression for Long Video-Language
  Understanding
Multimodal Large Language Models (MLLMs) have shown promising progress in
understanding and analyzing video content. However, processing long videos
remains a significant challenge constrained by LLM's context size. To address
this limitation, we propose LongVU, a spatiotemporal adaptive compression
mechanism thats reduces the number of video tokens while preserving visual
details of long videos. Our idea is based on leveraging cross-modal query and
inter-frame dependencies to adaptively reduce temporal and spatial redundancy
in videos. Specifically, we leverage DINOv2 features to remove redundant frames
that exhibit high similarity. Then we utilize text-guided cross-modal query for
selective frame feature reduction. Further, we perform spatial token reduction
across frames based on their temporal dependencies. Our adaptive compression
strategy effectively processes a large number of frames with little visual
information loss within given context length. Our LongVU consistently surpass
existing methods across a variety of video understanding benchmarks, especially
on hour-long video understanding tasks such as VideoMME and MLVU. Given a
light-weight LLM, our LongVU also scales effectively into a smaller size with
state-of-the-art video understanding performance.
6719f9b5248e67e1a920a554
Xiaoqian Shen
claimed_verified
2024-10-25T09:31:01.552Z
620c5b20c33ca69c4c82145b
/avatars/f3b532c47e434be30d1005fe6d0e25fb.svg
Xiaoqian Shen
user
shenxq
6719f9b5248e67e1a920a555
Yunyang Xiong
extracted_confirmed
2024-10-24T08:06:59.406Z
65304b62e7535baecd85d080
/avatars/6e546c7d1414bd92c5a7c8d8c404de92.svg
Yunyang Xiong
user
yunyangx
6719f9b5248e67e1a920a556
Changsheng Zhao
6719f9b5248e67e1a920a557
Lemeng Wu
claimed_verified
2024-10-25T09:30:59.791Z
6364acf56fa33d2f59397c59
/avatars/f642a7dcc098f50fef865f30f93a25c3.svg
Lemeng Wu
user
klight
6719f9b5248e67e1a920a558
Jun Chen
6719f9b5248e67e1a920a559
Chenchen Zhu
6719f9b5248e67e1a920a55a
Zechun Liu
6719f9b5248e67e1a920a55b
Fanyi Xiao
6719f9b5248e67e1a920a55c
Balakrishnan Varadarajan
6719f9b5248e67e1a920a55d
Florian Bordes
claimed_verified
2025-06-12T12:45:01.609Z
63166336894404e2506b8811
/avatars/ec51a1dd86511127cf83afd4bf7c0f52.svg
Florian Bordes
user
Fbordes
6719f9b5248e67e1a920a55e
Zhuang Liu
6719f9b5248e67e1a920a55f
Hu Xu
6719f9b5248e67e1a920a560
Hyunwoo J. Kim
claimed_verified
2025-06-16T15:51:36.371Z
63183a30a760ff8cff318be5
/avatars/1b314afd8d3319423915ffc2ae56d567.svg
Hyunwoo Kim
user
hkimcvml
6719f9b5248e67e1a920a561
Bilge Soran
6719f9b5248e67e1a920a562
Raghuraman Krishnamoorthi
6719f9b5248e67e1a920a563
Mohamed Elhoseiny
6719f9b5248e67e1a920a564
Vikas Chandra
LongVU: Spatiotemporal Adaptive Compression for Long Video-Language
  Understanding
Multimodal Large Language Models (MLLMs) have shown promising progress in
understanding and analyzing video content. However, processing long videos
remains a significant challenge constrained by LLM's context size. To address
this limitation, we propose LongVU, a spatiotemporal adaptive compression
mechanism thats reduces the number of video tokens while preserving visual
details of long videos. Our idea is based on leveraging cross-modal query and
inter-frame dependencies to adaptively reduce temporal and spatial redundancy
in videos. Specifically, we leverage DINOv2 features to remove redundant frames
that exhibit high similarity. Then we utilize text-guided cross-modal query for
selective frame feature reduction. Further, we perform spatial token reduction
across frames based on their temporal dependencies. Our adaptive compression
strategy effectively processes a large number of frames with little visual
information loss within given context length. Our LongVU consistently surpass
existing methods across a variety of video understanding benchmarks, especially
on hour-long video understanding tasks such as VideoMME and MLVU. Given a
light-weight LLM, our LongVU also scales effectively into a smaller size with
state-of-the-art video understanding performance.
LongVU is a spatiotemporal adaptive compression mechanism that enhances video understanding by reducing token redundancy while preserving visual details, enabling long video processing within the context size constraints of large language models.
Multimodal Large Language Models video tokens DINOv2 cross-modal query temporal redundancy spatial token reduction VideoMME MLVU