Token-Efficient Long Video Understanding for Multimodal LLMs
Recent advances in video-based multimodal large language models (Video-LLMs)
have significantly improved video understanding by processing videos as
sequences of image frames. However, many existing methods treat frames
independently in the vision backbone, lacking explicit temporal modeling, which
limits their ability to capture dynamic patterns and efficiently handle long
videos. To address these limitations, we introduce STORM
(Spatiotemporal TOken Reduction for
Multimodal LLMs), a novel architecture incorporating a dedicated
temporal encoder between the image encoder and the LLM. Our temporal encoder
leverages the Mamba State Space Model to integrate temporal information into
image tokens, generating enriched representations that preserve inter-frame
dynamics across the entire video sequence. This enriched encoding not only
enhances video reasoning capabilities but also enables effective token
reduction strategies, including test-time sampling and training-based temporal
and spatial pooling, substantially reducing computational demands on the LLM
without sacrificing key temporal information. By integrating these techniques,
our approach simultaneously reduces training and inference latency while
improving performance, enabling efficient and robust video understanding over
extended temporal contexts. Extensive evaluations show that STORM achieves
state-of-the-art results across various long video understanding benchmarks
(more than 5\% improvement on MLVU and LongVideoBench) while reducing the
computation costs by up to 8times and the decoding latency by
2.4-2.9times for the fixed numbers of input frames. Project page is
available at https://research.nvidia.com/labs/lpr/storm
67ca7baf6d5c2eafede56d35
Jindong Jiang
admin_assigned
2025-03-07T09:34:59.750Z
6449e5a3df4e6cb7eaefd2b8
/avatars/a671cb507d5e02b238d8cd631e71649d.svg
Jindong Jiang
user
jdps
67ca7baf6d5c2eafede56d36
Xiuyu Li
admin_assigned
2025-03-07T09:35:06.739Z
644570ba2d91b15b4c7f6311
/avatars/d5e66012066d0c330b8f23718b1499d8.svg
Xiuyu Li
user
xiuyul
67ca7baf6d5c2eafede56d37
Zhijian Liu
admin_assigned
2025-03-07T09:35:13.834Z
650dac79b959b0e1d41d7378
https://cdn-avatars.huggingface.co/v1/production/uploads/650dac79b959b0e1d41d7378/mzbN0MFk3k8b94FQ40I7L.jpeg
Zhijian Liu
user
zhijianliu
67ca7baf6d5c2eafede56d38
Muyang Li
admin_assigned
2025-03-07T09:37:06.962Z
66d8b322cf789857d384e5c4
/avatars/1276726b27d312f48e69f5ae982daa24.svg
Muyang Li
user
MuyangLI
67ca7baf6d5c2eafede56d39
Guo Chen
claimed_verified
2025-04-22T10:47:19.477Z
6392c73390b8e99a6779a7b0
/avatars/9ff824ab02848120aec5e8de6780bcf1.svg
Guo Chen
user
cg1177
67ca7baf6d5c2eafede56d3a
Zhiqi Li
admin_assigned
2025-03-07T09:36:57.283Z
672aae13b2f2dc21e18570e0
/avatars/0253107a2116d197dc0fe18660c2af90.svg
Zhiqi Li
user
zhiqilinv
67ca7baf6d5c2eafede56d3b
De-An Huang
admin_assigned
2025-03-07T09:36:27.913Z
641d1c5ec3983aa94915c162
/avatars/127985b837ecf61e43c835deee578b5e.svg
De-An Huang
user
deahuang
67ca7baf6d5c2eafede56d3c
Guilin Liu
admin_assigned
2025-03-07T09:36:17.904Z
6656eb16e50d7c40881a14f0
/avatars/c6822a51c8d5918debf6ee1d25fe1825.svg
GuilinLiu
user
GuilinLiu
67ca7baf6d5c2eafede56d3d
Zhiding Yu
admin_assigned
2025-03-07T09:36:09.646Z
66c8037c737ba92ae3fe0322
https://cdn-avatars.huggingface.co/v1/production/uploads/66c8037c737ba92ae3fe0322/WR_Yh5DWOVVh7IFlF24NM.jpeg
Zhiding Yu
user
Zhiding
67ca7baf6d5c2eafede56d3e
Kurt Keutzer
admin_assigned
2025-03-07T09:36:02.351Z
6251bf4b183aa4266924ad91
https://cdn-avatars.huggingface.co/v1/production/uploads/1678041834400-6251bf4b183aa4266924ad91.jpeg
Kurt Keutzer
user
kurtkeutzer
67ca7baf6d5c2eafede56d3f
Sungjin Ahn
67ca7baf6d5c2eafede56d40
Jan Kautz
67ca7baf6d5c2eafede56d41
Hongxu Yin
admin_assigned
2025-03-07T09:35:40.242Z
65a8b7f69aec1645994e7a15
/avatars/debc086f3fea029db22847bde80799a0.svg
Hongxu Yin
user
yinhongxu
67ca7baf6d5c2eafede56d42
Yao Lu
67ca7baf6d5c2eafede56d43
Song Han
67ca7baf6d5c2eafede56d44
Wonmin Byeon
admin_assigned
2025-03-07T09:35:32.527Z
66bf958296583c59b049085b
/avatars/04df8dd45835b7ea0991e242784e7810.svg
Wonmin Byeon
user
wbyeon
Token-Efficient Long Video Understanding for Multimodal LLMs
Recent advances in video-based multimodal large language models (Video-LLMs)
have significantly improved video understanding by processing videos as
sequences of image frames. However, many existing methods treat frames
independently in the vision backbone, lacking explicit temporal modeling, which
limits their ability to capture dynamic patterns and efficiently handle long
videos. To address these limitations, we introduce STORM
(Spatiotemporal TOken Reduction for
Multimodal LLMs), a novel architecture incorporating a dedicated
temporal encoder between the image encoder and the LLM. Our temporal encoder
leverages the Mamba State Space Model to integrate temporal information into
image tokens, generating enriched representations that preserve inter-frame
dynamics across the entire video sequence. This enriched encoding not only
enhances video reasoning capabilities but also enables effective token
reduction strategies, including test-time sampling and training-based temporal
and spatial pooling, substantially reducing computational demands on the LLM
without sacrificing key temporal information. By integrating these techniques,
our approach simultaneously reduces training and inference latency while
improving performance, enabling efficient and robust video understanding over
extended temporal contexts. Extensive evaluations show that STORM achieves
state-of-the-art results across various long video understanding benchmarks
(more than 5\% improvement on MLVU and LongVideoBench) while reducing the
computation costs by up to 8times and the decoding latency by
2.4-2.9times for the fixed numbers of input frames. Project page is
available at https://research.nvidia.com/labs/lpr/storm
STORM integrates a temporal encoder using the Mamba State Space Model to enhance video understanding and reduce computational costs in multimodal large language models.
Spatiotemporal Token Reduction temporal encoder Mamba State Space Model image tokens inter-frame dynamics token reduction test-time sampling training-based temporal and spatial pooling video reasoning video understanding MLVU LongVideoBench