title: Token-Efficient Long Video Understanding for Multimodal LLMs
summary: Recent advances in video-based multimodal large language models (Video-LLMs)
  have significantly improved video understanding by processing videos as
  sequences of image frames. However, many existing methods treat frames
  independently in the vision backbone, lacking explicit temporal modeling, which
  limits their ability to capture dynamic patterns and efficiently handle long
  videos. To address these limitations, we introduce STORM
  (Spatiotemporal TOken Reduction for
  Multimodal LLMs), a novel architecture incorporating a dedicated
  temporal encoder between the image encoder and the LLM. Our temporal encoder
  leverages the Mamba State Space Model to integrate temporal information into
  image tokens, generating enriched representations that preserve inter-frame
  dynamics across the entire video sequence. This enriched encoding not only
  enhances video reasoning capabilities but also enables effective token
  reduction strategies, including test-time sampling and training-based temporal
  and spatial pooling, substantially reducing computational demands on the LLM
  without sacrificing key temporal information. By integrating these techniques,
  our approach simultaneously reduces training and inference latency while
  improving performance, enabling efficient and robust video understanding over
  extended temporal contexts. Extensive evaluations show that STORM achieves
  state-of-the-art results across various long video understanding benchmarks
  (more than 5\% improvement on MLVU and LongVideoBench) while reducing the
  computation costs by up to 8times and the decoding latency by
  2.4-2.9times for the fixed numbers of input frames. Project page is
  available at https://research.nvidia.com/labs/lpr/storm
paper_authors._id: 67ca7baf6d5c2eafede56d35
paper_authors.name: Jindong Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:34:59.750Z
paper_authors._id: 6449e5a3df4e6cb7eaefd2b8
paper_authors.avatarUrl: /avatars/a671cb507d5e02b238d8cd631e71649d.svg
paper_authors.fullname: Jindong Jiang
paper_authors.type: user
paper_authors.user: jdps
paper_authors._id: 67ca7baf6d5c2eafede56d36
paper_authors.name: Xiuyu Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:35:06.739Z
paper_authors._id: 644570ba2d91b15b4c7f6311
paper_authors.avatarUrl: /avatars/d5e66012066d0c330b8f23718b1499d8.svg
paper_authors.fullname: Xiuyu Li
paper_authors.type: user
paper_authors.user: xiuyul
paper_authors._id: 67ca7baf6d5c2eafede56d37
paper_authors.name: Zhijian Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:35:13.834Z
paper_authors._id: 650dac79b959b0e1d41d7378
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/650dac79b959b0e1d41d7378/mzbN0MFk3k8b94FQ40I7L.jpeg
paper_authors.fullname: Zhijian Liu
paper_authors.type: user
paper_authors.user: zhijianliu
paper_authors._id: 67ca7baf6d5c2eafede56d38
paper_authors.name: Muyang Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:37:06.962Z
paper_authors._id: 66d8b322cf789857d384e5c4
paper_authors.avatarUrl: /avatars/1276726b27d312f48e69f5ae982daa24.svg
paper_authors.fullname: Muyang Li
paper_authors.type: user
paper_authors.user: MuyangLI
paper_authors._id: 67ca7baf6d5c2eafede56d39
paper_authors.name: Guo Chen
paper_authors._id: 67ca7baf6d5c2eafede56d3a
paper_authors.name: Zhiqi Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:36:57.283Z
paper_authors._id: 672aae13b2f2dc21e18570e0
paper_authors.avatarUrl: /avatars/0253107a2116d197dc0fe18660c2af90.svg
paper_authors.fullname: Zhiqi Li
paper_authors.type: user
paper_authors.user: zhiqilinv
paper_authors._id: 67ca7baf6d5c2eafede56d3b
paper_authors.name: De-An Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:36:27.913Z
paper_authors._id: 641d1c5ec3983aa94915c162
paper_authors.avatarUrl: /avatars/127985b837ecf61e43c835deee578b5e.svg
paper_authors.fullname: De-An Huang
paper_authors.type: user
paper_authors.user: deahuang
paper_authors._id: 67ca7baf6d5c2eafede56d3c
paper_authors.name: Guilin Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:36:17.904Z
paper_authors._id: 6656eb16e50d7c40881a14f0
paper_authors.avatarUrl: /avatars/c6822a51c8d5918debf6ee1d25fe1825.svg
paper_authors.fullname: GuilinLiu
paper_authors.type: user
paper_authors.user: GuilinLiu
paper_authors._id: 67ca7baf6d5c2eafede56d3d
paper_authors.name: Zhiding Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:36:09.646Z
paper_authors._id: 66c8037c737ba92ae3fe0322
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66c8037c737ba92ae3fe0322/WR_Yh5DWOVVh7IFlF24NM.jpeg
paper_authors.fullname: Zhiding Yu
paper_authors.type: user
paper_authors.user: Zhiding
paper_authors._id: 67ca7baf6d5c2eafede56d3e
paper_authors.name: Kurt Keutzer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:36:02.351Z
paper_authors._id: 6251bf4b183aa4266924ad91
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678041834400-6251bf4b183aa4266924ad91.jpeg
paper_authors.fullname: Kurt Keutzer
paper_authors.type: user
paper_authors.user: kurtkeutzer
paper_authors._id: 67ca7baf6d5c2eafede56d3f
paper_authors.name: Sungjin Ahn
paper_authors._id: 67ca7baf6d5c2eafede56d40
paper_authors.name: Jan Kautz
paper_authors._id: 67ca7baf6d5c2eafede56d41
paper_authors.name: Hongxu Yin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:35:40.242Z
paper_authors._id: 65a8b7f69aec1645994e7a15
paper_authors.avatarUrl: /avatars/debc086f3fea029db22847bde80799a0.svg
paper_authors.fullname: Hongxu Yin
paper_authors.type: user
paper_authors.user: yinhongxu
paper_authors._id: 67ca7baf6d5c2eafede56d42
paper_authors.name: Yao Lu
paper_authors._id: 67ca7baf6d5c2eafede56d43
paper_authors.name: Song Han
paper_authors._id: 67ca7baf6d5c2eafede56d44
paper_authors.name: Wonmin Byeon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:35:32.527Z
paper_authors._id: 66bf958296583c59b049085b
paper_authors.avatarUrl: /avatars/04df8dd45835b7ea0991e242784e7810.svg
paper_authors.fullname: Wonmin Byeon
paper_authors.type: user
paper_authors.user: wbyeon
paper_title: Token-Efficient Long Video Understanding for Multimodal LLMs
paper_summary: Recent advances in video-based multimodal large language models (Video-LLMs)
  have significantly improved video understanding by processing videos as
  sequences of image frames. However, many existing methods treat frames
  independently in the vision backbone, lacking explicit temporal modeling, which
  limits their ability to capture dynamic patterns and efficiently handle long
  videos. To address these limitations, we introduce STORM
  (Spatiotemporal TOken Reduction for
  Multimodal LLMs), a novel architecture incorporating a dedicated
  temporal encoder between the image encoder and the LLM. Our temporal encoder
  leverages the Mamba State Space Model to integrate temporal information into
  image tokens, generating enriched representations that preserve inter-frame
  dynamics across the entire video sequence. This enriched encoding not only
  enhances video reasoning capabilities but also enables effective token
  reduction strategies, including test-time sampling and training-based temporal
  and spatial pooling, substantially reducing computational demands on the LLM
  without sacrificing key temporal information. By integrating these techniques,
  our approach simultaneously reduces training and inference latency while
  improving performance, enabling efficient and robust video understanding over
  extended temporal contexts. Extensive evaluations show that STORM achieves
  state-of-the-art results across various long video understanding benchmarks
  (more than 5\% improvement on MLVU and LongVideoBench) while reducing the
  computation costs by up to 8times and the decoding latency by
  2.4-2.9times for the fixed numbers of input frames. Project page is
  available at https://research.nvidia.com/labs/lpr/storm
paper_ai_keywords: Spatiotemporal token reduction Mamba State Space Model Temporal encoder Image encoder Multimodal large language models Video reasoning Token reduction Test-time sampling Training-based temporal pooling Training-based spatial pooling Long video understanding MLVU LongVideoBench