title: An LMM for Efficient Video Understanding via Reinforced Compression of
    Video Cubes
summary: Large Multimodal Models (LMMs) uniformly perceive video frames, creating
  computational inefficiency for videos with inherently varying temporal
  information density. This paper present Quicksviewer, an LMM with new
  perceiving paradigm that partitions a video of nonuniform density into varying
  cubes using Gumbel Softmax, followed by a unified resampling for each cube to
  achieve efficient video understanding. This simple and intuitive approach
  dynamically compress video online based on its temporal density, significantly
  reducing spatiotemporal redundancy (overall 45times compression rate), while
  enabling efficient training with large receptive field. We train the model from
  a language backbone through three progressive stages, each incorporating
  lengthy videos on average of 420s/1fps thanks to the perceiving efficiency.
  With only 0.8M total video-text samples for training, our model outperforms the
  direct baseline employing a fixed partitioning strategy by a maximum of 8.72 in
  accuracy, demonstrating the effectiveness in performance. On Video-MME,
  Quicksviewer achieves SOTA under modest sequence lengths using just up to 5\%
  of tokens per frame required by baselines. With this paradigm, scaling up the
  number of input frames reveals a clear power law of the model capabilities. It
  is also empirically verified that the segments generated by the cubing network
  can help for analyzing continuous events in videos.
paper_authors._id: 68074ea84a1c690663814e56
paper_authors.name: Ji Qi
paper_authors._id: 68074ea84a1c690663814e57
paper_authors.name: Yuan Yao
paper_authors._id: 68074ea84a1c690663814e58
paper_authors.name: Yushi Bai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-22T11:05:19.003Z
paper_authors._id: 64ed568ccf6118a9379a61b8
paper_authors.avatarUrl: /avatars/6d040cbcb4a9b624cbe64c9d01cd5c88.svg
paper_authors.fullname: Yushi Bai
paper_authors.type: user
paper_authors.user: bys0318
paper_authors._id: 68074ea84a1c690663814e59
paper_authors.name: Bin Xu
paper_authors._id: 68074ea84a1c690663814e5a
paper_authors.name: Juanzi Li
paper_authors._id: 68074ea84a1c690663814e5b
paper_authors.name: Zhiyuan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-22T11:06:02.057Z
paper_authors._id: 6310a3cd531cc21f9e06de6a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6310a3cd531cc21f9e06de6a/aTGMx3O41lUARK9s3dAik.jpeg
paper_authors.fullname: Zhiyuan Liu
paper_authors.type: user
paper_authors.user: acharkq
paper_authors._id: 68074ea84a1c690663814e5c
paper_authors.name: Tat-Seng Chua
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-22T11:05:41.211Z
paper_authors._id: 6570ae84c4993b8fb96f41a8
paper_authors.avatarUrl: /avatars/21f7d79d46ac4df0ecff8eca7678b33f.svg
paper_authors.fullname: Tat-Seng Chua
paper_authors.type: user
paper_authors.user: chuats
paper_title: An LMM for Efficient Video Understanding via Reinforced Compression of
    Video Cubes
paper_summary: Large Multimodal Models (LMMs) uniformly perceive video frames, creating
  computational inefficiency for videos with inherently varying temporal
  information density. This paper present Quicksviewer, an LMM with new
  perceiving paradigm that partitions a video of nonuniform density into varying
  cubes using Gumbel Softmax, followed by a unified resampling for each cube to
  achieve efficient video understanding. This simple and intuitive approach
  dynamically compress video online based on its temporal density, significantly
  reducing spatiotemporal redundancy (overall 45times compression rate), while
  enabling efficient training with large receptive field. We train the model from
  a language backbone through three progressive stages, each incorporating
  lengthy videos on average of 420s/1fps thanks to the perceiving efficiency.
  With only 0.8M total video-text samples for training, our model outperforms the
  direct baseline employing a fixed partitioning strategy by a maximum of 8.72 in
  accuracy, demonstrating the effectiveness in performance. On Video-MME,
  Quicksviewer achieves SOTA under modest sequence lengths using just up to 5\%
  of tokens per frame required by baselines. With this paradigm, scaling up the
  number of input frames reveals a clear power law of the model capabilities. It
  is also empirically verified that the segments generated by the cubing network
  can help for analyzing continuous events in videos.
paper_ai_summary: Quicksviewer, a Large Multimodal Model using Gumbel Softmax for dynamic video partitioning, achieves efficient video understanding and outperforms fixed partitioning by reducing temporal redundancy.
paper_ai_keywords: Large Multimodal Models (LMMs) video frames computational efficiency temporal information density Gumbel Softmax spatiotemporal redundancy receptive field language backbone progressive stages video-MME token efficiency cubing network continuous event analysis