title: Flash-VStream: Memory-Based Real-Time Understanding for Long Video
    Streams
summary: Benefiting from the advancements in large language models and cross-modal
  alignment, existing multi-modal video understanding methods have achieved
  prominent performance in offline scenario. However, online video streams, as
  one of the most common media forms in the real world, have seldom received
  attention. Compared to offline videos, the 'dynamic' nature of online video
  streams poses challenges for the direct application of existing models and
  introduces new problems, such as the storage of extremely long-term
  information, interaction between continuous visual content and 'asynchronous'
  user questions. Therefore, in this paper we present Flash-VStream, a
  video-language model that simulates the memory mechanism of human. Our model is
  able to process extremely long video streams in real-time and respond to user
  queries simultaneously. Compared to existing models, Flash-VStream achieves
  significant reductions in inference latency and VRAM consumption, which is
  intimately related to performing understanding of online streaming video. In
  addition, given that existing video understanding benchmarks predominantly
  concentrate on offline scenario, we propose VStream-QA, a novel question
  answering benchmark specifically designed for online video streaming
  understanding. Comparisons with popular existing methods on the proposed
  benchmark demonstrate the superiority of our method for such challenging
  setting. To verify the generalizability of our approach, we further evaluate it
  on existing video understanding benchmarks and achieves state-of-the-art
  performance in offline scenarios as well. All code, models, and datasets are
  available at the https://invinciblewyq.github.io/vstream-page/
paper_authors._id: 666aa4f277d119acc29d4ba0
paper_authors.name: Haoji Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-08T21:14:08.479Z
paper_authors._id: 65253da6a895f70381f35444
paper_authors.avatarUrl: /avatars/e5cc8645e949d071887e89af37d1e3a3.svg
paper_authors.fullname: Haoji Zhang
paper_authors.type: user
paper_authors.user: zhang9302002
paper_authors._id: 666aa4f277d119acc29d4ba1
paper_authors.name: Yiqin Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-08T21:14:15.995Z
paper_authors._id: 63e065ae42591dda0b9b74cc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63e065ae42591dda0b9b74cc/U1zhG8saFcEo8bNewxugP.jpeg
paper_authors.fullname: Yiqin Wang
paper_authors.type: user
paper_authors.user: Yiqin
paper_authors._id: 666aa4f277d119acc29d4ba2
paper_authors.name: Yansong Tang
paper_authors._id: 666aa4f277d119acc29d4ba3
paper_authors.name: Yong Liu
paper_authors._id: 666aa4f277d119acc29d4ba4
paper_authors.name: Jiashi Feng
paper_authors._id: 666aa4f277d119acc29d4ba5
paper_authors.name: Jifeng Dai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-08T21:14:44.589Z
paper_authors._id: 64686f7172d9180d4ac8b4e4
paper_authors.avatarUrl: /avatars/db67dd6c4b2b41054ddcce5a18ade6f8.svg
paper_authors.fullname: Jifeng Dai
paper_authors.type: user
paper_authors.user: daijifeng
paper_authors._id: 666aa4f277d119acc29d4ba6
paper_authors.name: Xiaojie Jin
paper_title: Flash-VStream: Memory-Based Real-Time Understanding for Long Video
    Streams
paper_summary: Benefiting from the advancements in large language models and cross-modal
  alignment, existing multi-modal video understanding methods have achieved
  prominent performance in offline scenario. However, online video streams, as
  one of the most common media forms in the real world, have seldom received
  attention. Compared to offline videos, the 'dynamic' nature of online video
  streams poses challenges for the direct application of existing models and
  introduces new problems, such as the storage of extremely long-term
  information, interaction between continuous visual content and 'asynchronous'
  user questions. Therefore, in this paper we present Flash-VStream, a
  video-language model that simulates the memory mechanism of human. Our model is
  able to process extremely long video streams in real-time and respond to user
  queries simultaneously. Compared to existing models, Flash-VStream achieves
  significant reductions in inference latency and VRAM consumption, which is
  intimately related to performing understanding of online streaming video. In
  addition, given that existing video understanding benchmarks predominantly
  concentrate on offline scenario, we propose VStream-QA, a novel question
  answering benchmark specifically designed for online video streaming
  understanding. Comparisons with popular existing methods on the proposed
  benchmark demonstrate the superiority of our method for such challenging
  setting. To verify the generalizability of our approach, we further evaluate it
  on existing video understanding benchmarks and achieves state-of-the-art
  performance in offline scenarios as well. All code, models, and datasets are
  available at the https://invinciblewyq.github.io/vstream-page/
paper_ai_summary: Flash-VStream is a video-language model that efficiently processes online video streams in real-time and responds to user queries, achieving superior performance compared to existing methods on both online and offline video understanding benchmarks.
paper_ai_keywords: large language models cross-modal alignment online video streams dynamic nature storage asynchronous user questions Flash-VStream memory mechanism real-time processing inference latency VRAM consumption understanding of online streaming video VStream-QA question answering benchmark offline scenario video understanding benchmarks state-of-the-art performance