title: VideoLLaMB: Long-context Video Understanding with Recurrent Memory
    Bridges
summary: Recent advancements in large-scale video-language models have shown
  significant potential for real-time planning and detailed interactions.
  However, their high computational demands and the scarcity of annotated
  datasets limit their practicality for academic researchers. In this work, we
  introduce VideoLLaMB, a novel framework that utilizes temporal memory tokens
  within bridge layers to allow for the encoding of entire video sequences
  alongside historical visual data, effectively preserving semantic continuity
  and enhancing model performance across various tasks. This approach includes
  recurrent memory tokens and a SceneTilling algorithm, which segments videos
  into independent semantic units to preserve semantic integrity. Empirically,
  VideoLLaMB significantly outstrips existing video-language models,
  demonstrating a 5.5 points improvement over its competitors across three
  VideoQA benchmarks, and 2.06 points on egocentric planning. Comprehensive
  results on the MVBench show that VideoLLaMB-7B achieves markedly better results
  than previous 7B models of same LLM. Remarkably, it maintains robust
  performance as PLLaVA even as video length increases up to 8 times. Besides,
  the frame retrieval results on our specialized Needle in a Video Haystack
  (NIAVH) benchmark, further validate VideoLLaMB's prowess in accurately
  identifying specific frames within lengthy videos. Our SceneTilling algorithm
  also enables the generation of streaming video captions directly, without
  necessitating additional training. In terms of efficiency, VideoLLaMB, trained
  on 16 frames, supports up to 320 frames on a single Nvidia A100 GPU with linear
  GPU memory scaling, ensuring both high performance and cost-effectiveness,
  thereby setting a new foundation for long-form video-language models in both
  academic and practical applications.
paper_authors._id: 66d7d91fa2ddade9371e0b28
paper_authors.name: Yuxuan Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-04T07:25:24.968Z
paper_authors._id: 60b9e6837946aff342f734ae
paper_authors.avatarUrl: /avatars/a711a6aa35757dfd7b78b26098a964fc.svg
paper_authors.fullname: Yuxuan Wang
paper_authors.type: user
paper_authors.user: ColorfulAI
paper_authors._id: 66d7d91fa2ddade9371e0b29
paper_authors.name: Cihang Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:23:42.394Z
paper_authors._id: 645eb61da3c5cd8a16efffff
paper_authors.avatarUrl: /avatars/9112bfeed598dfabf9e077e69e09ecc9.svg
paper_authors.fullname: Cihang Xie
paper_authors.type: user
paper_authors.user: cihangxie
paper_authors._id: 66d7d91fa2ddade9371e0b2a
paper_authors.name: Yang Liu
paper_authors._id: 66d7d91fa2ddade9371e0b2b
paper_authors.name: Zilong Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T07:14:48.519Z
paper_authors._id: 63a95a6a7930fa8c7dd63d4e
paper_authors.avatarUrl: /avatars/d9d0420f7ddfe2f3a7e029fb05f1c89f.svg
paper_authors.fullname: Zilong Zheng
paper_authors.type: user
paper_authors.user: zlzheng
paper_title: VideoLLaMB: Long-context Video Understanding with Recurrent Memory
    Bridges
paper_summary: Recent advancements in large-scale video-language models have shown
  significant potential for real-time planning and detailed interactions.
  However, their high computational demands and the scarcity of annotated
  datasets limit their practicality for academic researchers. In this work, we
  introduce VideoLLaMB, a novel framework that utilizes temporal memory tokens
  within bridge layers to allow for the encoding of entire video sequences
  alongside historical visual data, effectively preserving semantic continuity
  and enhancing model performance across various tasks. This approach includes
  recurrent memory tokens and a SceneTilling algorithm, which segments videos
  into independent semantic units to preserve semantic integrity. Empirically,
  VideoLLaMB significantly outstrips existing video-language models,
  demonstrating a 5.5 points improvement over its competitors across three
  VideoQA benchmarks, and 2.06 points on egocentric planning. Comprehensive
  results on the MVBench show that VideoLLaMB-7B achieves markedly better results
  than previous 7B models of same LLM. Remarkably, it maintains robust
  performance as PLLaVA even as video length increases up to 8 times. Besides,
  the frame retrieval results on our specialized Needle in a Video Haystack
  (NIAVH) benchmark, further validate VideoLLaMB's prowess in accurately
  identifying specific frames within lengthy videos. Our SceneTilling algorithm
  also enables the generation of streaming video captions directly, without
  necessitating additional training. In terms of efficiency, VideoLLaMB, trained
  on 16 frames, supports up to 320 frames on a single Nvidia A100 GPU with linear
  GPU memory scaling, ensuring both high performance and cost-effectiveness,
  thereby setting a new foundation for long-form video-language models in both
  academic and practical applications.
paper_ai_summary: VideoLLaMB enhances video-language model performance through temporal memory tokens and SceneTilling algorithm, enabling accurate frame retrieval and efficient video caption generation without additional training.
paper_ai_keywords: VideoLLaMB temporal memory tokens bridge layers SceneTilling algorithm VideoQA benchmarks egocentric planning MVBench VideoLLaMB-7B PLLaVA Needle in a Video Haystack (NIAVH) benchmark GPU memory scaling