title: SALOVA: Segment-Augmented Long Video Assistant for Targeted Retrieval
    and Routing in Long-Form Video Analysis
summary: Despite advances in Large Multi-modal Models, applying them to long and
  untrimmed video content remains challenging due to limitations in context
  length and substantial memory overhead. These constraints often lead to
  significant information loss and reduced relevance in the model responses. With
  the exponential growth of video data across web platforms, understanding
  long-form video is crucial for advancing generalized intelligence. In this
  paper, we introduce SALOVA: Segment-Augmented LOng Video Assistant, a novel
  video-LLM framework designed to enhance the comprehension of lengthy video
  content through targeted retrieval process. We address two main challenges to
  achieve it: (i) We present the SceneWalk dataset, a high-quality collection of
  87.8K long videos, each densely captioned at the segment level to enable models
  to capture scene continuity and maintain rich descriptive context. (ii) We
  develop robust architectural designs integrating dynamic routing mechanism and
  spatio-temporal projector to efficiently retrieve and process relevant video
  segments based on user queries. Our framework mitigates the limitations of
  current video-LMMs by allowing for precise identification and retrieval of
  relevant video segments in response to queries, thereby improving the
  contextual relevance of the generated responses. Through extensive experiments,
  SALOVA demonstrates enhanced capability in processing complex long-form videos,
  showing significant capability to maintain contextual integrity across extended
  sequences.
paper_authors._id: 6746a1f0844d4bb7f7b79703
paper_authors.name: Junho Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-27T22:20:02.020Z
paper_authors._id: 653238bed0f5a9e537ed966d
paper_authors.avatarUrl: /avatars/e97a83e68e770baa1c5df847777cf213.svg
paper_authors.fullname: Junho Kim
paper_authors.type: user
paper_authors.user: arkimjh
paper_authors._id: 6746a1f0844d4bb7f7b79704
paper_authors.name: Hyunjun Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-03T11:16:34.008Z
paper_authors._id: 666419f8d8e1e34bc62166aa
paper_authors.avatarUrl: /avatars/9c419003788682800f0dced2f5b52f46.svg
paper_authors.fullname: Hyunjun Kim
paper_authors.type: user
paper_authors.user: kimhj709
paper_authors._id: 6746a1f0844d4bb7f7b79705
paper_authors.name: Hosu Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-02T09:34:00.366Z
paper_authors._id: 673060959e631f353ae1b5e0
paper_authors.avatarUrl: /avatars/d4b1e23de90ff1d02c38186a259b8d1e.svg
paper_authors.fullname: Hosu Lee
paper_authors.type: user
paper_authors.user: lakelee
paper_authors._id: 6746a1f0844d4bb7f7b79706
paper_authors.name: Yong Man Ro
paper_title: SALOVA: Segment-Augmented Long Video Assistant for Targeted Retrieval
    and Routing in Long-Form Video Analysis
paper_summary: Despite advances in Large Multi-modal Models, applying them to long and
  untrimmed video content remains challenging due to limitations in context
  length and substantial memory overhead. These constraints often lead to
  significant information loss and reduced relevance in the model responses. With
  the exponential growth of video data across web platforms, understanding
  long-form video is crucial for advancing generalized intelligence. In this
  paper, we introduce SALOVA: Segment-Augmented LOng Video Assistant, a novel
  video-LLM framework designed to enhance the comprehension of lengthy video
  content through targeted retrieval process. We address two main challenges to
  achieve it: (i) We present the SceneWalk dataset, a high-quality collection of
  87.8K long videos, each densely captioned at the segment level to enable models
  to capture scene continuity and maintain rich descriptive context. (ii) We
  develop robust architectural designs integrating dynamic routing mechanism and
  spatio-temporal projector to efficiently retrieve and process relevant video
  segments based on user queries. Our framework mitigates the limitations of
  current video-LMMs by allowing for precise identification and retrieval of
  relevant video segments in response to queries, thereby improving the
  contextual relevance of the generated responses. Through extensive experiments,
  SALOVA demonstrates enhanced capability in processing complex long-form videos,
  showing significant capability to maintain contextual integrity across extended
  sequences.
paper_ai_keywords: SceneWalks dataset video-LLM framework dynamic routing mechanism spatio-temporal projector