SALOVA: Segment-Augmented Long Video Assistant for Targeted Retrieval
  and Routing in Long-Form Video Analysis
Despite advances in Large Multi-modal Models, applying them to long and
untrimmed video content remains challenging due to limitations in context
length and substantial memory overhead. These constraints often lead to
significant information loss and reduced relevance in the model responses. With
the exponential growth of video data across web platforms, understanding
long-form video is crucial for advancing generalized intelligence. In this
paper, we introduce SALOVA: Segment-Augmented LOng Video Assistant, a novel
video-LLM framework designed to enhance the comprehension of lengthy video
content through targeted retrieval process. We address two main challenges to
achieve it: (i) We present the SceneWalk dataset, a high-quality collection of
87.8K long videos, each densely captioned at the segment level to enable models
to capture scene continuity and maintain rich descriptive context. (ii) We
develop robust architectural designs integrating dynamic routing mechanism and
spatio-temporal projector to efficiently retrieve and process relevant video
segments based on user queries. Our framework mitigates the limitations of
current video-LMMs by allowing for precise identification and retrieval of
relevant video segments in response to queries, thereby improving the
contextual relevance of the generated responses. Through extensive experiments,
SALOVA demonstrates enhanced capability in processing complex long-form videos,
showing significant capability to maintain contextual integrity across extended
sequences.
6746a1f0844d4bb7f7b79703
Junho Kim
claimed_verified
2024-11-27T22:20:02.020Z
653238bed0f5a9e537ed966d
/avatars/e97a83e68e770baa1c5df847777cf213.svg
Junho Kim
user
arkimjh
6746a1f0844d4bb7f7b79704
Hyunjun Kim
claimed_verified
2024-12-03T11:16:34.008Z
666419f8d8e1e34bc62166aa
/avatars/9c419003788682800f0dced2f5b52f46.svg
Hyunjun Kim
user
kimhj709
6746a1f0844d4bb7f7b79705
Hosu Lee
claimed_verified
2024-12-02T09:34:00.366Z
673060959e631f353ae1b5e0
/avatars/d4b1e23de90ff1d02c38186a259b8d1e.svg
Hosu Lee
user
lakelee
6746a1f0844d4bb7f7b79706
Yong Man Ro
SALOVA: Segment-Augmented Long Video Assistant for Targeted Retrieval
  and Routing in Long-Form Video Analysis
Despite advances in Large Multi-modal Models, applying them to long and
untrimmed video content remains challenging due to limitations in context
length and substantial memory overhead. These constraints often lead to
significant information loss and reduced relevance in the model responses. With
the exponential growth of video data across web platforms, understanding
long-form video is crucial for advancing generalized intelligence. In this
paper, we introduce SALOVA: Segment-Augmented LOng Video Assistant, a novel
video-LLM framework designed to enhance the comprehension of lengthy video
content through targeted retrieval process. We address two main challenges to
achieve it: (i) We present the SceneWalk dataset, a high-quality collection of
87.8K long videos, each densely captioned at the segment level to enable models
to capture scene continuity and maintain rich descriptive context. (ii) We
develop robust architectural designs integrating dynamic routing mechanism and
spatio-temporal projector to efficiently retrieve and process relevant video
segments based on user queries. Our framework mitigates the limitations of
current video-LMMs by allowing for precise identification and retrieval of
relevant video segments in response to queries, thereby improving the
contextual relevance of the generated responses. Through extensive experiments,
SALOVA demonstrates enhanced capability in processing complex long-form videos,
showing significant capability to maintain contextual integrity across extended
sequences.
SALOVA, a video-LLM framework, addresses limitations in processing long-form videos through a SceneWalk dataset and robust architectural designs that enable precise segment retrieval and enhanced contextual relevance.
Segment-Augmented LOng Video Assistant video-LLM SceneWalk dataset dynamic routing mechanism spatio-temporal projector contextual relevance context length memory overhead scene continuity descriptive context video segments complex long-form videos