TRACE: Temporal Grounding Video LLM via Causal Event Modeling
Video Temporal Grounding (VTG) is a crucial capability for video
understanding models and plays a vital role in downstream tasks such as video
browsing and editing. To effectively handle various tasks simultaneously and
enable zero-shot prediction, there is a growing trend in employing video LLMs
for VTG tasks. However, current video LLM-based methods rely exclusively on
natural language generation, lacking the ability to model the clear structure
inherent in videos, which restricts their effectiveness in tackling VTG tasks.
To address this issue, this paper first formally introduces causal event
modeling framework, which represents videos as sequences of events, and predict
the current event using previous events, video inputs, and textural
instructions. Each event consists of three components: timestamps, salient
scores, and textual captions. We then propose a novel task-interleaved video
LLM called TRACE to effectively implement the causal event modeling framework
in practice. The TRACE processes visual frames, timestamps, salient scores, and
text as distinct tasks, employing various encoders and decoding heads for each.
Task tokens are arranged in an interleaved sequence according to the causal
event modeling framework's formulation. Extensive experiments on various VTG
tasks and datasets demonstrate the superior performance of TRACE compared to
state-of-the-art video LLMs. Our model and code are available at
https://github.com/gyxxyg/TRACE.
67073b83d4311c2dd68fe1f1
Yongxin Guo
claimed_verified
2024-10-10T08:01:22.374Z
652c9bdf4d0eef4a2b3c577f
/avatars/6e92eee19ae2c43e46a8b294c02f9915.svg
Yongxin Guo
user
Yongxin-Guo
67073b83d4311c2dd68fe1f2
Jingyu Liu
67073b83d4311c2dd68fe1f3
Mingda Li
67073b83d4311c2dd68fe1f4
Xiaoying Tang
67073b83d4311c2dd68fe1f5
Qingbin Liu
admin_assigned
2024-10-10T09:16:10.450Z
6707481116abdf6976c68421
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/yEoJCremrGNoZrnI-xOOU.png
qingbinliu
user
qingbinliu
67073b83d4311c2dd68fe1f6
Xi Chen
TRACE: Temporal Grounding Video LLM via Causal Event Modeling
Video Temporal Grounding (VTG) is a crucial capability for video
understanding models and plays a vital role in downstream tasks such as video
browsing and editing. To effectively handle various tasks simultaneously and
enable zero-shot prediction, there is a growing trend in employing video LLMs
for VTG tasks. However, current video LLM-based methods rely exclusively on
natural language generation, lacking the ability to model the clear structure
inherent in videos, which restricts their effectiveness in tackling VTG tasks.
To address this issue, this paper first formally introduces causal event
modeling framework, which represents videos as sequences of events, and predict
the current event using previous events, video inputs, and textural
instructions. Each event consists of three components: timestamps, salient
scores, and textual captions. We then propose a novel task-interleaved video
LLM called TRACE to effectively implement the causal event modeling framework
in practice. The TRACE processes visual frames, timestamps, salient scores, and
text as distinct tasks, employing various encoders and decoding heads for each.
Task tokens are arranged in an interleaved sequence according to the causal
event modeling framework's formulation. Extensive experiments on various VTG
tasks and datasets demonstrate the superior performance of TRACE compared to
state-of-the-art video LLMs. Our model and code are available at
https://github.com/gyxxyg/TRACE.
A novel task-interleaved video LLM called TRACE improves video temporal grounding by using a causal event modeling framework to represent videos as sequences of structured events, enhancing performance over existing methods.
video LLMs Video Temporal Grounding causal event modeling framework task-interleaved video LLM TRACE visual frames timestamps salient scores textual captions task tokens