title: TRACE: Temporal Grounding Video LLM via Causal Event Modeling
summary: Video Temporal Grounding (VTG) is a crucial capability for video
  understanding models and plays a vital role in downstream tasks such as video
  browsing and editing. To effectively handle various tasks simultaneously and
  enable zero-shot prediction, there is a growing trend in employing video LLMs
  for VTG tasks. However, current video LLM-based methods rely exclusively on
  natural language generation, lacking the ability to model the clear structure
  inherent in videos, which restricts their effectiveness in tackling VTG tasks.
  To address this issue, this paper first formally introduces causal event
  modeling framework, which represents videos as sequences of events, and predict
  the current event using previous events, video inputs, and textural
  instructions. Each event consists of three components: timestamps, salient
  scores, and textual captions. We then propose a novel task-interleaved video
  LLM called TRACE to effectively implement the causal event modeling framework
  in practice. The TRACE processes visual frames, timestamps, salient scores, and
  text as distinct tasks, employing various encoders and decoding heads for each.
  Task tokens are arranged in an interleaved sequence according to the causal
  event modeling framework's formulation. Extensive experiments on various VTG
  tasks and datasets demonstrate the superior performance of TRACE compared to
  state-of-the-art video LLMs. Our model and code are available at
  https://github.com/gyxxyg/TRACE.
paper_authors._id: 67073b83d4311c2dd68fe1f1
paper_authors.name: Yongxin Guo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-10T08:01:22.374Z
paper_authors._id: 652c9bdf4d0eef4a2b3c577f
paper_authors.avatarUrl: /avatars/6e92eee19ae2c43e46a8b294c02f9915.svg
paper_authors.fullname: Yongxin Guo
paper_authors.type: user
paper_authors.user: Yongxin-Guo
paper_authors._id: 67073b83d4311c2dd68fe1f2
paper_authors.name: Jingyu Liu
paper_authors._id: 67073b83d4311c2dd68fe1f3
paper_authors.name: Mingda Li
paper_authors._id: 67073b83d4311c2dd68fe1f4
paper_authors.name: Xiaoying Tang
paper_authors._id: 67073b83d4311c2dd68fe1f5
paper_authors.name: Qingbin Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:16:10.450Z
paper_authors._id: 6707481116abdf6976c68421
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/yEoJCremrGNoZrnI-xOOU.png
paper_authors.fullname: qingbinliu
paper_authors.type: user
paper_authors.user: qingbinliu
paper_authors._id: 67073b83d4311c2dd68fe1f6
paper_authors.name: Xi Chen
paper_title: TRACE: Temporal Grounding Video LLM via Causal Event Modeling
paper_summary: Video Temporal Grounding (VTG) is a crucial capability for video
  understanding models and plays a vital role in downstream tasks such as video
  browsing and editing. To effectively handle various tasks simultaneously and
  enable zero-shot prediction, there is a growing trend in employing video LLMs
  for VTG tasks. However, current video LLM-based methods rely exclusively on
  natural language generation, lacking the ability to model the clear structure
  inherent in videos, which restricts their effectiveness in tackling VTG tasks.
  To address this issue, this paper first formally introduces causal event
  modeling framework, which represents videos as sequences of events, and predict
  the current event using previous events, video inputs, and textural
  instructions. Each event consists of three components: timestamps, salient
  scores, and textual captions. We then propose a novel task-interleaved video
  LLM called TRACE to effectively implement the causal event modeling framework
  in practice. The TRACE processes visual frames, timestamps, salient scores, and
  text as distinct tasks, employing various encoders and decoding heads for each.
  Task tokens are arranged in an interleaved sequence according to the causal
  event modeling framework's formulation. Extensive experiments on various VTG
  tasks and datasets demonstrate the superior performance of TRACE compared to
  state-of-the-art video LLMs. Our model and code are available at
  https://github.com/gyxxyg/TRACE.
paper_ai_summary: A novel task-interleaved video LLM called TRACE improves video temporal grounding by using a causal event modeling framework to represent videos as sequences of structured events, enhancing performance over existing methods.
paper_ai_keywords: video LLMs Video Temporal Grounding causal event modeling framework task-interleaved video LLM TRACE visual frames timestamps salient scores textual captions task tokens