title: MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware
    Multi-Segment Grounding
summary: Video temporal understanding is crucial for multimodal large language models
  (MLLMs) to reason over events in videos. Despite recent advances in general
  video understanding, current MLLMs still struggle with fine-grained temporal
  reasoning. While reinforcement learning (RL) has been explored to address this
  issue recently, existing RL approaches remain limited in effectiveness. In this
  work, we propose MUSEG, a novel RL-based method that enhances temporal
  understanding by introducing timestamp-aware multi-segment grounding. MUSEG
  enables MLLMs to align queries with multiple relevant video segments, promoting
  more comprehensive temporal reasoning. To facilitate effective learning, we
  design a customized RL training recipe with phased rewards that progressively
  guides the model toward temporally grounded reasoning. Extensive experiments on
  temporal grounding and time-sensitive video QA tasks demonstrate that MUSEG
  significantly outperforms existing methods and generalizes well across diverse
  temporal understanding scenarios. View our project at
  https://github.com/THUNLP-MT/MUSEG.
paper_authors._id: 6837eef0e237f02cd3c87963
paper_authors.name: Fuwen Luo
paper_authors._id: 6837eef0e237f02cd3c87964
paper_authors.name: Shengfeng Lou
paper_authors._id: 6837eef0e237f02cd3c87965
paper_authors.name: Chi Chen
paper_authors._id: 6837eef0e237f02cd3c87966
paper_authors.name: Ziyue Wang
paper_authors._id: 6837eef0e237f02cd3c87967
paper_authors.name: Chenliang Li
paper_authors._id: 6837eef0e237f02cd3c87968
paper_authors.name: Weizhou Shen
paper_authors._id: 6837eef0e237f02cd3c87969
paper_authors.name: Jiyue Guo
paper_authors._id: 6837eef0e237f02cd3c8796a
paper_authors.name: Peng Li
paper_authors._id: 6837eef0e237f02cd3c8796b
paper_authors.name: Ming Yan
paper_authors._id: 6837eef0e237f02cd3c8796c
paper_authors.name: Ji Zhang
paper_authors._id: 6837eef0e237f02cd3c8796d
paper_authors.name: Fei Huang
paper_authors._id: 6837eef0e237f02cd3c8796e
paper_authors.name: Yang Liu
paper_title: MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware
    Multi-Segment Grounding
paper_summary: Video temporal understanding is crucial for multimodal large language models
  (MLLMs) to reason over events in videos. Despite recent advances in general
  video understanding, current MLLMs still struggle with fine-grained temporal
  reasoning. While reinforcement learning (RL) has been explored to address this
  issue recently, existing RL approaches remain limited in effectiveness. In this
  work, we propose MUSEG, a novel RL-based method that enhances temporal
  understanding by introducing timestamp-aware multi-segment grounding. MUSEG
  enables MLLMs to align queries with multiple relevant video segments, promoting
  more comprehensive temporal reasoning. To facilitate effective learning, we
  design a customized RL training recipe with phased rewards that progressively
  guides the model toward temporally grounded reasoning. Extensive experiments on
  temporal grounding and time-sensitive video QA tasks demonstrate that MUSEG
  significantly outperforms existing methods and generalizes well across diverse
  temporal understanding scenarios. View our project at
  https://github.com/THUNLP-MT/MUSEG.
paper_ai_summary: MUSEG, an RL-based method with timestamp-aware multi-segment grounding, significantly enhances the temporal understanding of large language models by improving alignment with video segments and demonstrating superior performance in temporal reasoning tasks.
paper_ai_keywords: reinforcement learning timestamp-aware multi-segment grounding temporal understanding MUSEG phased rewards temporal grounding time-sensitive video QA