MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware
  Multi-Segment Grounding
Video temporal understanding is crucial for multimodal large language models
(MLLMs) to reason over events in videos. Despite recent advances in general
video understanding, current MLLMs still struggle with fine-grained temporal
reasoning. While reinforcement learning (RL) has been explored to address this
issue recently, existing RL approaches remain limited in effectiveness. In this
work, we propose MUSEG, a novel RL-based method that enhances temporal
understanding by introducing timestamp-aware multi-segment grounding. MUSEG
enables MLLMs to align queries with multiple relevant video segments, promoting
more comprehensive temporal reasoning. To facilitate effective learning, we
design a customized RL training recipe with phased rewards that progressively
guides the model toward temporally grounded reasoning. Extensive experiments on
temporal grounding and time-sensitive video QA tasks demonstrate that MUSEG
significantly outperforms existing methods and generalizes well across diverse
temporal understanding scenarios. View our project at
https://github.com/THUNLP-MT/MUSEG.
6837eef0e237f02cd3c87963
Fuwen Luo
6837eef0e237f02cd3c87964
Shengfeng Lou
6837eef0e237f02cd3c87965
Chi Chen
6837eef0e237f02cd3c87966
Ziyue Wang
6837eef0e237f02cd3c87967
Chenliang Li
6837eef0e237f02cd3c87968
Weizhou Shen
6837eef0e237f02cd3c87969
Jiyue Guo
6837eef0e237f02cd3c8796a
Peng Li
6837eef0e237f02cd3c8796b
Ming Yan
6837eef0e237f02cd3c8796c
Ji Zhang
6837eef0e237f02cd3c8796d
Fei Huang
6837eef0e237f02cd3c8796e
Yang Liu
MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware
  Multi-Segment Grounding
Video temporal understanding is crucial for multimodal large language models
(MLLMs) to reason over events in videos. Despite recent advances in general
video understanding, current MLLMs still struggle with fine-grained temporal
reasoning. While reinforcement learning (RL) has been explored to address this
issue recently, existing RL approaches remain limited in effectiveness. In this
work, we propose MUSEG, a novel RL-based method that enhances temporal
understanding by introducing timestamp-aware multi-segment grounding. MUSEG
enables MLLMs to align queries with multiple relevant video segments, promoting
more comprehensive temporal reasoning. To facilitate effective learning, we
design a customized RL training recipe with phased rewards that progressively
guides the model toward temporally grounded reasoning. Extensive experiments on
temporal grounding and time-sensitive video QA tasks demonstrate that MUSEG
significantly outperforms existing methods and generalizes well across diverse
temporal understanding scenarios. View our project at
https://github.com/THUNLP-MT/MUSEG.
MUSEG, an RL-based method with timestamp-aware multi-segment grounding, significantly enhances the temporal understanding of large language models by improving alignment with video segments and demonstrating superior performance in temporal reasoning tasks.
reinforcement learning timestamp-aware multi-segment grounding temporal understanding MUSEG phased rewards temporal grounding time-sensitive video QA