Temporal Preference Optimization for Long-Form Video Understanding
Despite significant advancements in video large multimodal models
(video-LMMs), achieving effective temporal grounding in long-form videos
remains a challenge for existing models. To address this limitation, we propose
Temporal Preference Optimization (TPO), a novel post-training framework
designed to enhance the temporal grounding capabilities of video-LMMs through
preference learning. TPO adopts a self-training approach that enables models to
differentiate between well-grounded and less accurate temporal responses by
leveraging curated preference datasets at two granularities: localized temporal
grounding, which focuses on specific video segments, and comprehensive temporal
grounding, which captures extended temporal dependencies across entire video
sequences. By optimizing on these preference datasets, TPO significantly
enhances temporal understanding while reducing reliance on manually annotated
data. Extensive experiments on three long-form video understanding
benchmarks--LongVideoBench, MLVU, and Video-MME--demonstrate the effectiveness
of TPO across two state-of-the-art video-LMMs. Notably, LLaVA-Video-TPO
establishes itself as the leading 7B model on the Video-MME benchmark,
underscoring the potential of TPO as a scalable and efficient solution for
advancing temporal reasoning in long-form video understanding. Project page:
https://ruili33.github.io/tpo_website.
679317f9d3ef2f790a539a28
Rui Li
claimed_verified
2025-01-24T10:17:34.282Z
6785fc7d17a2dfa3720ec082
/avatars/73e9d715bb16f14240c733c4843dfc22.svg
Rui Li
user
ruili0
679317f9d3ef2f790a539a29
Xiaohan Wang
admin_assigned
2025-01-24T10:11:30.999Z
65703fab7f50602340d23704
/avatars/324c45f5fba9cd8c38a89b30427c06b4.svg
Xiaohan Wang
user
nicholswang
679317f9d3ef2f790a539a2a
Yuhui Zhang
admin_assigned
2025-01-24T10:11:46.689Z
62da55164398e21bf7f0e292
https://cdn-avatars.huggingface.co/v1/production/uploads/62da55164398e21bf7f0e292/xjKkG8IA2IZZqCdjApSh3.jpeg
Yuhui Zhang
user
yuhuizhang
679317f9d3ef2f790a539a2b
Zeyu Wang
679317f9d3ef2f790a539a2c
Serena Yeung-Levy
admin_assigned
2025-01-24T10:09:52.788Z
677c8b2e92550a07fcad0f50
/avatars/2be26e8f25e98cfe5b1d227ee0409cd0.svg
Serena Yeung-Levy
user
yeunglevy
Temporal Preference Optimization for Long-Form Video Understanding
Despite significant advancements in video large multimodal models
(video-LMMs), achieving effective temporal grounding in long-form videos
remains a challenge for existing models. To address this limitation, we propose
Temporal Preference Optimization (TPO), a novel post-training framework
designed to enhance the temporal grounding capabilities of video-LMMs through
preference learning. TPO adopts a self-training approach that enables models to
differentiate between well-grounded and less accurate temporal responses by
leveraging curated preference datasets at two granularities: localized temporal
grounding, which focuses on specific video segments, and comprehensive temporal
grounding, which captures extended temporal dependencies across entire video
sequences. By optimizing on these preference datasets, TPO significantly
enhances temporal understanding while reducing reliance on manually annotated
data. Extensive experiments on three long-form video understanding
benchmarks--LongVideoBench, MLVU, and Video-MME--demonstrate the effectiveness
of TPO across two state-of-the-art video-LMMs. Notably, LLaVA-Video-TPO
establishes itself as the leading 7B model on the Video-MME benchmark,
underscoring the potential of TPO as a scalable and efficient solution for
advancing temporal reasoning in long-form video understanding. Project page:
https://ruili33.github.io/tpo_website.
Temporal Preference Optimization (TPO) enhances video-LMMs' temporal grounding through preference learning, improving performance on long-form video benchmarks with fewer manual annotations.
Temporal Preference Optimization TPO video-LMMs temporal grounding preference learning self-training localized temporal grounding comprehensive temporal grounding temporal understanding LongVideoBench MLVU Video-MME LLaVA-Video-TPO