UniVTG: Towards Unified Video-Language Temporal Grounding
Video Temporal Grounding (VTG), which aims to ground target clips from videos
(such as consecutive intervals or disjoint shots) according to custom language
queries (e.g., sentences or words), is key for video browsing on social media.
Most methods in this direction develop taskspecific models that are trained
with type-specific labels, such as moment retrieval (time interval) and
highlight detection (worthiness curve), which limits their abilities to
generalize to various VTG tasks and labels. In this paper, we propose to Unify
the diverse VTG labels and tasks, dubbed UniVTG, along three directions:
Firstly, we revisit a wide range of VTG labels and tasks and define a unified
formulation. Based on this, we develop data annotation schemes to create
scalable pseudo supervision. Secondly, we develop an effective and flexible
grounding model capable of addressing each task and making full use of each
label. Lastly, thanks to the unified framework, we are able to unlock temporal
grounding pretraining from large-scale diverse labels and develop stronger
grounding abilities e.g., zero-shot grounding. Extensive experiments on three
tasks (moment retrieval, highlight detection and video summarization) across
seven datasets (QVHighlights, Charades-STA, TACoS, Ego4D, YouTube Highlights,
TVSum, and QFVS) demonstrate the effectiveness and flexibility of our proposed
framework. The codes are available at https://github.com/showlab/UniVTG.
64c8811bc864d962edf15b57
Kevin Qinghong Lin
admin_assigned
2023-08-01T09:54:45.392Z
64440be5af034cdfd69ca3a7
https://cdn-avatars.huggingface.co/v1/production/uploads/64440be5af034cdfd69ca3a7/qmx24QiDFT29vleCxL9TX.jpeg
Qinghong (Kevin) Lin
user
KevinQHLin
64c8811bc864d962edf15b58
Pengchuan Zhang
admin_assigned
2023-08-01T09:54:53.664Z
640056f5261cfa61f39b4c35
/avatars/25aa5dee42caa69327d158a669f6bf3a.svg
Pengchuan Zhang
user
pzzhang
64c8811bc864d962edf15b59
Joya Chen
admin_assigned
2023-08-01T09:55:20.635Z
642435a1a3adbc7142c3b0a6
https://cdn-avatars.huggingface.co/v1/production/uploads/642435a1a3adbc7142c3b0a6/wgLT_w9jNWRU3O0jU0646.jpeg
Joya Chen
user
chenjoya
64c8811bc864d962edf15b5a
Shraman Pramanick
admin_assigned
2023-08-01T09:55:43.623Z
64aec10a9135f79b09117476
/avatars/83db978a906782b6df71d178b7a0df89.svg
Shraman Pramanick
user
Shraman
64c8811bc864d962edf15b5b
Difei Gao
64c8811bc864d962edf15b5c
Alex Jinpeng Wang
admin_assigned
2023-08-01T09:56:00.013Z
62333a88fd7bb4a39b92d387
https://cdn-avatars.huggingface.co/v1/production/uploads/62333a88fd7bb4a39b92d387/e21AhpcXq37Ak_7rZ-Ca9.png
Alex Jinpeng Wang
user
Awiny
64c8811bc864d962edf15b5d
Rui Yan
64c8811bc864d962edf15b5e
Mike Zheng Shou
UniVTG: Towards Unified Video-Language Temporal Grounding
Video Temporal Grounding (VTG), which aims to ground target clips from videos
(such as consecutive intervals or disjoint shots) according to custom language
queries (e.g., sentences or words), is key for video browsing on social media.
Most methods in this direction develop taskspecific models that are trained
with type-specific labels, such as moment retrieval (time interval) and
highlight detection (worthiness curve), which limits their abilities to
generalize to various VTG tasks and labels. In this paper, we propose to Unify
the diverse VTG labels and tasks, dubbed UniVTG, along three directions:
Firstly, we revisit a wide range of VTG labels and tasks and define a unified
formulation. Based on this, we develop data annotation schemes to create
scalable pseudo supervision. Secondly, we develop an effective and flexible
grounding model capable of addressing each task and making full use of each
label. Lastly, thanks to the unified framework, we are able to unlock temporal
grounding pretraining from large-scale diverse labels and develop stronger
grounding abilities e.g., zero-shot grounding. Extensive experiments on three
tasks (moment retrieval, highlight detection and video summarization) across
seven datasets (QVHighlights, Charades-STA, TACoS, Ego4D, YouTube Highlights,
TVSum, and QFVS) demonstrate the effectiveness and flexibility of our proposed
framework. The codes are available at https://github.com/showlab/UniVTG.
UniVTG unifies various video temporal grounding tasks and labels through a unified framework, enabling effective and flexible model development and delivering strong performance across multiple datasets.
Video Temporal Grounding (VTG) moment retrieval highlight detection video summarization unified formulation pseudo supervision grounding model pretraining zero-shot grounding