title: UniVTG: Towards Unified Video-Language Temporal Grounding
summary: Video Temporal Grounding (VTG), which aims to ground target clips from videos
  (such as consecutive intervals or disjoint shots) according to custom language
  queries (e.g., sentences or words), is key for video browsing on social media.
  Most methods in this direction develop taskspecific models that are trained
  with type-specific labels, such as moment retrieval (time interval) and
  highlight detection (worthiness curve), which limits their abilities to
  generalize to various VTG tasks and labels. In this paper, we propose to Unify
  the diverse VTG labels and tasks, dubbed UniVTG, along three directions:
  Firstly, we revisit a wide range of VTG labels and tasks and define a unified
  formulation. Based on this, we develop data annotation schemes to create
  scalable pseudo supervision. Secondly, we develop an effective and flexible
  grounding model capable of addressing each task and making full use of each
  label. Lastly, thanks to the unified framework, we are able to unlock temporal
  grounding pretraining from large-scale diverse labels and develop stronger
  grounding abilities e.g., zero-shot grounding. Extensive experiments on three
  tasks (moment retrieval, highlight detection and video summarization) across
  seven datasets (QVHighlights, Charades-STA, TACoS, Ego4D, YouTube Highlights,
  TVSum, and QFVS) demonstrate the effectiveness and flexibility of our proposed
  framework. The codes are available at https://github.com/showlab/UniVTG.
paper_authors._id: 64c8811bc864d962edf15b57
paper_authors.name: Kevin Qinghong Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-01T09:54:45.392Z
paper_authors._id: 64440be5af034cdfd69ca3a7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64440be5af034cdfd69ca3a7/qmx24QiDFT29vleCxL9TX.jpeg
paper_authors.fullname: Qinghong (Kevin) Lin
paper_authors.type: user
paper_authors.user: KevinQHLin
paper_authors._id: 64c8811bc864d962edf15b58
paper_authors.name: Pengchuan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-01T09:54:53.664Z
paper_authors._id: 640056f5261cfa61f39b4c35
paper_authors.avatarUrl: /avatars/25aa5dee42caa69327d158a669f6bf3a.svg
paper_authors.fullname: Pengchuan Zhang
paper_authors.type: user
paper_authors.user: pzzhang
paper_authors._id: 64c8811bc864d962edf15b59
paper_authors.name: Joya Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-01T09:55:20.635Z
paper_authors._id: 642435a1a3adbc7142c3b0a6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642435a1a3adbc7142c3b0a6/EBmQ7LnfdTdyuhSUti0-d.png
paper_authors.fullname: Joya Chen
paper_authors.type: user
paper_authors.user: chenjoya
paper_authors._id: 64c8811bc864d962edf15b5a
paper_authors.name: Shraman Pramanick
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-01T09:55:43.623Z
paper_authors._id: 64aec10a9135f79b09117476
paper_authors.avatarUrl: /avatars/88a6bcfeee99c12496940575a2fc65e9.svg
paper_authors.fullname: Shraman Pramanick
paper_authors.type: user
paper_authors.user: Shraman
paper_authors._id: 64c8811bc864d962edf15b5b
paper_authors.name: Difei Gao
paper_authors._id: 64c8811bc864d962edf15b5c
paper_authors.name: Alex Jinpeng Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-01T09:56:00.013Z
paper_authors._id: 62333a88fd7bb4a39b92d387
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62333a88fd7bb4a39b92d387/e21AhpcXq37Ak_7rZ-Ca9.png
paper_authors.fullname: Alex Jinpeng Wang
paper_authors.type: user
paper_authors.user: Awiny
paper_authors._id: 64c8811bc864d962edf15b5d
paper_authors.name: Rui Yan
paper_authors._id: 64c8811bc864d962edf15b5e
paper_authors.name: Mike Zheng Shou
paper_title: UniVTG: Towards Unified Video-Language Temporal Grounding
paper_summary: Video Temporal Grounding (VTG), which aims to ground target clips from videos
  (such as consecutive intervals or disjoint shots) according to custom language
  queries (e.g., sentences or words), is key for video browsing on social media.
  Most methods in this direction develop taskspecific models that are trained
  with type-specific labels, such as moment retrieval (time interval) and
  highlight detection (worthiness curve), which limits their abilities to
  generalize to various VTG tasks and labels. In this paper, we propose to Unify
  the diverse VTG labels and tasks, dubbed UniVTG, along three directions:
  Firstly, we revisit a wide range of VTG labels and tasks and define a unified
  formulation. Based on this, we develop data annotation schemes to create
  scalable pseudo supervision. Secondly, we develop an effective and flexible
  grounding model capable of addressing each task and making full use of each
  label. Lastly, thanks to the unified framework, we are able to unlock temporal
  grounding pretraining from large-scale diverse labels and develop stronger
  grounding abilities e.g., zero-shot grounding. Extensive experiments on three
  tasks (moment retrieval, highlight detection and video summarization) across
  seven datasets (QVHighlights, Charades-STA, TACoS, Ego4D, YouTube Highlights,
  TVSum, and QFVS) demonstrate the effectiveness and flexibility of our proposed
  framework. The codes are available at https://github.com/showlab/UniVTG.
paper_ai_keywords: Video Temporal Grounding (VTG) bounding boxes disjoint shots moment retrieval highlight detection bounding bounding worthiness curve unified formulation scalable pseudo supervision grounding model temporal grounding pretraining zero-shot grounding video summarization