title: VidChapters-7M: Video Chapters at Scale
summary: Segmenting long videos into chapters enables users to quickly navigate to the
  information of their interest. This important topic has been understudied due
  to the lack of publicly released datasets. To address this issue, we present
  VidChapters-7M, a dataset of 817K user-chaptered videos including 7M chapters
  in total. VidChapters-7M is automatically created from videos online in a
  scalable manner by scraping user-annotated chapters and hence without any
  additional manual annotation. We introduce the following three tasks based on
  this data. First, the video chapter generation task consists of temporally
  segmenting the video and generating a chapter title for each segment. To
  further dissect the problem, we also define two variants of this task: video
  chapter generation given ground-truth boundaries, which requires generating a
  chapter title given an annotated video segment, and video chapter grounding,
  which requires temporally localizing a chapter given its annotated title. We
  benchmark both simple baselines and state-of-the-art video-language models for
  these three tasks. We also show that pretraining on VidChapters-7M transfers
  well to dense video captioning tasks in both zero-shot and finetuning settings,
  largely improving the state of the art on the YouCook2 and ViTT benchmarks.
  Finally, our experiments reveal that downstream performance scales well with
  the size of the pretraining dataset. Our dataset, code, and models are publicly
  available at https://antoyang.github.io/vidchapters.html.
paper_authors._id: 65127d53ea5ccf1113db531f
paper_authors.name: Antoine Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-26T13:23:38.308Z
paper_authors._id: 6512da122f79557a90b1717f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6512da122f79557a90b1717f/-rL2Iu_HHPEi1zWOL4cZi.jpeg
paper_authors.fullname: Antoine Yang
paper_authors.type: user
paper_authors.user: antoineyang
paper_authors._id: 65127d53ea5ccf1113db5320
paper_authors.name: Arsha Nagrani
paper_authors._id: 65127d53ea5ccf1113db5321
paper_authors.name: Ivan Laptev
paper_authors._id: 65127d53ea5ccf1113db5322
paper_authors.name: Josef Sivic
paper_authors._id: 65127d53ea5ccf1113db5323
paper_authors.name: Cordelia Schmid
paper_title: VidChapters-7M: Video Chapters at Scale
paper_summary: Segmenting long videos into chapters enables users to quickly navigate to the
  information of their interest. This important topic has been understudied due
  to the lack of publicly released datasets. To address this issue, we present
  VidChapters-7M, a dataset of 817K user-chaptered videos including 7M chapters
  in total. VidChapters-7M is automatically created from videos online in a
  scalable manner by scraping user-annotated chapters and hence without any
  additional manual annotation. We introduce the following three tasks based on
  this data. First, the video chapter generation task consists of temporally
  segmenting the video and generating a chapter title for each segment. To
  further dissect the problem, we also define two variants of this task: video
  chapter generation given ground-truth boundaries, which requires generating a
  chapter title given an annotated video segment, and video chapter grounding,
  which requires temporally localizing a chapter given its annotated title. We
  benchmark both simple baselines and state-of-the-art video-language models for
  these three tasks. We also show that pretraining on VidChapters-7M transfers
  well to dense video captioning tasks in both zero-shot and finetuning settings,
  largely improving the state of the art on the YouCook2 and ViTT benchmarks.
  Finally, our experiments reveal that downstream performance scales well with
  the size of the pretraining dataset. Our dataset, code, and models are publicly
  available at https://antoyang.github.io/vidchapters.html.
paper_ai_keywords: video chapter generation video chapter generation given ground-truth boundaries video chapter grounding video-language models dense video captioning zero-shot setting finetuning settings YouCook2 ViTT benchmarks