title: Video ReCap: Recursive Captioning of Hour-Long Videos
summary: Most video captioning models are designed to process short video clips of few
  seconds and output text describing low-level visual concepts (e.g., objects,
  scenes, atomic actions). However, most real-world videos last for minutes or
  hours and have a complex hierarchical structure spanning different temporal
  granularities. We propose Video ReCap, a recursive video captioning model that
  can process video inputs of dramatically different lengths (from 1 second to 2
  hours) and output video captions at multiple hierarchy levels. The recursive
  video-language architecture exploits the synergy between different video
  hierarchies and can process hour-long videos efficiently. We utilize a
  curriculum learning training scheme to learn the hierarchical structure of
  videos, starting from clip-level captions describing atomic actions, then
  focusing on segment-level descriptions, and concluding with generating
  summaries for hour-long videos. Furthermore, we introduce Ego4D-HCap dataset by
  augmenting Ego4D with 8,267 manually collected long-range video summaries. Our
  recursive model can flexibly generate captions at different hierarchy levels
  while also being useful for other complex video understanding tasks, such as
  VideoQA on EgoSchema. Data, code, and models are available at:
  https://sites.google.com/view/vidrecap
paper_authors._id: 65d571b9882e75a4e90c6714
paper_authors.name: Md Mohaiminul Islam
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-21T16:29:31.755Z
paper_authors._id: 64e424dbd2af12910d916ff7
paper_authors.avatarUrl: /avatars/d763c32bc4ee2daffa42311fa9e912da.svg
paper_authors.fullname: Md Mohaiminul Islam
paper_authors.type: user
paper_authors.user: mmiemon
paper_authors._id: 65d571b9882e75a4e90c6715
paper_authors.name: Ngan Ho
paper_authors._id: 65d571b9882e75a4e90c6716
paper_authors.name: Xitong Yang
paper_authors._id: 65d571b9882e75a4e90c6717
paper_authors.name: Tushar Nagarajan
paper_authors._id: 65d571b9882e75a4e90c6718
paper_authors.name: Lorenzo Torresani
paper_authors._id: 65d571b9882e75a4e90c6719
paper_authors.name: Gedas Bertasius
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-21T16:29:35.306Z
paper_authors._id: 65d60993c2c6418adc99206a
paper_authors.avatarUrl: /avatars/fb1f9d5af7022bddf4cb090cd477108d.svg
paper_authors.fullname: Gedas Bertasius
paper_authors.type: user
paper_authors.user: gberta
paper_title: Video ReCap: Recursive Captioning of Hour-Long Videos
paper_summary: Most video captioning models are designed to process short video clips of few
  seconds and output text describing low-level visual concepts (e.g., objects,
  scenes, atomic actions). However, most real-world videos last for minutes or
  hours and have a complex hierarchical structure spanning different temporal
  granularities. We propose Video ReCap, a recursive video captioning model that
  can process video inputs of dramatically different lengths (from 1 second to 2
  hours) and output video captions at multiple hierarchy levels. The recursive
  video-language architecture exploits the synergy between different video
  hierarchies and can process hour-long videos efficiently. We utilize a
  curriculum learning training scheme to learn the hierarchical structure of
  videos, starting from clip-level captions describing atomic actions, then
  focusing on segment-level descriptions, and concluding with generating
  summaries for hour-long videos. Furthermore, we introduce Ego4D-HCap dataset by
  augmenting Ego4D with 8,267 manually collected long-range video summaries. Our
  recursive model can flexibly generate captions at different hierarchy levels
  while also being useful for other complex video understanding tasks, such as
  VideoQA on EgoSchema. Data, code, and models are available at:
  https://sites.google.com/view/vidrecap
paper_ai_keywords: recursive video captioning model recursive video-language architecture curriculum learning training scheme clip-level captions segment-level descriptions hour-long videos Ego4D-HCap dataset VideoQA EgoSchema