title: Large-scale Pre-training for Grounded Video Caption Generation
summary: We propose a novel approach for captioning and object grounding in video,
  where the objects in the caption are grounded in the video via temporally dense
  bounding boxes. We introduce the following contributions. First, we present a
  large-scale automatic annotation method that aggregates captions grounded with
  bounding boxes across individual frames into temporally dense and consistent
  bounding box annotations. We apply this approach on the HowTo100M dataset to
  construct a large-scale pre-training dataset, named HowToGround1M. We also
  introduce a Grounded Video Caption Generation model, dubbed GROVE, and
  pre-train the model on HowToGround1M. Second, we introduce a new dataset,
  called iGround, of 3500 videos with manually annotated captions and dense
  spatio-temporally grounded bounding boxes. This allows us to measure progress
  on this challenging problem, as well as to fine-tune our model on this
  small-scale but high-quality data. Third, we demonstrate that our approach
  achieves state-of-the-art results on the proposed iGround dataset compared to a
  number of baselines, as well as on the VidSTG and ActivityNet-Entities
  datasets. We perform extensive ablations that demonstrate the importance of
  pre-training using our automatically annotated HowToGround1M dataset followed
  by fine-tuning on the manually annotated iGround dataset and validate the key
  technical contributions of our model.
paper_authors._id: 67d78ff6f789a7b68993ab6b
paper_authors.name: Evangelos Kazakos
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-17T08:44:14.132Z
paper_authors._id: 62f38b19261bc5fb2e06652c
paper_authors.avatarUrl: /avatars/0a7f7d63e1096f5d52bcb3be8e236c87.svg
paper_authors.fullname: Evangelos Kazakos
paper_authors.type: user
paper_authors.user: ekazakos
paper_authors._id: 67d78ff6f789a7b68993ab6c
paper_authors.name: Cordelia Schmid
paper_authors._id: 67d78ff6f789a7b68993ab6d
paper_authors.name: Josef Sivic
paper_title: Large-scale Pre-training for Grounded Video Caption Generation
paper_summary: We propose a novel approach for captioning and object grounding in video,
  where the objects in the caption are grounded in the video via temporally dense
  bounding boxes. We introduce the following contributions. First, we present a
  large-scale automatic annotation method that aggregates captions grounded with
  bounding boxes across individual frames into temporally dense and consistent
  bounding box annotations. We apply this approach on the HowTo100M dataset to
  construct a large-scale pre-training dataset, named HowToGround1M. We also
  introduce a Grounded Video Caption Generation model, dubbed GROVE, and
  pre-train the model on HowToGround1M. Second, we introduce a new dataset,
  called iGround, of 3500 videos with manually annotated captions and dense
  spatio-temporally grounded bounding boxes. This allows us to measure progress
  on this challenging problem, as well as to fine-tune our model on this
  small-scale but high-quality data. Third, we demonstrate that our approach
  achieves state-of-the-art results on the proposed iGround dataset compared to a
  number of baselines, as well as on the VidSTG and ActivityNet-Entities
  datasets. We perform extensive ablations that demonstrate the importance of
  pre-training using our automatically annotated HowToGround1M dataset followed
  by fine-tuning on the manually annotated iGround dataset and validate the key
  technical contributions of our model.
paper_ai_summary: A novel method for video captioning and object grounding using temporally dense bounding boxes achieves state-of-the-art results on multiple datasets.
paper_ai_keywords: large-scale automatic annotation temporally dense bounding boxes pre-training dataset HowToGround1M Grounded Video Caption Generation GROVE iGround spatio-temporally grounded bounding boxes VidSTG ActivityNet-Entities ablations