title: Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers
summary: The quality of the data and annotation upper-bounds the quality of a
  downstream model. While there exist large text corpora and image-text pairs,
  high-quality video-text data is much harder to collect. First of all, manual
  labeling is more time-consuming, as it requires an annotator to watch an entire
  video. Second, videos have a temporal dimension, consisting of several scenes
  stacked together, and showing multiple actions. Accordingly, to establish a
  video dataset with high-quality captions, we propose an automatic approach
  leveraging multimodal inputs, such as textual video description, subtitles, and
  individual video frames. Specifically, we curate 3.8M high-resolution videos
  from the publicly available HD-VILA-100M dataset. We then split them into
  semantically consistent video clips, and apply multiple cross-modality teacher
  models to obtain captions for each video. Next, we finetune a retrieval model
  on a small subset where the best caption of each video is manually selected and
  then employ the model in the whole dataset to select the best caption as the
  annotation. In this way, we get 70M videos paired with high-quality text
  captions. We dub the dataset as Panda-70M. We show the value of the proposed
  dataset on three downstream tasks: video captioning, video and text retrieval,
  and text-driven video generation. The models trained on the proposed data score
  substantially better on the majority of metrics across all the tasks.
paper_authors._id: 65e14f3c66d097f80ad8099e
paper_authors.name: Tsai-Shien Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T09:41:05.877Z
paper_authors._id: 6330f570b68c7453d2ee8945
paper_authors.avatarUrl: /avatars/5b394a72f277aa6c886b21b477010ab0.svg
paper_authors.fullname: Tsai-Shien Chen
paper_authors.type: user
paper_authors.user: tschen
paper_authors._id: 65e14f3c66d097f80ad8099f
paper_authors.name: Aliaksandr Siarohin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T09:41:14.186Z
paper_authors._id: 64276311eb9a0ed86180715b
paper_authors.avatarUrl: /avatars/76f933cd549f10e5e2db379de235d304.svg
paper_authors.fullname: Aliaksandr Siarohin
paper_authors.type: user
paper_authors.user: aliaksandr-siarohin
paper_authors._id: 65e14f3c66d097f80ad809a0
paper_authors.name: Willi Menapace
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T09:41:21.214Z
paper_authors._id: 6315358a362e3e95ea538081
paper_authors.avatarUrl: /avatars/3b089a25a87c2e83c6b23ccb5d2dc73e.svg
paper_authors.fullname: Willi Menapace
paper_authors.type: user
paper_authors.user: willi-menapace
paper_authors._id: 65e14f3c66d097f80ad809a1
paper_authors.name: Ekaterina Deyneka
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T09:41:28.453Z
paper_authors._id: 635764d4d11d85992d00e014
paper_authors.avatarUrl: /avatars/f4234ec5c56ac956545643751c5db9aa.svg
paper_authors.fullname: Ekaterina Deyneka
paper_authors.type: user
paper_authors.user: edeyneka
paper_authors._id: 65e14f3c66d097f80ad809a2
paper_authors.name: Hsiang-wei Chao
paper_authors._id: 65e14f3c66d097f80ad809a3
paper_authors.name: Byung Eun Jeon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T09:41:42.308Z
paper_authors._id: 6537f6f452d82b8877ffc5c4
paper_authors.avatarUrl: /avatars/ec3acfc4272001714e485cac181cb993.svg
paper_authors.fullname: Byung Eun Jeon
paper_authors.type: user
paper_authors.user: byungeunjeon
paper_authors._id: 65e14f3c66d097f80ad809a4
paper_authors.name: Yuwei Fang
paper_authors._id: 65e14f3c66d097f80ad809a5
paper_authors.name: Hsin-Ying Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T09:41:58.752Z
paper_authors._id: 631bbd918de8e645af71eed9
paper_authors.avatarUrl: /avatars/1cb94deadc288d07c571c9289085548c.svg
paper_authors.fullname: Hsin-Ying Lee
paper_authors.type: user
paper_authors.user: james371507
paper_authors._id: 65e14f3c66d097f80ad809a6
paper_authors.name: Jian Ren
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-01T09:26:21.451Z
paper_authors._id: 61f19829233c91cbd2f79e70
paper_authors.avatarUrl: /avatars/a0735a94542b4f7cda5aed8bc4be0538.svg
paper_authors.fullname: Jian Ren
paper_authors.type: user
paper_authors.user: alanspike
paper_authors._id: 65e14f3c66d097f80ad809a7
paper_authors.name: Ming-Hsuan Yang
paper_authors._id: 65e14f3c66d097f80ad809a8
paper_authors.name: Sergey Tulyakov
paper_title: Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers
paper_summary: The quality of the data and annotation upper-bounds the quality of a
  downstream model. While there exist large text corpora and image-text pairs,
  high-quality video-text data is much harder to collect. First of all, manual
  labeling is more time-consuming, as it requires an annotator to watch an entire
  video. Second, videos have a temporal dimension, consisting of several scenes
  stacked together, and showing multiple actions. Accordingly, to establish a
  video dataset with high-quality captions, we propose an automatic approach
  leveraging multimodal inputs, such as textual video description, subtitles, and
  individual video frames. Specifically, we curate 3.8M high-resolution videos
  from the publicly available HD-VILA-100M dataset. We then split them into
  semantically consistent video clips, and apply multiple cross-modality teacher
  models to obtain captions for each video. Next, we finetune a retrieval model
  on a small subset where the best caption of each video is manually selected and
  then employ the model in the whole dataset to select the best caption as the
  annotation. In this way, we get 70M videos paired with high-quality text
  captions. We dub the dataset as Panda-70M. We show the value of the proposed
  dataset on three downstream tasks: video captioning, video and text retrieval,
  and text-driven video generation. The models trained on the proposed data score
  substantially better on the majority of metrics across all the tasks.
paper_ai_summary: A large-scale dataset of video-text pairs, automatically curated and annotated using multimodal models, significantly improves performance in video captioning, retrieval, and generation tasks.
paper_ai_keywords: multimodal inputs textual video description subtitles individual video frames cross-modality teacher models retrieval model semantic consistency video captioning video and text retrieval text-driven video generation