Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers
The quality of the data and annotation upper-bounds the quality of a
downstream model. While there exist large text corpora and image-text pairs,
high-quality video-text data is much harder to collect. First of all, manual
labeling is more time-consuming, as it requires an annotator to watch an entire
video. Second, videos have a temporal dimension, consisting of several scenes
stacked together, and showing multiple actions. Accordingly, to establish a
video dataset with high-quality captions, we propose an automatic approach
leveraging multimodal inputs, such as textual video description, subtitles, and
individual video frames. Specifically, we curate 3.8M high-resolution videos
from the publicly available HD-VILA-100M dataset. We then split them into
semantically consistent video clips, and apply multiple cross-modality teacher
models to obtain captions for each video. Next, we finetune a retrieval model
on a small subset where the best caption of each video is manually selected and
then employ the model in the whole dataset to select the best caption as the
annotation. In this way, we get 70M videos paired with high-quality text
captions. We dub the dataset as Panda-70M. We show the value of the proposed
dataset on three downstream tasks: video captioning, video and text retrieval,
and text-driven video generation. The models trained on the proposed data score
substantially better on the majority of metrics across all the tasks.
65e14f3c66d097f80ad8099e
Tsai-Shien Chen
admin_assigned
2024-03-01T09:41:05.877Z
6330f570b68c7453d2ee8945
/avatars/5b394a72f277aa6c886b21b477010ab0.svg
Tsai-Shien Chen
user
tschen
65e14f3c66d097f80ad8099f
Aliaksandr Siarohin
admin_assigned
2024-03-01T09:41:14.186Z
64276311eb9a0ed86180715b
/avatars/76f933cd549f10e5e2db379de235d304.svg
Aliaksandr Siarohin
user
aliaksandr-siarohin
65e14f3c66d097f80ad809a0
Willi Menapace
admin_assigned
2024-03-01T09:41:21.214Z
6315358a362e3e95ea538081
/avatars/3b089a25a87c2e83c6b23ccb5d2dc73e.svg
Willi Menapace
user
willi-menapace
65e14f3c66d097f80ad809a1
Ekaterina Deyneka
admin_assigned
2024-03-01T09:41:28.453Z
635764d4d11d85992d00e014
/avatars/f4234ec5c56ac956545643751c5db9aa.svg
Ekaterina Deyneka
user
edeyneka
65e14f3c66d097f80ad809a2
Hsiang-wei Chao
65e14f3c66d097f80ad809a3
Byung Eun Jeon
admin_assigned
2024-03-01T09:41:42.308Z
6537f6f452d82b8877ffc5c4
/avatars/ec3acfc4272001714e485cac181cb993.svg
Byung Eun Jeon
user
byungeunjeon
65e14f3c66d097f80ad809a4
Yuwei Fang
65e14f3c66d097f80ad809a5
Hsin-Ying Lee
admin_assigned
2024-03-01T09:41:58.752Z
631bbd918de8e645af71eed9
/avatars/1cb94deadc288d07c571c9289085548c.svg
Hsin-Ying Lee
user
james371507
65e14f3c66d097f80ad809a6
Jian Ren
claimed_verified
2024-03-01T09:26:21.451Z
61f19829233c91cbd2f79e70
/avatars/a0735a94542b4f7cda5aed8bc4be0538.svg
Jian Ren
user
alanspike
65e14f3c66d097f80ad809a7
Ming-Hsuan Yang
65e14f3c66d097f80ad809a8
Sergey Tulyakov
Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers
The quality of the data and annotation upper-bounds the quality of a
downstream model. While there exist large text corpora and image-text pairs,
high-quality video-text data is much harder to collect. First of all, manual
labeling is more time-consuming, as it requires an annotator to watch an entire
video. Second, videos have a temporal dimension, consisting of several scenes
stacked together, and showing multiple actions. Accordingly, to establish a
video dataset with high-quality captions, we propose an automatic approach
leveraging multimodal inputs, such as textual video description, subtitles, and
individual video frames. Specifically, we curate 3.8M high-resolution videos
from the publicly available HD-VILA-100M dataset. We then split them into
semantically consistent video clips, and apply multiple cross-modality teacher
models to obtain captions for each video. Next, we finetune a retrieval model
on a small subset where the best caption of each video is manually selected and
then employ the model in the whole dataset to select the best caption as the
annotation. In this way, we get 70M videos paired with high-quality text
captions. We dub the dataset as Panda-70M. We show the value of the proposed
dataset on three downstream tasks: video captioning, video and text retrieval,
and text-driven video generation. The models trained on the proposed data score
substantially better on the majority of metrics across all the tasks.
A large-scale dataset of video-text pairs, automatically curated and annotated using multimodal models, significantly improves performance in video captioning, retrieval, and generation tasks.
multimodal inputs textual video description subtitles individual video frames cross-modality teacher models retrieval model semantic consistency video captioning video and text retrieval text-driven video generation