Distilling Vision-Language Models on Millions of Videos
The recent advance in vision-language models is largely attributed to the
abundance of image-text data. We aim to replicate this success for
video-language models, but there simply is not enough human-curated video-text
data available. We thus resort to fine-tuning a video-language model from a
strong image-language baseline with synthesized instructional data. The
resulting video-language model is then used to auto-label millions of videos to
generate high-quality captions. We show the adapted video-language model
performs well on a wide range of video-language benchmarks. For instance, it
surpasses the best prior result on open-ended NExT-QA by 2.8%. Besides, our
model generates detailed descriptions for previously unseen videos, which
provide better textual supervision than existing methods. Experiments show that
a video-language dual-encoder model contrastively trained on these
auto-generated captions is 3.8% better than the strongest baseline that also
leverages vision-language models. Our best model outperforms state-of-the-art
methods on MSR-VTT zero-shot text-to-video retrieval by 6%.
65a0a233712e1321d0218825
Yue Zhao
claimed_verified
2024-06-15T07:46:38.549Z
638fe91639f7e2a7f9d2a8c6
https://cdn-avatars.huggingface.co/v1/production/uploads/638fe91639f7e2a7f9d2a8c6/hB7DMVODcdAEUdQnXxWA8.jpeg
Yue Zhao
user
zhaoyue-zephyrus
65a0a233712e1321d0218826
Long Zhao
admin_assigned
2024-01-12T10:44:06.358Z
650c249887dcda6616baa040
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/tIVfxoAHAJ0sWyDMkaarA.jpeg
Long Zhao
user
garyzhao9012
65a0a233712e1321d0218827
Xingyi Zhou
admin_assigned
2024-01-12T10:44:12.690Z
629ee11846b4826be2d94676
/avatars/95ab04122ad626d74bf8a65d7b2bf48a.svg
Xingyi Zhou
user
zhouxy
65a0a233712e1321d0218828
Jialin Wu
65a0a233712e1321d0218829
Chun-Te Chu
65a0a233712e1321d021882a
Hui Miao
65a0a233712e1321d021882b
Florian Schroff
65a0a233712e1321d021882c
Hartwig Adam
65a0a233712e1321d021882d
Ting Liu
claimed_verified
2024-02-22T22:28:19.819Z
64a78e9bacc69c3bbaf12b3d
/avatars/09cd26234e0e0c82f5cc2c02e710163c.svg
Ting Liu
user
tingianliu
65a0a233712e1321d021882e
Boqing Gong
65a0a233712e1321d021882f
Philipp Krähenbühl
65a0a233712e1321d0218830
Liangzhe Yuan
Distilling Vision-Language Models on Millions of Videos
The recent advance in vision-language models is largely attributed to the
abundance of image-text data. We aim to replicate this success for
video-language models, but there simply is not enough human-curated video-text
data available. We thus resort to fine-tuning a video-language model from a
strong image-language baseline with synthesized instructional data. The
resulting video-language model is then used to auto-label millions of videos to
generate high-quality captions. We show the adapted video-language model
performs well on a wide range of video-language benchmarks. For instance, it
surpasses the best prior result on open-ended NExT-QA by 2.8%. Besides, our
model generates detailed descriptions for previously unseen videos, which
provide better textual supervision than existing methods. Experiments show that
a video-language dual-encoder model contrastively trained on these
auto-generated captions is 3.8% better than the strongest baseline that also
leverages vision-language models. Our best model outperforms state-of-the-art
methods on MSR-VTT zero-shot text-to-video retrieval by 6%.
A video-language model, fine-tuned from an image-language baseline with synthesized data, outperforms existing methods on various benchmarks by leveraging auto-generated captions for better textual supervision.
vision-language models video-language models fine-tuning synthesized instructional data auto-labeling captions video-language benchmarks open-ended NExT-QA contrastively trained dual-encoder model zero-shot text-to-video retrieval state-of-the-art methods