title: Distilling Vision-Language Models on Millions of Videos
summary: The recent advance in vision-language models is largely attributed to the
  abundance of image-text data. We aim to replicate this success for
  video-language models, but there simply is not enough human-curated video-text
  data available. We thus resort to fine-tuning a video-language model from a
  strong image-language baseline with synthesized instructional data. The
  resulting video-language model is then used to auto-label millions of videos to
  generate high-quality captions. We show the adapted video-language model
  performs well on a wide range of video-language benchmarks. For instance, it
  surpasses the best prior result on open-ended NExT-QA by 2.8%. Besides, our
  model generates detailed descriptions for previously unseen videos, which
  provide better textual supervision than existing methods. Experiments show that
  a video-language dual-encoder model contrastively trained on these
  auto-generated captions is 3.8% better than the strongest baseline that also
  leverages vision-language models. Our best model outperforms state-of-the-art
  methods on MSR-VTT zero-shot text-to-video retrieval by 6%.
paper_authors._id: 65a0a233712e1321d0218825
paper_authors.name: Yue Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-15T07:46:38.549Z
paper_authors._id: 638fe91639f7e2a7f9d2a8c6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/638fe91639f7e2a7f9d2a8c6/hB7DMVODcdAEUdQnXxWA8.jpeg
paper_authors.fullname: Yue Zhao
paper_authors.type: user
paper_authors.user: zhaoyue-zephyrus
paper_authors._id: 65a0a233712e1321d0218826
paper_authors.name: Long Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:44:06.358Z
paper_authors._id: 650c249887dcda6616baa040
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/tIVfxoAHAJ0sWyDMkaarA.jpeg
paper_authors.fullname: Long Zhao
paper_authors.type: user
paper_authors.user: garyzhao9012
paper_authors._id: 65a0a233712e1321d0218827
paper_authors.name: Xingyi Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:44:12.690Z
paper_authors._id: 629ee11846b4826be2d94676
paper_authors.avatarUrl: /avatars/95ab04122ad626d74bf8a65d7b2bf48a.svg
paper_authors.fullname: Xingyi Zhou
paper_authors.type: user
paper_authors.user: zhouxy
paper_authors._id: 65a0a233712e1321d0218828
paper_authors.name: Jialin Wu
paper_authors._id: 65a0a233712e1321d0218829
paper_authors.name: Chun-Te Chu
paper_authors._id: 65a0a233712e1321d021882a
paper_authors.name: Hui Miao
paper_authors._id: 65a0a233712e1321d021882b
paper_authors.name: Florian Schroff
paper_authors._id: 65a0a233712e1321d021882c
paper_authors.name: Hartwig Adam
paper_authors._id: 65a0a233712e1321d021882d
paper_authors.name: Ting Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-22T22:28:19.819Z
paper_authors._id: 64a78e9bacc69c3bbaf12b3d
paper_authors.avatarUrl: /avatars/09cd26234e0e0c82f5cc2c02e710163c.svg
paper_authors.fullname: Ting Liu
paper_authors.type: user
paper_authors.user: tingianliu
paper_authors._id: 65a0a233712e1321d021882e
paper_authors.name: Boqing Gong
paper_authors._id: 65a0a233712e1321d021882f
paper_authors.name: Philipp Krähenbühl
paper_authors._id: 65a0a233712e1321d0218830
paper_authors.name: Liangzhe Yuan
paper_title: Distilling Vision-Language Models on Millions of Videos
paper_summary: The recent advance in vision-language models is largely attributed to the
  abundance of image-text data. We aim to replicate this success for
  video-language models, but there simply is not enough human-curated video-text
  data available. We thus resort to fine-tuning a video-language model from a
  strong image-language baseline with synthesized instructional data. The
  resulting video-language model is then used to auto-label millions of videos to
  generate high-quality captions. We show the adapted video-language model
  performs well on a wide range of video-language benchmarks. For instance, it
  surpasses the best prior result on open-ended NExT-QA by 2.8%. Besides, our
  model generates detailed descriptions for previously unseen videos, which
  provide better textual supervision than existing methods. Experiments show that
  a video-language dual-encoder model contrastively trained on these
  auto-generated captions is 3.8% better than the strongest baseline that also
  leverages vision-language models. Our best model outperforms state-of-the-art
  methods on MSR-VTT zero-shot text-to-video retrieval by 6%.
paper_ai_summary: A video-language model, fine-tuned from an image-language baseline with synthesized data, outperforms existing methods on various benchmarks by leveraging auto-generated captions for better textual supervision.
paper_ai_keywords: vision-language models video-language models fine-tuning synthesized instructional data auto-labeling captions video-language benchmarks open-ended NExT-QA contrastively trained dual-encoder model zero-shot text-to-video retrieval state-of-the-art methods