title: VideoPrism: A Foundational Visual Encoder for Video Understanding
summary: We introduce VideoPrism, a general-purpose video encoder that tackles diverse
  video understanding tasks with a single frozen model. We pretrain VideoPrism on
  a heterogeneous corpus containing 36M high-quality video-caption pairs and 582M
  video clips with noisy parallel text (e.g., ASR transcripts). The pretraining
  approach improves upon masked autoencoding by global-local distillation of
  semantic video embeddings and a token shuffling scheme, enabling VideoPrism to
  focus primarily on the video modality while leveraging the invaluable text
  associated with videos. We extensively test VideoPrism on four broad groups of
  video understanding tasks, from web video question answering to CV for science,
  achieving state-of-the-art performance on 30 out of 33 video understanding
  benchmarks.
paper_authors._id: 65d573c188f4a905891d40cc
paper_authors.name: Long Zhao
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-02-21T04:41:13.955Z
paper_authors._id: 650c249887dcda6616baa040
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/tIVfxoAHAJ0sWyDMkaarA.jpeg
paper_authors.fullname: Long Zhao
paper_authors.type: user
paper_authors.user: garyzhao9012
paper_authors._id: 65d573c188f4a905891d40cd
paper_authors.name: Nitesh B. Gundavarapu
paper_authors._id: 65d573c188f4a905891d40ce
paper_authors.name: Liangzhe Yuan
paper_authors._id: 65d573c188f4a905891d40cf
paper_authors.name: Hao Zhou
paper_authors._id: 65d573c188f4a905891d40d0
paper_authors.name: Shen Yan
paper_authors._id: 65d573c188f4a905891d40d1
paper_authors.name: Jennifer J. Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-21T10:15:45.333Z
paper_authors._id: 64cec85201931c6016f6f1b3
paper_authors.avatarUrl: /avatars/77a4ebbb3995600cd01d385b2a6084c4.svg
paper_authors.fullname: Jennifer J. Sun
paper_authors.type: user
paper_authors.user: JenJSun
paper_authors._id: 65d573c188f4a905891d40d2
paper_authors.name: Luke Friedman
paper_authors._id: 65d573c188f4a905891d40d3
paper_authors.name: Rui Qian
paper_authors._id: 65d573c188f4a905891d40d4
paper_authors.name: Tobias Weyand
paper_authors._id: 65d573c188f4a905891d40d5
paper_authors.name: Yue Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-15T07:46:18.796Z
paper_authors._id: 638fe91639f7e2a7f9d2a8c6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/638fe91639f7e2a7f9d2a8c6/hB7DMVODcdAEUdQnXxWA8.jpeg
paper_authors.fullname: Yue Zhao
paper_authors.type: user
paper_authors.user: zhaoyue-zephyrus
paper_authors._id: 65d573c188f4a905891d40d6
paper_authors.name: Rachel Hornung
paper_authors._id: 65d573c188f4a905891d40d7
paper_authors.name: Florian Schroff
paper_authors._id: 65d573c188f4a905891d40d8
paper_authors.name: Ming-Hsuan Yang
paper_authors._id: 65d573c188f4a905891d40d9
paper_authors.name: David A. Ross
paper_authors._id: 65d573c188f4a905891d40da
paper_authors.name: Huisheng Wang
paper_authors._id: 65d573c188f4a905891d40db
paper_authors.name: Hartwig Adam
paper_authors._id: 65d573c188f4a905891d40dc
paper_authors.name: Mikhail Sirotenko
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-02-21T14:33:00.920Z
paper_authors._id: 65b13272dfa3789ef39937ce
paper_authors.avatarUrl: /avatars/c7803adb1b82b881946b3b0aeaee5f4c.svg
paper_authors.fullname: Mikhail Sirotenko
paper_authors.type: user
paper_authors.user: msirotenko-g
paper_authors._id: 65d573c188f4a905891d40dd
paper_authors.name: Ting Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-22T22:27:59.395Z
paper_authors._id: 64a78e9bacc69c3bbaf12b3d
paper_authors.avatarUrl: /avatars/09cd26234e0e0c82f5cc2c02e710163c.svg
paper_authors.fullname: Ting Liu
paper_authors.type: user
paper_authors.user: tingianliu
paper_authors._id: 65d573c188f4a905891d40de
paper_authors.name: Boqing Gong
paper_title: VideoPrism: A Foundational Visual Encoder for Video Understanding
paper_summary: We introduce VideoPrism, a general-purpose video encoder that tackles diverse
  video understanding tasks with a single frozen model. We pretrain VideoPrism on
  a heterogeneous corpus containing 36M high-quality video-caption pairs and 582M
  video clips with noisy parallel text (e.g., ASR transcripts). The pretraining
  approach improves upon masked autoencoding by global-local distillation of
  semantic video embeddings and a token shuffling scheme, enabling VideoPrism to
  focus primarily on the video modality while leveraging the invaluable text
  associated with videos. We extensively test VideoPrism on four broad groups of
  video understanding tasks, from web video question answering to CV for science,
  achieving state-of-the-art performance on 30 out of 33 video understanding
  benchmarks.
paper_ai_summary: VideoPrism, a pretrained video encoder, achieves top performance across various video understanding tasks by utilizing global-local distillation and token shuffling of semantic video embeddings enhanced with associated text.
paper_ai_keywords: VideoPrism masked autoencoding global-local distillation semantic video embeddings token shuffling scheme video understanding tasks web video question answering CV for science video understanding benchmarks