title: VideoGLUE: Video General Understanding Evaluation of Foundation Models
summary: We evaluate existing foundation models video understanding capabilities using
  a carefully designed experiment protocol consisting of three hallmark tasks
  (action recognition, temporal localization, and spatiotemporal localization),
  eight datasets well received by the community, and four adaptation methods
  tailoring a foundation model (FM) for a downstream task. Moreover, we propose a
  scalar VideoGLUE score (VGS) to measure an FMs efficacy and efficiency when
  adapting to general video understanding tasks. Our main findings are as
  follows. First, task-specialized models significantly outperform the six FMs
  studied in this work, in sharp contrast to what FMs have achieved in natural
  language and image understanding. Second,video-native FMs, whose pretraining
  data contains the video modality, are generally better than image-native FMs in
  classifying motion-rich videos, localizing actions in time, and understanding a
  video of more than one action. Third, the video-native FMs can perform well on
  video tasks under light adaptations to downstream tasks(e.g., freezing the FM
  backbones), while image-native FMs win in full end-to-end finetuning. The first
  two observations reveal the need and tremendous opportunities to conduct
  research on video-focused FMs, and the last confirms that both tasks and
  adaptation methods matter when it comes to the evaluation of FMs.
paper_authors._id: 64a776501e4dd9f35494e10a
paper_authors.name: Liangzhe Yuan
paper_authors._id: 64a776501e4dd9f35494e10b
paper_authors.name: Nitesh Bharadwaj Gundavarapu
paper_authors._id: 64a776501e4dd9f35494e10c
paper_authors.name: Long Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T07:22:57.458Z
paper_authors._id: 650c249887dcda6616baa040
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/tIVfxoAHAJ0sWyDMkaarA.jpeg
paper_authors.fullname: Long Zhao
paper_authors.type: user
paper_authors.user: garyzhao9012
paper_authors._id: 64a776501e4dd9f35494e10d
paper_authors.name: Hao Zhou
paper_authors._id: 64a776501e4dd9f35494e10e
paper_authors.name: Yin Cui
paper_authors._id: 64a776501e4dd9f35494e10f
paper_authors.name: Lu Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-23T11:18:22.043Z
paper_authors._id: 64a5cba3bea0116f8f7187a7
paper_authors.avatarUrl: /avatars/97b3bad82e359c57d1ab7d6e3aa7748b.svg
paper_authors.fullname: Lu Jiang
paper_authors.type: user
paper_authors.user: roadjiang
paper_authors._id: 64a776501e4dd9f35494e110
paper_authors.name: Xuan Yang
paper_authors._id: 64a776501e4dd9f35494e111
paper_authors.name: Menglin Jia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-10T09:24:38.330Z
paper_authors._id: 648a4681fcda61143190645e
paper_authors.avatarUrl: /avatars/270bca51ef4d8453920713e9a33c46b3.svg
paper_authors.fullname: Menglin Jia
paper_authors.type: user
paper_authors.user: kmnp
paper_authors._id: 64a776501e4dd9f35494e112
paper_authors.name: Tobias Weyand
paper_authors._id: 64a776501e4dd9f35494e113
paper_authors.name: Luke Friedman
paper_authors._id: 64a776501e4dd9f35494e114
paper_authors.name: Mikhail Sirotenko
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-21T16:30:11.812Z
paper_authors._id: 65b13272dfa3789ef39937ce
paper_authors.avatarUrl: /avatars/c7803adb1b82b881946b3b0aeaee5f4c.svg
paper_authors.fullname: Mikhail Sirotenko
paper_authors.type: user
paper_authors.user: msirotenko-g
paper_authors._id: 64a776501e4dd9f35494e115
paper_authors.name: Huisheng Wang
paper_authors._id: 64a776501e4dd9f35494e116
paper_authors.name: Florian Schroff
paper_authors._id: 64a776501e4dd9f35494e117
paper_authors.name: Hartwig Adam
paper_authors._id: 64a776501e4dd9f35494e118
paper_authors.name: Ming-Hsuan Yang
paper_authors._id: 64a776501e4dd9f35494e119
paper_authors.name: Ting Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-10T09:24:33.909Z
paper_authors._id: 64a78e9bacc69c3bbaf12b3d
paper_authors.avatarUrl: /avatars/09cd26234e0e0c82f5cc2c02e710163c.svg
paper_authors.fullname: Ting Liu
paper_authors.type: user
paper_authors.user: tingianliu
paper_authors._id: 64a776501e4dd9f35494e11a
paper_authors.name: Boqing Gong
paper_title: VideoGLUE: Video General Understanding Evaluation of Foundation Models
paper_summary: We evaluate existing foundation models video understanding capabilities using
  a carefully designed experiment protocol consisting of three hallmark tasks
  (action recognition, temporal localization, and spatiotemporal localization),
  eight datasets well received by the community, and four adaptation methods
  tailoring a foundation model (FM) for a downstream task. Moreover, we propose a
  scalar VideoGLUE score (VGS) to measure an FMs efficacy and efficiency when
  adapting to general video understanding tasks. Our main findings are as
  follows. First, task-specialized models significantly outperform the six FMs
  studied in this work, in sharp contrast to what FMs have achieved in natural
  language and image understanding. Second,video-native FMs, whose pretraining
  data contains the video modality, are generally better than image-native FMs in
  classifying motion-rich videos, localizing actions in time, and understanding a
  video of more than one action. Third, the video-native FMs can perform well on
  video tasks under light adaptations to downstream tasks(e.g., freezing the FM
  backbones), while image-native FMs win in full end-to-end finetuning. The first
  two observations reveal the need and tremendous opportunities to conduct
  research on video-focused FMs, and the last confirms that both tasks and
  adaptation methods matter when it comes to the evaluation of FMs.
paper_ai_keywords: foundation models video understanding experiment protocol action recognition temporal localization spatiotemporal localization datasets adaptation methods scalar VideoGLUE score (VGS) task-specialized models video-native FMs image-native FMs motion-rich videos freezing the FM backbones full end-to-end finetuning