title: Video-MME: The First-Ever Comprehensive Evaluation Benchmark of
    Multi-modal LLMs in Video Analysis
summary: In the quest for artificial general intelligence, Multi-modal Large Language
  Models (MLLMs) have emerged as a focal point in recent advancements. However,
  the predominant focus remains on developing their capabilities in static image
  understanding. The potential of MLLMs in processing sequential visual data is
  still insufficiently explored, highlighting the absence of a comprehensive,
  high-quality assessment of their performance. In this paper, we introduce
  Video-MME, the first-ever full-spectrum, Multi-Modal Evaluation benchmark of
  MLLMs in Video analysis. Our work distinguishes from existing benchmarks
  through four key features: 1) Diversity in video types, spanning 6 primary
  visual domains with 30 subfields to ensure broad scenario generalizability; 2)
  Duration in temporal dimension, encompassing both short-, medium-, and
  long-term videos, ranging from 11 seconds to 1 hour, for robust contextual
  dynamics; 3) Breadth in data modalities, integrating multi-modal inputs besides
  video frames, including subtitles and audios, to unveil the all-round
  capabilities of MLLMs; 4) Quality in annotations, utilizing rigorous manual
  labeling by expert annotators to facilitate precise and reliable model
  assessment. 900 videos with a total of 256 hours are manually selected and
  annotated by repeatedly viewing all the video content, resulting in 2,700
  question-answer pairs. With Video-MME, we extensively evaluate various
  state-of-the-art MLLMs, including GPT-4 series and Gemini 1.5 Pro, as well as
  open-source image models like InternVL-Chat-V1.5 and video models like
  LLaVA-NeXT-Video. Our experiments reveal that Gemini 1.5 Pro is the
  best-performing commercial model, significantly outperforming the open-source
  models. Our dataset along with these findings underscores the need for further
  improvements in handling longer sequences and multi-modal data. Project Page:
  https://video-mme.github.io
paper_authors._id: 665d3a594dcff58f9a454a9d
paper_authors.name: Chaoyou Fu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-03T07:33:48.733Z
paper_authors._id: 653b268cd1041ca9188954da
paper_authors.avatarUrl: /avatars/2529a821580f5d87ee46867c5dc5b1e5.svg
paper_authors.fullname: BradyFU
paper_authors.type: user
paper_authors.user: BradyFU
paper_authors._id: 665d3a594dcff58f9a454a9e
paper_authors.name: Yuhan Dai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-03T07:32:32.502Z
paper_authors._id: 6314c9d18d85ad332fa2062a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6314c9d18d85ad332fa2062a/HvRSYOsGxdcYXAX_wen2l.jpeg
paper_authors.fullname: Yuhan Dai
paper_authors.type: user
paper_authors.user: dirkiedye
paper_authors._id: 665d3a594dcff58f9a454a9f
paper_authors.name: Yondong Luo
paper_authors._id: 665d3a594dcff58f9a454aa0
paper_authors.name: Lei Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-03T10:20:00.642Z
paper_authors._id: 6038d6d0612f5eef3cc05ea9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6038d6d0612f5eef3cc05ea9/ryhvAX5djQpD5OrIlZQ1f.jpeg
paper_authors.fullname: Lei Li
paper_authors.type: user
paper_authors.user: tobiaslee
paper_authors._id: 665d3a594dcff58f9a454aa1
paper_authors.name: Shuhuai Ren
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-03T07:05:29.809Z
paper_authors._id: 60d2e681b8448e1785bbda06
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1624434302056-noauth.jpeg
paper_authors.fullname: Shuhuai Ren
paper_authors.type: user
paper_authors.user: ShuhuaiRen
paper_authors._id: 665d3a594dcff58f9a454aa2
paper_authors.name: Renrui Zhang
paper_authors._id: 665d3a594dcff58f9a454aa3
paper_authors.name: Zihan Wang
paper_authors._id: 665d3a594dcff58f9a454aa4
paper_authors.name: Chenyu Zhou
paper_authors._id: 665d3a594dcff58f9a454aa5
paper_authors.name: Yunhang Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-03T08:24:04.429Z
paper_authors._id: 6483143902f98c3f05aff915
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6483143902f98c3f05aff915/ZhWFFgrlRsQf4MXiInh5p.jpeg
paper_authors.fullname: 沈云航 Yunhang Shen
paper_authors.type: user
paper_authors.user: shenyunhang
paper_authors._id: 665d3a594dcff58f9a454aa6
paper_authors.name: Mengdan Zhang
paper_authors._id: 665d3a594dcff58f9a454aa7
paper_authors.name: Peixian Chen
paper_authors._id: 665d3a594dcff58f9a454aa8
paper_authors.name: Yanwei Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-03T08:24:30.382Z
paper_authors._id: 643ff78cdc984afcbbbc3b1a
paper_authors.avatarUrl: /avatars/eec5198ce88aaf8156840bec0d190a7f.svg
paper_authors.fullname: Yanwei Li
paper_authors.type: user
paper_authors.user: YanweiLi
paper_authors._id: 665d3a594dcff58f9a454aa9
paper_authors.name: Shaohui Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-03T08:24:42.575Z
paper_authors._id: 6481ace6578646b5c2370b7e
paper_authors.avatarUrl: /avatars/218ffacc705c1fe0e23ef1217f76b374.svg
paper_authors.fullname: linshaohui
paper_authors.type: user
paper_authors.user: kimlin123
paper_authors._id: 665d3a594dcff58f9a454aaa
paper_authors.name: Sirui Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-03T08:24:48.975Z
paper_authors._id: 653a55ddae6a4080ce3eb860
paper_authors.avatarUrl: /avatars/44a62d1e0b72a42690d1658e5b42f3a4.svg
paper_authors.fullname: Sirui Zhao
paper_authors.type: user
paper_authors.user: SiruiZhao
paper_authors._id: 665d3a594dcff58f9a454aab
paper_authors.name: Ke Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-09T10:12:43.560Z
paper_authors._id: 63280915eeee4dd858083092
paper_authors.avatarUrl: /avatars/78347af4af42527d53e88d9969c5c934.svg
paper_authors.fullname: Ke Li
paper_authors.type: user
paper_authors.user: tristanli
paper_authors._id: 665d3a594dcff58f9a454aac
paper_authors.name: Tong Xu
paper_authors._id: 665d3a594dcff58f9a454aad
paper_authors.name: Xiawu Zheng
paper_authors._id: 665d3a594dcff58f9a454aae
paper_authors.name: Enhong Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-03T08:25:14.464Z
paper_authors._id: 64dde9ccd1885643ca14b2d4
paper_authors.avatarUrl: /avatars/04d83c2c7477212ad55eec24aa69eaa2.svg
paper_authors.fullname: EnHong Chen
paper_authors.type: user
paper_authors.user: Hongni
paper_authors._id: 665d3a594dcff58f9a454aaf
paper_authors.name: Rongrong Ji
paper_authors._id: 665d3a594dcff58f9a454ab0
paper_authors.name: Xing Sun
paper_title: Video-MME: The First-Ever Comprehensive Evaluation Benchmark of
    Multi-modal LLMs in Video Analysis
paper_summary: In the quest for artificial general intelligence, Multi-modal Large Language
  Models (MLLMs) have emerged as a focal point in recent advancements. However,
  the predominant focus remains on developing their capabilities in static image
  understanding. The potential of MLLMs in processing sequential visual data is
  still insufficiently explored, highlighting the absence of a comprehensive,
  high-quality assessment of their performance. In this paper, we introduce
  Video-MME, the first-ever full-spectrum, Multi-Modal Evaluation benchmark of
  MLLMs in Video analysis. Our work distinguishes from existing benchmarks
  through four key features: 1) Diversity in video types, spanning 6 primary
  visual domains with 30 subfields to ensure broad scenario generalizability; 2)
  Duration in temporal dimension, encompassing both short-, medium-, and
  long-term videos, ranging from 11 seconds to 1 hour, for robust contextual
  dynamics; 3) Breadth in data modalities, integrating multi-modal inputs besides
  video frames, including subtitles and audios, to unveil the all-round
  capabilities of MLLMs; 4) Quality in annotations, utilizing rigorous manual
  labeling by expert annotators to facilitate precise and reliable model
  assessment. 900 videos with a total of 256 hours are manually selected and
  annotated by repeatedly viewing all the video content, resulting in 2,700
  question-answer pairs. With Video-MME, we extensively evaluate various
  state-of-the-art MLLMs, including GPT-4 series and Gemini 1.5 Pro, as well as
  open-source image models like InternVL-Chat-V1.5 and video models like
  LLaVA-NeXT-Video. Our experiments reveal that Gemini 1.5 Pro is the
  best-performing commercial model, significantly outperforming the open-source
  models. Our dataset along with these findings underscores the need for further
  improvements in handling longer sequences and multi-modal data. Project Page:
  https://video-mme.github.io
paper_ai_summary: Video-MME is a comprehensive, high-quality benchmark that evaluates Multi-modal Large Language Models in video analysis, covering diverse video types, durations, modalities, and precise annotations.
paper_ai_keywords: Multi-modal Large Language Models Video-MME video analysis visual domains temporal dimension multi-modal inputs subtitles audios annotations state-of-the-art models GPT-4 Gemini 1.5 Pro InternVL-Chat-V1.5 LLaVA-NeXT-Video