Video-MME: The First-Ever Comprehensive Evaluation Benchmark of
  Multi-modal LLMs in Video Analysis
In the quest for artificial general intelligence, Multi-modal Large Language
Models (MLLMs) have emerged as a focal point in recent advancements. However,
the predominant focus remains on developing their capabilities in static image
understanding. The potential of MLLMs in processing sequential visual data is
still insufficiently explored, highlighting the absence of a comprehensive,
high-quality assessment of their performance. In this paper, we introduce
Video-MME, the first-ever full-spectrum, Multi-Modal Evaluation benchmark of
MLLMs in Video analysis. Our work distinguishes from existing benchmarks
through four key features: 1) Diversity in video types, spanning 6 primary
visual domains with 30 subfields to ensure broad scenario generalizability; 2)
Duration in temporal dimension, encompassing both short-, medium-, and
long-term videos, ranging from 11 seconds to 1 hour, for robust contextual
dynamics; 3) Breadth in data modalities, integrating multi-modal inputs besides
video frames, including subtitles and audios, to unveil the all-round
capabilities of MLLMs; 4) Quality in annotations, utilizing rigorous manual
labeling by expert annotators to facilitate precise and reliable model
assessment. 900 videos with a total of 256 hours are manually selected and
annotated by repeatedly viewing all the video content, resulting in 2,700
question-answer pairs. With Video-MME, we extensively evaluate various
state-of-the-art MLLMs, including GPT-4 series and Gemini 1.5 Pro, as well as
open-source image models like InternVL-Chat-V1.5 and video models like
LLaVA-NeXT-Video. Our experiments reveal that Gemini 1.5 Pro is the
best-performing commercial model, significantly outperforming the open-source
models. Our dataset along with these findings underscores the need for further
improvements in handling longer sequences and multi-modal data. Project Page:
https://video-mme.github.io
665d3a594dcff58f9a454a9d
Chaoyou Fu
admin_assigned
2024-06-03T07:33:48.733Z
653b268cd1041ca9188954da
/avatars/2529a821580f5d87ee46867c5dc5b1e5.svg
BradyFU
user
BradyFU
665d3a594dcff58f9a454a9e
Yuhan Dai
admin_assigned
2024-06-03T07:32:32.502Z
6314c9d18d85ad332fa2062a
https://cdn-avatars.huggingface.co/v1/production/uploads/6314c9d18d85ad332fa2062a/HvRSYOsGxdcYXAX_wen2l.jpeg
Yuhan Dai
user
dirkiedye
665d3a594dcff58f9a454a9f
Yondong Luo
665d3a594dcff58f9a454aa0
Lei Li
admin_assigned
2024-06-03T10:20:00.642Z
6038d6d0612f5eef3cc05ea9
https://cdn-avatars.huggingface.co/v1/production/uploads/6038d6d0612f5eef3cc05ea9/ryhvAX5djQpD5OrIlZQ1f.jpeg
Lei Li
user
tobiaslee
665d3a594dcff58f9a454aa1
Shuhuai Ren
claimed_verified
2024-06-03T07:05:29.809Z
60d2e681b8448e1785bbda06
https://cdn-avatars.huggingface.co/v1/production/uploads/1624434302056-noauth.jpeg
Shuhuai Ren
user
ShuhuaiRen
665d3a594dcff58f9a454aa2
Renrui Zhang
665d3a594dcff58f9a454aa3
Zihan Wang
665d3a594dcff58f9a454aa4
Chenyu Zhou
665d3a594dcff58f9a454aa5
Yunhang Shen
admin_assigned
2024-06-03T08:24:04.429Z
6483143902f98c3f05aff915
https://cdn-avatars.huggingface.co/v1/production/uploads/6483143902f98c3f05aff915/ZhWFFgrlRsQf4MXiInh5p.jpeg
沈云航 Yunhang Shen
user
shenyunhang
665d3a594dcff58f9a454aa6
Mengdan Zhang
665d3a594dcff58f9a454aa7
Peixian Chen
665d3a594dcff58f9a454aa8
Yanwei Li
admin_assigned
2024-06-03T08:24:30.382Z
643ff78cdc984afcbbbc3b1a
/avatars/eec5198ce88aaf8156840bec0d190a7f.svg
Yanwei Li
user
YanweiLi
665d3a594dcff58f9a454aa9
Shaohui Lin
admin_assigned
2024-06-03T08:24:42.575Z
6481ace6578646b5c2370b7e
/avatars/218ffacc705c1fe0e23ef1217f76b374.svg
linshaohui
user
kimlin123
665d3a594dcff58f9a454aaa
Sirui Zhao
admin_assigned
2024-06-03T08:24:48.975Z
653a55ddae6a4080ce3eb860
/avatars/44a62d1e0b72a42690d1658e5b42f3a4.svg
Sirui Zhao
user
SiruiZhao
665d3a594dcff58f9a454aab
Ke Li
claimed_verified
2025-01-09T10:12:43.560Z
63280915eeee4dd858083092
/avatars/78347af4af42527d53e88d9969c5c934.svg
Ke Li
user
tristanli
665d3a594dcff58f9a454aac
Tong Xu
665d3a594dcff58f9a454aad
Xiawu Zheng
665d3a594dcff58f9a454aae
Enhong Chen
admin_assigned
2024-06-03T08:25:14.464Z
64dde9ccd1885643ca14b2d4
/avatars/04d83c2c7477212ad55eec24aa69eaa2.svg
EnHong Chen
user
Hongni
665d3a594dcff58f9a454aaf
Rongrong Ji
665d3a594dcff58f9a454ab0
Xing Sun
Video-MME: The First-Ever Comprehensive Evaluation Benchmark of
  Multi-modal LLMs in Video Analysis
In the quest for artificial general intelligence, Multi-modal Large Language
Models (MLLMs) have emerged as a focal point in recent advancements. However,
the predominant focus remains on developing their capabilities in static image
understanding. The potential of MLLMs in processing sequential visual data is
still insufficiently explored, highlighting the absence of a comprehensive,
high-quality assessment of their performance. In this paper, we introduce
Video-MME, the first-ever full-spectrum, Multi-Modal Evaluation benchmark of
MLLMs in Video analysis. Our work distinguishes from existing benchmarks
through four key features: 1) Diversity in video types, spanning 6 primary
visual domains with 30 subfields to ensure broad scenario generalizability; 2)
Duration in temporal dimension, encompassing both short-, medium-, and
long-term videos, ranging from 11 seconds to 1 hour, for robust contextual
dynamics; 3) Breadth in data modalities, integrating multi-modal inputs besides
video frames, including subtitles and audios, to unveil the all-round
capabilities of MLLMs; 4) Quality in annotations, utilizing rigorous manual
labeling by expert annotators to facilitate precise and reliable model
assessment. 900 videos with a total of 256 hours are manually selected and
annotated by repeatedly viewing all the video content, resulting in 2,700
question-answer pairs. With Video-MME, we extensively evaluate various
state-of-the-art MLLMs, including GPT-4 series and Gemini 1.5 Pro, as well as
open-source image models like InternVL-Chat-V1.5 and video models like
LLaVA-NeXT-Video. Our experiments reveal that Gemini 1.5 Pro is the
best-performing commercial model, significantly outperforming the open-source
models. Our dataset along with these findings underscores the need for further
improvements in handling longer sequences and multi-modal data. Project Page:
https://video-mme.github.io
Video-MME is a comprehensive, high-quality benchmark that evaluates Multi-modal Large Language Models in video analysis, covering diverse video types, durations, modalities, and precise annotations.
Multi-modal Large Language Models Video-MME video analysis visual domains temporal dimension multi-modal inputs subtitles audios annotations state-of-the-art models GPT-4 Gemini 1.5 Pro InternVL-Chat-V1.5 LLaVA-NeXT-Video