VideoEval-Pro: Robust and Realistic Long Video Understanding Evaluation
Large multimodal models (LMMs) have recently emerged as a powerful tool for
long video understanding (LVU), prompting the development of standardized LVU
benchmarks to evaluate their performance. However, our investigation reveals a
rather sober lesson for existing LVU benchmarks. First, most existing
benchmarks rely heavily on multiple-choice questions (MCQs), whose evaluation
results are inflated due to the possibility of guessing the correct answer;
Second, a significant portion of questions in these benchmarks have strong
priors to allow models to answer directly without even reading the input video.
For example, Gemini-1.5-Pro can achieve over 50\% accuracy given a random frame
from a long video on Video-MME. We also observe that increasing the number of
frames does not necessarily lead to improvement on existing benchmarks, which
is counterintuitive. As a result, the validity and robustness of current LVU
benchmarks are undermined, impeding a faithful assessment of LMMs' long-video
understanding capability. To tackle this problem, we propose VideoEval-Pro, a
realistic LVU benchmark containing questions with open-ended short-answer,
which truly require understanding the entire video. VideoEval-Pro assesses both
segment-level and full-video understanding through perception and reasoning
tasks. By evaluating 21 proprietary and open-source video LMMs, we conclude the
following findings: (1) video LMMs show drastic performance (>25\%) drops on
open-ended questions compared with MCQs; (2) surprisingly, higher MCQ scores do
not lead to higher open-ended scores on VideoEval-Pro; (3) compared to other
MCQ benchmarks, VideoEval-Pro benefits more from increasing the number of input
frames. Our results show that VideoEval-Pro offers a more realistic and
reliable measure of long video understanding, providing a clearer view of
progress in this domain.
682d488557686b8c44f257fa
Wentao Ma
claimed_verified
2025-05-21T08:40:31.165Z
65c387c807a1445dfe1e9452
https://cdn-avatars.huggingface.co/v1/production/uploads/65c387c807a1445dfe1e9452/NCsYIYrCQxQcPBiMe_4ZI.jpeg
Wentao Ma
user
tonymwt
682d488557686b8c44f257fb
Weiming Ren
claimed_verified
2025-05-21T08:40:28.689Z
64405a9d518271b0d1beea38
/avatars/b702474588fd7090773320422417a582.svg
Weiming Ren
user
wren93
682d488557686b8c44f257fc
Yiming Jia
682d488557686b8c44f257fd
Zhuofeng Li
claimed_verified
2025-05-21T08:40:35.151Z
66349404f2c753240d02952a
/avatars/4f207cf5807d9629b9f4f7d13875b840.svg
ZhuofengLi
user
ZhuofengLi
682d488557686b8c44f257fe
Ping Nie
682d488557686b8c44f257ff
Ge Zhang
claimed_verified
2025-06-18T12:17:46.529Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
682d488557686b8c44f25800
Wenhu Chen
VideoEval-Pro: Robust and Realistic Long Video Understanding Evaluation
Large multimodal models (LMMs) have recently emerged as a powerful tool for
long video understanding (LVU), prompting the development of standardized LVU
benchmarks to evaluate their performance. However, our investigation reveals a
rather sober lesson for existing LVU benchmarks. First, most existing
benchmarks rely heavily on multiple-choice questions (MCQs), whose evaluation
results are inflated due to the possibility of guessing the correct answer;
Second, a significant portion of questions in these benchmarks have strong
priors to allow models to answer directly without even reading the input video.
For example, Gemini-1.5-Pro can achieve over 50\% accuracy given a random frame
from a long video on Video-MME. We also observe that increasing the number of
frames does not necessarily lead to improvement on existing benchmarks, which
is counterintuitive. As a result, the validity and robustness of current LVU
benchmarks are undermined, impeding a faithful assessment of LMMs' long-video
understanding capability. To tackle this problem, we propose VideoEval-Pro, a
realistic LVU benchmark containing questions with open-ended short-answer,
which truly require understanding the entire video. VideoEval-Pro assesses both
segment-level and full-video understanding through perception and reasoning
tasks. By evaluating 21 proprietary and open-source video LMMs, we conclude the
following findings: (1) video LMMs show drastic performance (>25\%) drops on
open-ended questions compared with MCQs; (2) surprisingly, higher MCQ scores do
not lead to higher open-ended scores on VideoEval-Pro; (3) compared to other
MCQ benchmarks, VideoEval-Pro benefits more from increasing the number of input
frames. Our results show that VideoEval-Pro offers a more realistic and
reliable measure of long video understanding, providing a clearer view of
progress in this domain.
VideoEval-Pro, a benchmark using open-ended questions, provides a more accurate measure of long video understanding compared to existing multiple-choice question benchmarks.
large multimodal models long video understanding LVU benchmarks multiple-choice questions Video-MME VideoEval-Pro segment-level understanding full-video understanding perception tasks reasoning tasks