title: VideoEval-Pro: Robust and Realistic Long Video Understanding Evaluation
summary: Large multimodal models (LMMs) have recently emerged as a powerful tool for
  long video understanding (LVU), prompting the development of standardized LVU
  benchmarks to evaluate their performance. However, our investigation reveals a
  rather sober lesson for existing LVU benchmarks. First, most existing
  benchmarks rely heavily on multiple-choice questions (MCQs), whose evaluation
  results are inflated due to the possibility of guessing the correct answer;
  Second, a significant portion of questions in these benchmarks have strong
  priors to allow models to answer directly without even reading the input video.
  For example, Gemini-1.5-Pro can achieve over 50\% accuracy given a random frame
  from a long video on Video-MME. We also observe that increasing the number of
  frames does not necessarily lead to improvement on existing benchmarks, which
  is counterintuitive. As a result, the validity and robustness of current LVU
  benchmarks are undermined, impeding a faithful assessment of LMMs' long-video
  understanding capability. To tackle this problem, we propose VideoEval-Pro, a
  realistic LVU benchmark containing questions with open-ended short-answer,
  which truly require understanding the entire video. VideoEval-Pro assesses both
  segment-level and full-video understanding through perception and reasoning
  tasks. By evaluating 21 proprietary and open-source video LMMs, we conclude the
  following findings: (1) video LMMs show drastic performance (>25\%) drops on
  open-ended questions compared with MCQs; (2) surprisingly, higher MCQ scores do
  not lead to higher open-ended scores on VideoEval-Pro; (3) compared to other
  MCQ benchmarks, VideoEval-Pro benefits more from increasing the number of input
  frames. Our results show that VideoEval-Pro offers a more realistic and
  reliable measure of long video understanding, providing a clearer view of
  progress in this domain.
paper_authors._id: 682d488557686b8c44f257fa
paper_authors.name: Wentao Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-21T08:40:31.165Z
paper_authors._id: 65c387c807a1445dfe1e9452
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65c387c807a1445dfe1e9452/t0VnwQh2wRZ9W_UGTZ8zt.jpeg
paper_authors.fullname: Wentao Ma
paper_authors.type: user
paper_authors.user: tonymwt
paper_authors._id: 682d488557686b8c44f257fb
paper_authors.name: Weiming Ren
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-21T08:40:28.689Z
paper_authors._id: 64405a9d518271b0d1beea38
paper_authors.avatarUrl: /avatars/b702474588fd7090773320422417a582.svg
paper_authors.fullname: Weiming Ren
paper_authors.type: user
paper_authors.user: wren93
paper_authors._id: 682d488557686b8c44f257fc
paper_authors.name: Yiming Jia
paper_authors._id: 682d488557686b8c44f257fd
paper_authors.name: Zhuofeng Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-21T08:40:35.151Z
paper_authors._id: 66349404f2c753240d02952a
paper_authors.avatarUrl: /avatars/4f207cf5807d9629b9f4f7d13875b840.svg
paper_authors.fullname: ZhuofengLi
paper_authors.type: user
paper_authors.user: ZhuofengLi
paper_authors._id: 682d488557686b8c44f257fe
paper_authors.name: Ping Nie
paper_authors._id: 682d488557686b8c44f257ff
paper_authors.name: Ge Zhang
paper_authors._id: 682d488557686b8c44f25800
paper_authors.name: Wenhu Chen
paper_title: VideoEval-Pro: Robust and Realistic Long Video Understanding Evaluation
paper_summary: Large multimodal models (LMMs) have recently emerged as a powerful tool for
  long video understanding (LVU), prompting the development of standardized LVU
  benchmarks to evaluate their performance. However, our investigation reveals a
  rather sober lesson for existing LVU benchmarks. First, most existing
  benchmarks rely heavily on multiple-choice questions (MCQs), whose evaluation
  results are inflated due to the possibility of guessing the correct answer;
  Second, a significant portion of questions in these benchmarks have strong
  priors to allow models to answer directly without even reading the input video.
  For example, Gemini-1.5-Pro can achieve over 50\% accuracy given a random frame
  from a long video on Video-MME. We also observe that increasing the number of
  frames does not necessarily lead to improvement on existing benchmarks, which
  is counterintuitive. As a result, the validity and robustness of current LVU
  benchmarks are undermined, impeding a faithful assessment of LMMs' long-video
  understanding capability. To tackle this problem, we propose VideoEval-Pro, a
  realistic LVU benchmark containing questions with open-ended short-answer,
  which truly require understanding the entire video. VideoEval-Pro assesses both
  segment-level and full-video understanding through perception and reasoning
  tasks. By evaluating 21 proprietary and open-source video LMMs, we conclude the
  following findings: (1) video LMMs show drastic performance (>25\%) drops on
  open-ended questions compared with MCQs; (2) surprisingly, higher MCQ scores do
  not lead to higher open-ended scores on VideoEval-Pro; (3) compared to other
  MCQ benchmarks, VideoEval-Pro benefits more from increasing the number of input
  frames. Our results show that VideoEval-Pro offers a more realistic and
  reliable measure of long video understanding, providing a clearer view of
  progress in this domain.
paper_ai_summary: VideoEval-Pro, a benchmark using open-ended questions, provides a more accurate measure of long video understanding compared to existing multiple-choice question benchmarks.
paper_ai_keywords: large multimodal models long video understanding LVU benchmarks multiple-choice questions Video-MME VideoEval-Pro segment-level understanding full-video understanding perception tasks reasoning tasks