title: Breaking Down Video LLM Benchmarks: Knowledge, Spatial Perception, or
    True Temporal Understanding?
summary: Existing video understanding benchmarks often conflate knowledge-based and
  purely image-based questions, rather than clearly isolating a model's temporal
  reasoning ability, which is the key aspect that distinguishes video
  understanding from other modalities. We identify two major limitations that
  obscure whether higher scores truly indicate stronger understanding of the
  dynamic content in videos: (1) strong language priors, where models can answer
  questions without watching the video; and (2) shuffling invariance, where
  models maintain similar performance on certain questions even when video frames
  are temporally shuffled. To alleviate these issues, we propose VBenchComp, an
  automated pipeline that categorizes questions into different domains:
  LLM-Answerable, Semantic, and Temporal. Specifically, LLM-Answerable questions
  can be answered without viewing the video; Semantic questions remain answerable
  even when the video frames are shuffled; and Temporal questions require
  understanding the correct temporal order of frames. The rest of the questions
  are labeled as Others. This can enable fine-grained evaluation of different
  capabilities of a video LLM. Our analysis reveals nuanced model weaknesses that
  are hidden by traditional overall scores, and we offer insights and
  recommendations for designing future benchmarks that more accurately assess
  video LLMs.
paper_authors._id: 68396605ac00da416d094bbf
paper_authors.name: Bo Feng
paper_authors._id: 68396605ac00da416d094bc0
paper_authors.name: Zhengfeng Lai
paper_authors._id: 68396605ac00da416d094bc1
paper_authors.name: Shiyu Li
paper_authors._id: 68396605ac00da416d094bc2
paper_authors.name: Zizhen Wang
paper_authors._id: 68396605ac00da416d094bc3
paper_authors.name: Simon Wang
paper_authors._id: 68396605ac00da416d094bc4
paper_authors.name: Ping Huang
paper_authors._id: 68396605ac00da416d094bc5
paper_authors.name: Meng Cao
paper_title: Breaking Down Video LLM Benchmarks: Knowledge, Spatial Perception, or
    True Temporal Understanding?
paper_summary: Existing video understanding benchmarks often conflate knowledge-based and
  purely image-based questions, rather than clearly isolating a model's temporal
  reasoning ability, which is the key aspect that distinguishes video
  understanding from other modalities. We identify two major limitations that
  obscure whether higher scores truly indicate stronger understanding of the
  dynamic content in videos: (1) strong language priors, where models can answer
  questions without watching the video; and (2) shuffling invariance, where
  models maintain similar performance on certain questions even when video frames
  are temporally shuffled. To alleviate these issues, we propose VBenchComp, an
  automated pipeline that categorizes questions into different domains:
  LLM-Answerable, Semantic, and Temporal. Specifically, LLM-Answerable questions
  can be answered without viewing the video; Semantic questions remain answerable
  even when the video frames are shuffled; and Temporal questions require
  understanding the correct temporal order of frames. The rest of the questions
  are labeled as Others. This can enable fine-grained evaluation of different
  capabilities of a video LLM. Our analysis reveals nuanced model weaknesses that
  are hidden by traditional overall scores, and we offer insights and
  recommendations for designing future benchmarks that more accurately assess
  video LLMs.
paper_ai_summary: VBenchComp, an automated pipeline, categorizes video LLM questions into different domains to evaluate temporal reasoning and isolate model weaknesses beyond overall scores.
paper_ai_keywords: video understanding LLM-Answerable Semantic Temporal