title: Video SimpleQA: Towards Factuality Evaluation in Large Video Language
    Models
summary: Recent advancements in Large Video Language Models (LVLMs) have highlighted
  their potential for multi-modal understanding, yet evaluating their factual
  grounding in video contexts remains a critical unsolved challenge. To address
  this gap, we introduce Video SimpleQA, the first comprehensive benchmark
  tailored for factuality evaluation of LVLMs. Our work distinguishes from
  existing video benchmarks through the following key features: 1) Knowledge
  required: demanding integration of external knowledge beyond the explicit
  narrative; 2) Fact-seeking question: targeting objective, undisputed events or
  relationships, avoiding subjective interpretation; 3) Definitive & short-form
  answer: Answers are crafted as unambiguous and definitively correct in a short
  format, enabling automated evaluation through LLM-as-a-judge frameworks with
  minimal scoring variance; 4) External-source verified: All annotations undergo
  rigorous validation against authoritative external references to ensure the
  reliability; 5) Temporal reasoning required: The annotated question types
  encompass both static single-frame understanding and dynamic temporal
  reasoning, explicitly evaluating LVLMs factuality under the long-context
  dependencies. We extensively evaluate 41 state-of-the-art LVLMs and summarize
  key findings as follows: 1) Current LVLMs exhibit notable deficiencies in
  factual adherence, particularly for open-source models. The best-performing
  model Gemini-1.5-Pro achieves merely an F-score of 54.4%; 2) Test-time compute
  paradigms show insignificant performance gains, revealing fundamental
  constraints for enhancing factuality through post-hoc computation; 3)
  Retrieval-Augmented Generation demonstrates consistent improvements at the cost
  of additional inference time overhead, presenting a critical
  efficiency-performance trade-off.
paper_authors._id: 67e226f401cdb8cf3a1c7cd8
paper_authors.name: Meng Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:22:38.738Z
paper_authors._id: 640222f83e3d0f2745b097b2
paper_authors.avatarUrl: /avatars/c5dbac84734855369a7f57b051f16caa.svg
paper_authors.fullname: Meng Cao
paper_authors.type: user
paper_authors.user: mengcao
paper_authors._id: 67e226f401cdb8cf3a1c7cd9
paper_authors.name: Pengfei Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-31T08:15:11.513Z
paper_authors._id: 67597dfd8cd6382952537ddb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/67597dfd8cd6382952537ddb/veH232xoIvxep7vpTtR8m.jpeg
paper_authors.fullname: Pengfei Hu
paper_authors.type: user
paper_authors.user: PengfeiHu
paper_authors._id: 67e226f401cdb8cf3a1c7cda
paper_authors.name: Yingyao Wang
paper_authors._id: 67e226f401cdb8cf3a1c7cdb
paper_authors.name: Jihao Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:22:23.718Z
paper_authors._id: 65733c1b244aefdfc45cc771
paper_authors.avatarUrl: /avatars/7223cedbeed065c28a400e130cea30ae.svg
paper_authors.fullname: Jihao Guo
paper_authors.type: user
paper_authors.user: grejioh
paper_authors._id: 67e226f401cdb8cf3a1c7cdc
paper_authors.name: Haoran Tang
paper_authors._id: 67e226f401cdb8cf3a1c7cdd
paper_authors.name: Haoze Zhao
paper_authors._id: 67e226f401cdb8cf3a1c7cde
paper_authors.name: Jiahua Dong
paper_authors._id: 67e226f401cdb8cf3a1c7cdf
paper_authors.name: Wangbo Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:23:02.455Z
paper_authors._id: 63f095be6309c84d5f48848a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63f095be6309c84d5f48848a/pL2CKi-r-0mMfIGhYSAsm.jpeg
paper_authors.fullname: Wangbo Yu
paper_authors.type: user
paper_authors.user: Drexubery
paper_authors._id: 67e226f401cdb8cf3a1c7ce0
paper_authors.name: Ge Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:22:08.334Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 67e226f401cdb8cf3a1c7ce1
paper_authors.name: Ian Reid
paper_authors._id: 67e226f401cdb8cf3a1c7ce2
paper_authors.name: Xiaodan Liang
paper_title: Video SimpleQA: Towards Factuality Evaluation in Large Video Language
    Models
paper_summary: Recent advancements in Large Video Language Models (LVLMs) have highlighted
  their potential for multi-modal understanding, yet evaluating their factual
  grounding in video contexts remains a critical unsolved challenge. To address
  this gap, we introduce Video SimpleQA, the first comprehensive benchmark
  tailored for factuality evaluation of LVLMs. Our work distinguishes from
  existing video benchmarks through the following key features: 1) Knowledge
  required: demanding integration of external knowledge beyond the explicit
  narrative; 2) Fact-seeking question: targeting objective, undisputed events or
  relationships, avoiding subjective interpretation; 3) Definitive & short-form
  answer: Answers are crafted as unambiguous and definitively correct in a short
  format, enabling automated evaluation through LLM-as-a-judge frameworks with
  minimal scoring variance; 4) External-source verified: All annotations undergo
  rigorous validation against authoritative external references to ensure the
  reliability; 5) Temporal reasoning required: The annotated question types
  encompass both static single-frame understanding and dynamic temporal
  reasoning, explicitly evaluating LVLMs factuality under the long-context
  dependencies. We extensively evaluate 41 state-of-the-art LVLMs and summarize
  key findings as follows: 1) Current LVLMs exhibit notable deficiencies in
  factual adherence, particularly for open-source models. The best-performing
  model Gemini-1.5-Pro achieves merely an F-score of 54.4%; 2) Test-time compute
  paradigms show insignificant performance gains, revealing fundamental
  constraints for enhancing factuality through post-hoc computation; 3)
  Retrieval-Augmented Generation demonstrates consistent improvements at the cost
  of additional inference time overhead, presenting a critical
  efficiency-performance trade-off.
paper_ai_summary: Video SimpleQA evaluates the factual grounding of LVLMs in videos through a benchmark requiring external knowledge integration, fact-seeking questions, and temporal reasoning.
paper_ai_keywords: Large Video Language Models LVLMs Video SimpleQA factual grounding knowledge integration fact-seeking questions short-form answers external-source verification temporal reasoning LLM-as-a-judge Gemini-1.5-Pro F-score Retrieval-Augmented Generation