title: TVBench: Redesigning Video-Language Evaluation
summary: Large language models have demonstrated impressive performance when
  integrated with vision models even enabling video understanding. However,
  evaluating these video models presents its own unique challenges, for which
  several benchmarks have been proposed. In this paper, we show that the
  currently most used video-language benchmarks can be solved without requiring
  much temporal reasoning. We identified three main issues in existing datasets:
  (i) static information from single frames is often sufficient to solve the
  tasks (ii) the text of the questions and candidate answers is overly
  informative, allowing models to answer correctly without relying on any visual
  input (iii) world knowledge alone can answer many of the questions, making the
  benchmarks a test of knowledge replication rather than visual reasoning. In
  addition, we found that open-ended question-answering benchmarks for video
  understanding suffer from similar issues while the automatic evaluation process
  with LLMs is unreliable, making it an unsuitable alternative. As a solution, we
  propose TVBench, a novel open-source video multiple-choice question-answering
  benchmark, and demonstrate through extensive evaluations that it requires a
  high level of temporal understanding. Surprisingly, we find that most recent
  state-of-the-art video-language models perform similarly to random performance
  on TVBench, with only Gemini-Pro and Tarsier clearly surpassing this baseline.
paper_authors._id: 6708e94c1f9b09c1ab200e8a
paper_authors.name: Daniel Cores
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T09:11:20.594Z
paper_authors._id: 6616922bff43b65a8ece7ccc
paper_authors.avatarUrl: /avatars/983799cedcad7c75a57f215630e7ab4a.svg
paper_authors.fullname: Daniel Cores
paper_authors.type: user
paper_authors.user: dcores
paper_authors._id: 6708e94c1f9b09c1ab200e8b
paper_authors.name: Michael Dorkenwald
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T12:16:07.727Z
paper_authors._id: 6566436e3d526e3297c1b4a1
paper_authors.avatarUrl: /avatars/e433ffccb0f9c2d91649f9988655fedd.svg
paper_authors.fullname: Michael Dorkenwald
paper_authors.type: user
paper_authors.user: mdorkenw
paper_authors._id: 6708e94c1f9b09c1ab200e8c
paper_authors.name: Manuel Mucientes
paper_authors._id: 6708e94c1f9b09c1ab200e8d
paper_authors.name: Cees G. M. Snoek
paper_authors._id: 6708e94c1f9b09c1ab200e8e
paper_authors.name: Yuki M. Asano
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T09:11:22.379Z
paper_authors._id: 637d21239a5217b88b7549c3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637d21239a5217b88b7549c3/LrIGPiva5VGVZG87rTAJz.jpeg
paper_authors.fullname: Yuki Asano
paper_authors.type: user
paper_authors.user: yukimasano
paper_title: TVBench: Redesigning Video-Language Evaluation
paper_summary: Large language models have demonstrated impressive performance when
  integrated with vision models even enabling video understanding. However,
  evaluating these video models presents its own unique challenges, for which
  several benchmarks have been proposed. In this paper, we show that the
  currently most used video-language benchmarks can be solved without requiring
  much temporal reasoning. We identified three main issues in existing datasets:
  (i) static information from single frames is often sufficient to solve the
  tasks (ii) the text of the questions and candidate answers is overly
  informative, allowing models to answer correctly without relying on any visual
  input (iii) world knowledge alone can answer many of the questions, making the
  benchmarks a test of knowledge replication rather than visual reasoning. In
  addition, we found that open-ended question-answering benchmarks for video
  understanding suffer from similar issues while the automatic evaluation process
  with LLMs is unreliable, making it an unsuitable alternative. As a solution, we
  propose TVBench, a novel open-source video multiple-choice question-answering
  benchmark, and demonstrate through extensive evaluations that it requires a
  high level of temporal understanding. Surprisingly, we find that most recent
  state-of-the-art video-language models perform similarly to random performance
  on TVBench, with only Gemini-Pro and Tarsier clearly surpassing this baseline.
paper_ai_summary: A new video-language benchmark, TVBench, is introduced to require high-level temporal understanding in video question-answering, showing that most state-of-the-art models do not perform well on this task.
paper_ai_keywords: video-language models video understanding temporal reasoning TVBench open-source benchmark multiple-choice question-answering Gemini-Pro Tarsier