title: E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding
summary: Recent advances in Video Large Language Models (Video-LLMs) have demonstrated
  their great potential in general-purpose video understanding. To verify the
  significance of these models, a number of benchmarks have been proposed to
  diagnose their capabilities in different scenarios. However, existing
  benchmarks merely evaluate models through video-level question-answering,
  lacking fine-grained event-level assessment and task diversity. To fill this
  gap, we introduce E.T. Bench (Event-Level & Time-Sensitive Video Understanding
  Benchmark), a large-scale and high-quality benchmark for open-ended event-level
  video understanding. Categorized within a 3-level task taxonomy, E.T. Bench
  encompasses 7.3K samples under 12 tasks with 7K videos (251.4h total length)
  under 8 domains, providing comprehensive evaluations. We extensively evaluated
  8 Image-LLMs and 12 Video-LLMs on our benchmark, and the results reveal that
  state-of-the-art models for coarse-level (video-level) understanding struggle
  to solve our fine-grained tasks, e.g., grounding event-of-interests within
  videos, largely due to the short video context length, improper time
  representations, and lack of multi-event training data. Focusing on these
  issues, we further propose a strong baseline model, E.T. Chat, together with an
  instruction-tuning dataset E.T. Instruct 164K tailored for fine-grained
  event-level understanding. Our simple but effective solution demonstrates
  superior performance in multiple scenarios.
paper_authors._id: 66f703ea91c723ca3194dcca
paper_authors.name: Ye Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-05T07:58:52.585Z
paper_authors._id: 6250eb5c1fd03f78d0ae550f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1649470297585-noauth.jpeg
paper_authors.fullname: Ye Liu
paper_authors.type: user
paper_authors.user: yeliudev
paper_authors._id: 66f703ea91c723ca3194dccb
paper_authors.name: Zongyang Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-09T13:34:04.459Z
paper_authors._id: 61bb00f6c4ac95d207b25f1b
paper_authors.avatarUrl: /avatars/3b6eba701d64518d6f694942f5b2e9a9.svg
paper_authors.fullname: Zongyang Ma
paper_authors.type: user
paper_authors.user: zyma
paper_authors._id: 66f703ea91c723ca3194dccc
paper_authors.name: Zhongang Qi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-19T08:40:07.402Z
paper_authors._id: 660cc64f1b14f691990c0ea0
paper_authors.avatarUrl: /avatars/f172d3120b22d745a41cc3f2eb499ce6.svg
paper_authors.fullname: Zhongang Qi
paper_authors.type: user
paper_authors.user: phoenixqza
paper_authors._id: 66f703ea91c723ca3194dccd
paper_authors.name: Yang Wu
paper_authors._id: 66f703ea91c723ca3194dcce
paper_authors.name: Ying Shan
paper_authors._id: 66f703ea91c723ca3194dccf
paper_authors.name: Chang Wen Chen
paper_title: E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding
paper_summary: Recent advances in Video Large Language Models (Video-LLMs) have demonstrated
  their great potential in general-purpose video understanding. To verify the
  significance of these models, a number of benchmarks have been proposed to
  diagnose their capabilities in different scenarios. However, existing
  benchmarks merely evaluate models through video-level question-answering,
  lacking fine-grained event-level assessment and task diversity. To fill this
  gap, we introduce E.T. Bench (Event-Level & Time-Sensitive Video Understanding
  Benchmark), a large-scale and high-quality benchmark for open-ended event-level
  video understanding. Categorized within a 3-level task taxonomy, E.T. Bench
  encompasses 7.3K samples under 12 tasks with 7K videos (251.4h total length)
  under 8 domains, providing comprehensive evaluations. We extensively evaluated
  8 Image-LLMs and 12 Video-LLMs on our benchmark, and the results reveal that
  state-of-the-art models for coarse-level (video-level) understanding struggle
  to solve our fine-grained tasks, e.g., grounding event-of-interests within
  videos, largely due to the short video context length, improper time
  representations, and lack of multi-event training data. Focusing on these
  issues, we further propose a strong baseline model, E.T. Chat, together with an
  instruction-tuning dataset E.T. Instruct 164K tailored for fine-grained
  event-level understanding. Our simple but effective solution demonstrates
  superior performance in multiple scenarios.
paper_ai_summary: E.T. Bench is a comprehensive benchmark for event-level video understanding, highlighting challenges in current models and introducing a strong baseline solution.
paper_ai_keywords: Video Large Language Models Video-LLMs Image-LLMs video-level question-answering event-level assessment task diversity Event-Level & Time-Sensitive Video Understanding Benchmark E.T. Bench task taxonomy grounding event-of-interests time representations multi-event training data E.T. Chat E.T. Instruct