E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding
Recent advances in Video Large Language Models (Video-LLMs) have demonstrated
their great potential in general-purpose video understanding. To verify the
significance of these models, a number of benchmarks have been proposed to
diagnose their capabilities in different scenarios. However, existing
benchmarks merely evaluate models through video-level question-answering,
lacking fine-grained event-level assessment and task diversity. To fill this
gap, we introduce E.T. Bench (Event-Level & Time-Sensitive Video Understanding
Benchmark), a large-scale and high-quality benchmark for open-ended event-level
video understanding. Categorized within a 3-level task taxonomy, E.T. Bench
encompasses 7.3K samples under 12 tasks with 7K videos (251.4h total length)
under 8 domains, providing comprehensive evaluations. We extensively evaluated
8 Image-LLMs and 12 Video-LLMs on our benchmark, and the results reveal that
state-of-the-art models for coarse-level (video-level) understanding struggle
to solve our fine-grained tasks, e.g., grounding event-of-interests within
videos, largely due to the short video context length, improper time
representations, and lack of multi-event training data. Focusing on these
issues, we further propose a strong baseline model, E.T. Chat, together with an
instruction-tuning dataset E.T. Instruct 164K tailored for fine-grained
event-level understanding. Our simple but effective solution demonstrates
superior performance in multiple scenarios.
66f703ea91c723ca3194dcca
Ye Liu
claimed_verified
2024-11-05T07:58:52.585Z
6250eb5c1fd03f78d0ae550f
https://cdn-avatars.huggingface.co/v1/production/uploads/1649470297585-noauth.jpeg
Ye Liu
user
yeliudev
66f703ea91c723ca3194dccb
Zongyang Ma
admin_assigned
2024-10-09T13:34:04.459Z
61bb00f6c4ac95d207b25f1b
/avatars/3b6eba701d64518d6f694942f5b2e9a9.svg
Zongyang Ma
user
zyma
66f703ea91c723ca3194dccc
Zhongang Qi
claimed_verified
2024-11-19T08:40:07.402Z
660cc64f1b14f691990c0ea0
/avatars/f172d3120b22d745a41cc3f2eb499ce6.svg
Zhongang Qi
user
phoenixqza
66f703ea91c723ca3194dccd
Yang Wu
66f703ea91c723ca3194dcce
Ying Shan
66f703ea91c723ca3194dccf
Chang Wen Chen
E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding
Recent advances in Video Large Language Models (Video-LLMs) have demonstrated
their great potential in general-purpose video understanding. To verify the
significance of these models, a number of benchmarks have been proposed to
diagnose their capabilities in different scenarios. However, existing
benchmarks merely evaluate models through video-level question-answering,
lacking fine-grained event-level assessment and task diversity. To fill this
gap, we introduce E.T. Bench (Event-Level & Time-Sensitive Video Understanding
Benchmark), a large-scale and high-quality benchmark for open-ended event-level
video understanding. Categorized within a 3-level task taxonomy, E.T. Bench
encompasses 7.3K samples under 12 tasks with 7K videos (251.4h total length)
under 8 domains, providing comprehensive evaluations. We extensively evaluated
8 Image-LLMs and 12 Video-LLMs on our benchmark, and the results reveal that
state-of-the-art models for coarse-level (video-level) understanding struggle
to solve our fine-grained tasks, e.g., grounding event-of-interests within
videos, largely due to the short video context length, improper time
representations, and lack of multi-event training data. Focusing on these
issues, we further propose a strong baseline model, E.T. Chat, together with an
instruction-tuning dataset E.T. Instruct 164K tailored for fine-grained
event-level understanding. Our simple but effective solution demonstrates
superior performance in multiple scenarios.
E.T. Bench is a comprehensive benchmark for event-level video understanding, highlighting challenges in current models and introducing a strong baseline solution.
Video Large Language Models Video-LLMs Image-LLMs video-level question-answering event-level assessment task diversity Event-Level & Time-Sensitive Video Understanding Benchmark E.T. Bench task taxonomy grounding event-of-interests time representations multi-event training data E.T. Chat E.T. Instruct