ShotBench: Expert-Level Cinematic Understanding in Vision-Language
  Models
Cinematography, the fundamental visual language of film, is essential for
conveying narrative, emotion, and aesthetic quality. While recent
Vision-Language Models (VLMs) demonstrate strong general visual understanding,
their proficiency in comprehending the nuanced cinematic grammar embedded
within individual shots remains largely unexplored and lacks robust evaluation.
This critical gap limits both fine-grained visual comprehension and the
precision of AI-assisted video generation. To address this, we introduce
ShotBench, a comprehensive benchmark specifically designed for cinematic
language understanding. It features over 3.5k expert-annotated QA pairs from
images and video clips, meticulously curated from over 200 acclaimed
(predominantly Oscar-nominated) films and spanning eight key cinematography
dimensions. Our evaluation of 24 leading VLMs on ShotBench reveals their
substantial limitations: even the top-performing model achieves less than 60%
average accuracy, particularly struggling with fine-grained visual cues and
complex spatial reasoning. To catalyze advancement in this domain, we construct
ShotQA, a large-scale multimodal dataset comprising approximately 70k cinematic
QA pairs. Leveraging ShotQA, we develop ShotVL through supervised fine-tuning
and Group Relative Policy Optimization. ShotVL significantly outperforms all
existing open-source and proprietary models on ShotBench, establishing new
state-of-the-art performance. We open-source our models, data, and code to
foster rapid progress in this crucial area of AI-driven cinematic understanding
and generation.
6861fb7a9e7509383d29ab4b
Hongbo Liu
6861fb7a9e7509383d29ab4c
Jingwen He
6861fb7a9e7509383d29ab4d
Yi Jin
6861fb7a9e7509383d29ab4e
Dian Zheng
6861fb7a9e7509383d29ab4f
Yuhao Dong
claimed_verified
2025-07-10T07:10:13.594Z
652965773a416e1f2173443b
https://cdn-avatars.huggingface.co/v1/production/uploads/652965773a416e1f2173443b/y9MB8YgHzbwCXAc4EI9T3.jpeg
Yuhao Dong
user
THUdyh
6861fb7a9e7509383d29ab50
Fan Zhang
6861fb7a9e7509383d29ab51
Ziqi Huang
6861fb7a9e7509383d29ab52
Yinan He
6861fb7a9e7509383d29ab53
Yangguang Li
6861fb7a9e7509383d29ab54
Weichao Chen
6861fb7a9e7509383d29ab55
Yu Qiao
6861fb7a9e7509383d29ab56
Wanli Ouyang
6861fb7a9e7509383d29ab57
Shengjie Zhao
6861fb7a9e7509383d29ab58
Ziwei Liu
ShotBench: Expert-Level Cinematic Understanding in Vision-Language
  Models
Cinematography, the fundamental visual language of film, is essential for
conveying narrative, emotion, and aesthetic quality. While recent
Vision-Language Models (VLMs) demonstrate strong general visual understanding,
their proficiency in comprehending the nuanced cinematic grammar embedded
within individual shots remains largely unexplored and lacks robust evaluation.
This critical gap limits both fine-grained visual comprehension and the
precision of AI-assisted video generation. To address this, we introduce
ShotBench, a comprehensive benchmark specifically designed for cinematic
language understanding. It features over 3.5k expert-annotated QA pairs from
images and video clips, meticulously curated from over 200 acclaimed
(predominantly Oscar-nominated) films and spanning eight key cinematography
dimensions. Our evaluation of 24 leading VLMs on ShotBench reveals their
substantial limitations: even the top-performing model achieves less than 60%
average accuracy, particularly struggling with fine-grained visual cues and
complex spatial reasoning. To catalyze advancement in this domain, we construct
ShotQA, a large-scale multimodal dataset comprising approximately 70k cinematic
QA pairs. Leveraging ShotQA, we develop ShotVL through supervised fine-tuning
and Group Relative Policy Optimization. ShotVL significantly outperforms all
existing open-source and proprietary models on ShotBench, establishing new
state-of-the-art performance. We open-source our models, data, and code to
foster rapid progress in this crucial area of AI-driven cinematic understanding
and generation.
ShotBench and ShotQA datasets, along with ShotVL model, enhance AI's understanding and generation capabilities by specifically targeting nuanced cinematic language comprehension.
Vision-Language Models VLMs ShotBench QA pairs cinematic grammar fine-grained visual comprehension AI-assisted video generation ShotQA multimodal dataset supervised fine-tuning Group Relative Policy Optimization ShotVL AI-driven cinematic understanding state-of-the-art performance