VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained
  Video Reasoning via Core Frame Selection
The advancement of Large Vision Language Models (LVLMs) has significantly
improved multimodal understanding, yet challenges remain in video reasoning
tasks due to the scarcity of high-quality, large-scale datasets. Existing video
question-answering (VideoQA) datasets often rely on costly manual annotations
with insufficient granularity or automatic construction methods with redundant
frame-by-frame analysis, limiting their scalability and effectiveness for
complex reasoning. To address these challenges, we introduce VideoEspresso, a
novel dataset that features VideoQA pairs preserving essential spatial details
and temporal coherence, along with multimodal annotations of intermediate
reasoning steps. Our construction pipeline employs a semantic-aware method to
reduce redundancy, followed by generating QA pairs using GPT-4o. We further
develop video Chain-of-Thought (CoT) annotations to enrich reasoning processes,
guiding GPT-4o in extracting logical relationships from QA pairs and video
content. To exploit the potential of high-quality VideoQA pairs, we propose a
Hybrid LVLMs Collaboration framework, featuring a Frame Selector and a
two-stage instruction fine-tuned reasoning LVLM. This framework adaptively
selects core frames and performs CoT reasoning using multimodal evidence.
Evaluated on our proposed benchmark with 14 tasks against 9 popular LVLMs, our
method outperforms existing baselines on most tasks, demonstrating superior
video reasoning capabilities. Our code and dataset will be released at:
https://github.com/hshjerry/VideoEspresso
6743eb8cde9997dd2682a74f
Songhao Han
claimed_verified
2024-11-26T21:21:24.087Z
662885b9b87483ae5a9ee5c9
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/jDXWlfjVECACkh5qvFLRn.png
Songhao Han
user
hshjerry0315
6743eb8cde9997dd2682a750
Wei Huang
claimed_verified
2024-11-26T21:21:21.888Z
656db3f53dc1d277e5a64410
https://cdn-avatars.huggingface.co/v1/production/uploads/656db3f53dc1d277e5a64410/9kiY2K3MCRcBDk7MrkTBK.png
Wei Huang
user
AaronHuangWei
6743eb8cde9997dd2682a751
Hairong Shi
admin_assigned
2024-11-25T21:23:05.513Z
650698169310ce8c402a95b6
/avatars/82ea3a461b7a72f12057fa2334e2c80f.svg
Hairong Shi
user
Nanase1234
6743eb8cde9997dd2682a752
Le Zhuo
admin_assigned
2024-11-25T21:20:54.459Z
6358a167f56b03ec9147074d
/avatars/e54ea7bf0c240cf76d538296efb3976c.svg
Le Zhuo
user
JackyZhuo
6743eb8cde9997dd2682a753
Xiu Su
6743eb8cde9997dd2682a754
Shifeng Zhang
6743eb8cde9997dd2682a755
Xu Zhou
6743eb8cde9997dd2682a756
Xiaojuan Qi
claimed_verified
2025-07-15T19:14:28.227Z
6875266f9cd3191dfddc7071
/avatars/64c581910833b111e9a7bae5b8740229.svg
xiaojuan qi
user
xjqi
6743eb8cde9997dd2682a757
Yue Liao
admin_assigned
2024-11-25T21:20:37.613Z
670f827bb94a3734d270f707
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/D6qCPBMJAUgozfG7YTwky.png
Yue Liao
user
morninghaze
6743eb8cde9997dd2682a758
Si Liu
VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained
  Video Reasoning via Core Frame Selection
The advancement of Large Vision Language Models (LVLMs) has significantly
improved multimodal understanding, yet challenges remain in video reasoning
tasks due to the scarcity of high-quality, large-scale datasets. Existing video
question-answering (VideoQA) datasets often rely on costly manual annotations
with insufficient granularity or automatic construction methods with redundant
frame-by-frame analysis, limiting their scalability and effectiveness for
complex reasoning. To address these challenges, we introduce VideoEspresso, a
novel dataset that features VideoQA pairs preserving essential spatial details
and temporal coherence, along with multimodal annotations of intermediate
reasoning steps. Our construction pipeline employs a semantic-aware method to
reduce redundancy, followed by generating QA pairs using GPT-4o. We further
develop video Chain-of-Thought (CoT) annotations to enrich reasoning processes,
guiding GPT-4o in extracting logical relationships from QA pairs and video
content. To exploit the potential of high-quality VideoQA pairs, we propose a
Hybrid LVLMs Collaboration framework, featuring a Frame Selector and a
two-stage instruction fine-tuned reasoning LVLM. This framework adaptively
selects core frames and performs CoT reasoning using multimodal evidence.
Evaluated on our proposed benchmark with 14 tasks against 9 popular LVLMs, our
method outperforms existing baselines on most tasks, demonstrating superior
video reasoning capabilities. Our code and dataset will be released at:
https://github.com/hshjerry/VideoEspresso
VideoEspresso, a novel dataset with high-quality VideoQA pairs and multimodal annotations, enhances LVLMs for video reasoning through a semantic-aware pipeline and hybrid collaboration framework.
Large Vision Language Models (LVLMs) video reasoning tasks video question-answering (VideoQA) VideoEspresso semantic-aware method GPT-4o video Chain-of-Thought (CoT) annotations Frame Selector two-stage instruction fine-tuned reasoning LVLM hybrid collaboration framework multimodal evidence