title: VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained
    Video Reasoning via Core Frame Selection
summary: The advancement of Large Vision Language Models (LVLMs) has significantly
  improved multimodal understanding, yet challenges remain in video reasoning
  tasks due to the scarcity of high-quality, large-scale datasets. Existing video
  question-answering (VideoQA) datasets often rely on costly manual annotations
  with insufficient granularity or automatic construction methods with redundant
  frame-by-frame analysis, limiting their scalability and effectiveness for
  complex reasoning. To address these challenges, we introduce VideoEspresso, a
  novel dataset that features VideoQA pairs preserving essential spatial details
  and temporal coherence, along with multimodal annotations of intermediate
  reasoning steps. Our construction pipeline employs a semantic-aware method to
  reduce redundancy, followed by generating QA pairs using GPT-4o. We further
  develop video Chain-of-Thought (CoT) annotations to enrich reasoning processes,
  guiding GPT-4o in extracting logical relationships from QA pairs and video
  content. To exploit the potential of high-quality VideoQA pairs, we propose a
  Hybrid LVLMs Collaboration framework, featuring a Frame Selector and a
  two-stage instruction fine-tuned reasoning LVLM. This framework adaptively
  selects core frames and performs CoT reasoning using multimodal evidence.
  Evaluated on our proposed benchmark with 14 tasks against 9 popular LVLMs, our
  method outperforms existing baselines on most tasks, demonstrating superior
  video reasoning capabilities. Our code and dataset will be released at:
  https://github.com/hshjerry/VideoEspresso
paper_authors._id: 6743eb8cde9997dd2682a74f
paper_authors.name: Songhao Han
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-26T21:21:24.087Z
paper_authors._id: 662885b9b87483ae5a9ee5c9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/jDXWlfjVECACkh5qvFLRn.png
paper_authors.fullname: Jerry Han
paper_authors.type: user
paper_authors.user: hshjerry0315
paper_authors._id: 6743eb8cde9997dd2682a750
paper_authors.name: Wei Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-26T21:21:21.888Z
paper_authors._id: 656db3f53dc1d277e5a64410
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/656db3f53dc1d277e5a64410/9kiY2K3MCRcBDk7MrkTBK.png
paper_authors.fullname: Wei Huang
paper_authors.type: user
paper_authors.user: AaronHuangWei
paper_authors._id: 6743eb8cde9997dd2682a751
paper_authors.name: Hairong Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:23:05.513Z
paper_authors._id: 650698169310ce8c402a95b6
paper_authors.avatarUrl: /avatars/82ea3a461b7a72f12057fa2334e2c80f.svg
paper_authors.fullname: Hairong Shi
paper_authors.type: user
paper_authors.user: Nanase1234
paper_authors._id: 6743eb8cde9997dd2682a752
paper_authors.name: Le Zhuo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:20:54.459Z
paper_authors._id: 6358a167f56b03ec9147074d
paper_authors.avatarUrl: /avatars/e54ea7bf0c240cf76d538296efb3976c.svg
paper_authors.fullname: Le Zhuo
paper_authors.type: user
paper_authors.user: JackyZhuo
paper_authors._id: 6743eb8cde9997dd2682a753
paper_authors.name: Xiu Su
paper_authors._id: 6743eb8cde9997dd2682a754
paper_authors.name: Shifeng Zhang
paper_authors._id: 6743eb8cde9997dd2682a755
paper_authors.name: Xu Zhou
paper_authors._id: 6743eb8cde9997dd2682a756
paper_authors.name: Xiaojuan Qi
paper_authors._id: 6743eb8cde9997dd2682a757
paper_authors.name: Yue Liao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:20:37.613Z
paper_authors._id: 670f827bb94a3734d270f707
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/D6qCPBMJAUgozfG7YTwky.png
paper_authors.fullname: Yue Liao
paper_authors.type: user
paper_authors.user: morninghaze
paper_authors._id: 6743eb8cde9997dd2682a758
paper_authors.name: Si Liu
paper_title: VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained
    Video Reasoning via Core Frame Selection
paper_summary: The advancement of Large Vision Language Models (LVLMs) has significantly
  improved multimodal understanding, yet challenges remain in video reasoning
  tasks due to the scarcity of high-quality, large-scale datasets. Existing video
  question-answering (VideoQA) datasets often rely on costly manual annotations
  with insufficient granularity or automatic construction methods with redundant
  frame-by-frame analysis, limiting their scalability and effectiveness for
  complex reasoning. To address these challenges, we introduce VideoEspresso, a
  novel dataset that features VideoQA pairs preserving essential spatial details
  and temporal coherence, along with multimodal annotations of intermediate
  reasoning steps. Our construction pipeline employs a semantic-aware method to
  reduce redundancy, followed by generating QA pairs using GPT-4o. We further
  develop video Chain-of-Thought (CoT) annotations to enrich reasoning processes,
  guiding GPT-4o in extracting logical relationships from QA pairs and video
  content. To exploit the potential of high-quality VideoQA pairs, we propose a
  Hybrid LVLMs Collaboration framework, featuring a Frame Selector and a
  two-stage instruction fine-tuned reasoning LVLM. This framework adaptively
  selects core frames and performs CoT reasoning using multimodal evidence.
  Evaluated on our proposed benchmark with 14 tasks against 9 popular LVLMs, our
  method outperforms existing baselines on most tasks, demonstrating superior
  video reasoning capabilities. Our code and dataset will be released at:
  https://github.com/hshjerry/VideoEspresso
paper_ai_keywords: Large Vision Language Models (LVLMs) VideoQA multimodal annotations semantic-aware method Chain-of-Thought (CoT) Frame Selector two-stage instruction fine-tuned reasoning LVLM video reasoning capabilities