title: VCR-Bench: A Comprehensive Evaluation Framework for Video
    Chain-of-Thought Reasoning
summary: The advancement of Chain-of-Thought (CoT) reasoning has significantly
  enhanced the capabilities of large language models (LLMs) and large
  vision-language models (LVLMs). However, a rigorous evaluation framework for
  video CoT reasoning remains absent. Current video benchmarks fail to adequately
  assess the reasoning process and expose whether failures stem from deficiencies
  in perception or reasoning capabilities. Therefore, we introduce VCR-Bench, a
  novel benchmark designed to comprehensively evaluate LVLMs' Video
  Chain-of-Thought Reasoning capabilities. VCR-Bench comprises 859 videos
  spanning a variety of video content and durations, along with 1,034
  high-quality question-answer pairs. Each pair is manually annotated with a
  stepwise CoT rationale, where every step is tagged to indicate its association
  with the perception or reasoning capabilities. Furthermore, we design seven
  distinct task dimensions and propose the CoT score to assess the entire CoT
  process based on the stepwise tagged CoT rationals. Extensive experiments on
  VCR-Bench highlight substantial limitations in current LVLMs. Even the
  top-performing model, o1, only achieves a 62.8% CoT score and an 56.7%
  accuracy, while most models score below 40%. Experiments show most models score
  lower on perception than reasoning steps, revealing LVLMs' key bottleneck in
  temporal-spatial information processing for complex video reasoning. A robust
  positive correlation between the CoT score and accuracy confirms the validity
  of our evaluation framework and underscores the critical role of CoT reasoning
  in solving complex video reasoning tasks. We hope VCR-Bench to serve as a
  standardized evaluation framework and expose the actual drawbacks in complex
  video reasoning task.
paper_authors._id: 67f886f1a0a44c8f05b7a124
paper_authors.name: Yukun Qi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-11T08:24:01.756Z
paper_authors._id: 66553907965ea394ee85f04c
paper_authors.avatarUrl: /avatars/e0adfe83e9ec51b118e359ddc5c37a1b.svg
paper_authors.fullname: qyk
paper_authors.type: user
paper_authors.user: yukunqi
paper_authors._id: 67f886f1a0a44c8f05b7a125
paper_authors.name: Yiming Zhao
paper_authors._id: 67f886f1a0a44c8f05b7a126
paper_authors.name: Yu Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-14T09:47:06.461Z
paper_authors._id: 665d652e0f35c005de892108
paper_authors.avatarUrl: /avatars/240bebdc7fdc6d50719c65de0e3cf1cd.svg
paper_authors.fullname: Yu Zeng
paper_authors.type: user
paper_authors.user: YuZeng260
paper_authors._id: 67f886f1a0a44c8f05b7a127
paper_authors.name: Xikun Bao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-11T08:24:37.813Z
paper_authors._id: 672046bd4b2e5a664aca3084
paper_authors.avatarUrl: /avatars/6635c617216eca67b38b64deec634fee.svg
paper_authors.fullname: xikun bao
paper_authors.type: user
paper_authors.user: ChthollyTree
paper_authors._id: 67f886f1a0a44c8f05b7a128
paper_authors.name: Wenxuan Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-11T08:24:49.236Z
paper_authors._id: 67dc162ec8c00778e8689f42
paper_authors.avatarUrl: /avatars/7abcd41d4d466ab751f22048050d7f53.svg
paper_authors.fullname: Wenxuan Huang
paper_authors.type: user
paper_authors.user: Osilly
paper_authors._id: 67f886f1a0a44c8f05b7a129
paper_authors.name: Lin Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-11T07:25:22.829Z
paper_authors._id: 64b02ec0e5000ae8a572ced5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b02ec0e5000ae8a572ced5/6ifLntBU2ICQK7SW8WxKU.png
paper_authors.fullname: Lin Chen
paper_authors.type: user
paper_authors.user: Lin-Chen
paper_authors._id: 67f886f1a0a44c8f05b7a12a
paper_authors.name: Zehui Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-11T08:24:56.718Z
paper_authors._id: 64892d31cbda0d1cdb956897
paper_authors.avatarUrl: /avatars/3cdafe03a8295124636347d15a099aaf.svg
paper_authors.fullname: Zehui Chen
paper_authors.type: user
paper_authors.user: lovesnowbest
paper_authors._id: 67f886f1a0a44c8f05b7a12b
paper_authors.name: Jie Zhao
paper_authors._id: 67f886f1a0a44c8f05b7a12c
paper_authors.name: Zhongang Qi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-11T08:25:12.007Z
paper_authors._id: 660cc64f1b14f691990c0ea0
paper_authors.avatarUrl: /avatars/f172d3120b22d745a41cc3f2eb499ce6.svg
paper_authors.fullname: Zhongang Qi
paper_authors.type: user
paper_authors.user: phoenixqza
paper_authors._id: 67f886f1a0a44c8f05b7a12d
paper_authors.name: Feng Zhao
paper_title: VCR-Bench: A Comprehensive Evaluation Framework for Video
    Chain-of-Thought Reasoning
paper_summary: The advancement of Chain-of-Thought (CoT) reasoning has significantly
  enhanced the capabilities of large language models (LLMs) and large
  vision-language models (LVLMs). However, a rigorous evaluation framework for
  video CoT reasoning remains absent. Current video benchmarks fail to adequately
  assess the reasoning process and expose whether failures stem from deficiencies
  in perception or reasoning capabilities. Therefore, we introduce VCR-Bench, a
  novel benchmark designed to comprehensively evaluate LVLMs' Video
  Chain-of-Thought Reasoning capabilities. VCR-Bench comprises 859 videos
  spanning a variety of video content and durations, along with 1,034
  high-quality question-answer pairs. Each pair is manually annotated with a
  stepwise CoT rationale, where every step is tagged to indicate its association
  with the perception or reasoning capabilities. Furthermore, we design seven
  distinct task dimensions and propose the CoT score to assess the entire CoT
  process based on the stepwise tagged CoT rationals. Extensive experiments on
  VCR-Bench highlight substantial limitations in current LVLMs. Even the
  top-performing model, o1, only achieves a 62.8% CoT score and an 56.7%
  accuracy, while most models score below 40%. Experiments show most models score
  lower on perception than reasoning steps, revealing LVLMs' key bottleneck in
  temporal-spatial information processing for complex video reasoning. A robust
  positive correlation between the CoT score and accuracy confirms the validity
  of our evaluation framework and underscores the critical role of CoT reasoning
  in solving complex video reasoning tasks. We hope VCR-Bench to serve as a
  standardized evaluation framework and expose the actual drawbacks in complex
  video reasoning task.
paper_ai_summary: VCR-Bench is a new benchmark designed to evaluate video chain-of-thought reasoning in large vision-language models, assessing both perception and reasoning capabilities.
paper_ai_keywords: Chain-of-Thought (CoT) large language models (LLMs) large vision-language models (LVLMs) video chain-of-thought reasoning VCR-Bench stepwise CoT rationale task dimensions CoT score complex video reasoning perception reasoning temporal-spatial information processing