title: Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?
summary: Recent advances in CoT reasoning and RL post-training have been reported to
  enhance video reasoning capabilities of MLLMs. This progress naturally raises a
  question: can these models perform complex video reasoning in a manner
  comparable to human experts? However, existing video benchmarks primarily
  evaluate visual perception and grounding abilities, with questions that can be
  answered based on explicit prompts or isolated visual cues. Such benchmarks do
  not fully capture the intricacies of real-world reasoning, where humans must
  actively search for, integrate, and analyze multiple clues before reaching a
  conclusion. To address this issue, we present Video-Holmes, a benchmark
  inspired by the reasoning process of Sherlock Holmes, designed to evaluate the
  complex video reasoning capabilities of MLLMs. Video-Holmes consists of 1,837
  questions derived from 270 manually annotated suspense short films, which spans
  seven carefully designed tasks. Each task is constructed by first identifying
  key events and causal relationships within films, and then designing questions
  that require models to actively locate and connect multiple relevant visual
  clues scattered across different video segments. Our comprehensive evaluation
  of state-of-the-art MLLMs reveals that, while these models generally excel at
  visual perception, they encounter substantial difficulties with integrating
  information and often miss critical clues. For example, the best-performing
  model, Gemini-2.5-Pro, achieves an accuracy of only 45%, with most models
  scoring below 40%. We aim that Video-Holmes can serve as a "Holmes-test" for
  multimodal reasoning, motivating models to reason more like humans and
  emphasizing the ongoing challenges in this field. The benchmark is released in
  https://github.com/TencentARC/Video-Holmes.
paper_authors._id: 68366975d4ea32a1b4eedd82
paper_authors.name: Junhao Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T08:58:22.568Z
paper_authors._id: 6506b77a773ceaa8d52ecea1
paper_authors.avatarUrl: /avatars/0e769a0795063e1491c44760a4a83097.svg
paper_authors.fullname: CJH
paper_authors.type: user
paper_authors.user: Howe666
paper_authors._id: 68366975d4ea32a1b4eedd83
paper_authors.name: Yuying Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-28T12:23:21.224Z
paper_authors._id: 6455cc8f654d8bccae50e4d4
paper_authors.avatarUrl: /avatars/506a9992e5bf52e06d37cc22e4b307c0.svg
paper_authors.fullname: Yuying Ge
paper_authors.type: user
paper_authors.user: tttoaster
paper_authors._id: 68366975d4ea32a1b4eedd84
paper_authors.name: Teng Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-05T15:08:35.735Z
paper_authors._id: 63a9414e32ed73936ec0a0c8
paper_authors.avatarUrl: /avatars/f181e1bb480502c2680be5296a036bdd.svg
paper_authors.fullname: wybertwang
paper_authors.type: user
paper_authors.user: wybertwang
paper_authors._id: 68366975d4ea32a1b4eedd85
paper_authors.name: Yixiao Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-28T12:23:47.334Z
paper_authors._id: 640e9762b03f4cd29f58d982
paper_authors.avatarUrl: /avatars/81da37d628163fe3e094b247c7c3a3b5.svg
paper_authors.fullname: Yixiao Ge
paper_authors.type: user
paper_authors.user: yxgeee
paper_authors._id: 68366975d4ea32a1b4eedd86
paper_authors.name: Jing Liao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-28T12:24:02.238Z
paper_authors._id: 65e77726767bfc7d109c45bf
paper_authors.avatarUrl: /avatars/24e68c86e06055ea1209598ba49ce8b9.svg
paper_authors.fullname: Jing Liao
paper_authors.type: user
paper_authors.user: CeciliaJL
paper_authors._id: 68366975d4ea32a1b4eedd87
paper_authors.name: Ying Shan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-28T12:24:43.904Z
paper_authors._id: 63ca3ddc04c979828310bfcb
paper_authors.avatarUrl: /avatars/615e0d8622950b4408b40d550f02a894.svg
paper_authors.fullname: Ying Shan
paper_authors.type: user
paper_authors.user: yshan2u
paper_title: Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?
paper_summary: Recent advances in CoT reasoning and RL post-training have been reported to
  enhance video reasoning capabilities of MLLMs. This progress naturally raises a
  question: can these models perform complex video reasoning in a manner
  comparable to human experts? However, existing video benchmarks primarily
  evaluate visual perception and grounding abilities, with questions that can be
  answered based on explicit prompts or isolated visual cues. Such benchmarks do
  not fully capture the intricacies of real-world reasoning, where humans must
  actively search for, integrate, and analyze multiple clues before reaching a
  conclusion. To address this issue, we present Video-Holmes, a benchmark
  inspired by the reasoning process of Sherlock Holmes, designed to evaluate the
  complex video reasoning capabilities of MLLMs. Video-Holmes consists of 1,837
  questions derived from 270 manually annotated suspense short films, which spans
  seven carefully designed tasks. Each task is constructed by first identifying
  key events and causal relationships within films, and then designing questions
  that require models to actively locate and connect multiple relevant visual
  clues scattered across different video segments. Our comprehensive evaluation
  of state-of-the-art MLLMs reveals that, while these models generally excel at
  visual perception, they encounter substantial difficulties with integrating
  information and often miss critical clues. For example, the best-performing
  model, Gemini-2.5-Pro, achieves an accuracy of only 45%, with most models
  scoring below 40%. We aim that Video-Holmes can serve as a "Holmes-test" for
  multimodal reasoning, motivating models to reason more like humans and
  emphasizing the ongoing challenges in this field. The benchmark is released in
  https://github.com/TencentARC/Video-Holmes.
paper_ai_summary: Video-Holmes benchmark evaluates complex video reasoning capabilities of MLLMs using suspense short films and reveals significant challenges in information integration compared to human experts.
paper_ai_keywords: CoT reasoning RL post-training MLLMs Visual perception Grounding abilities Video-Holmes Suspense short films Multimodal reasoning Holmes-test