title: V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoning
summary: Human processes video reasoning in a sequential spatio-temporal reasoning
  logic, we first identify the relevant frames ("when") and then analyse the
  spatial relationships ("where") between key objects, and finally leverage these
  relationships to draw inferences ("what"). However, can Video Large Language
  Models (Video-LLMs) also "reason through a sequential spatio-temporal logic" in
  videos? Existing Video-LLM benchmarks primarily focus on assessing object
  presence, neglecting relational reasoning. Consequently, it is difficult to
  measure whether a model truly comprehends object interactions (actions/events)
  in videos or merely relies on pre-trained "memory" of co-occurrences as biases
  in generating answers. In this work, we introduce a Video Spatio-Temporal
  Reasoning (V-STaR) benchmark to address these shortcomings. The key idea is to
  decompose video understanding into a Reverse Spatio-Temporal Reasoning (RSTR)
  task that simultaneously evaluates what objects are present, when events occur,
  and where they are located while capturing the underlying Chain-of-thought
  (CoT) logic. To support this evaluation, we construct a dataset to elicit the
  spatial-temporal reasoning process of Video-LLMs. It contains coarse-to-fine
  CoT questions generated by a semi-automated GPT-4-powered pipeline, embedding
  explicit reasoning chains to mimic human cognition. Experiments from 14
  Video-LLMs on our V-STaR reveal significant gaps between current Video-LLMs and
  the needs for robust and consistent spatio-temporal reasoning.
paper_authors._id: 67d8ca56e94f1237cb3ba3ca
paper_authors.name: Zixu Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-18T09:31:58.589Z
paper_authors._id: 667ee096b0fad0fdee319ed4
paper_authors.avatarUrl: /avatars/d9df687e8522d47f7fcefe40fd9b575b.svg
paper_authors.fullname: Zixu Cheng
paper_authors.type: user
paper_authors.user: Cade921
paper_authors._id: 67d8ca56e94f1237cb3ba3cb
paper_authors.name: Jian Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-18T08:08:20.643Z
paper_authors._id: 65e1b6e9501590df0173cbd3
paper_authors.avatarUrl: /avatars/a73e2139700e23eff455734c99cef5ba.svg
paper_authors.fullname: Jian Hu
paper_authors.type: user
paper_authors.user: lwpyh
paper_authors._id: 67d8ca56e94f1237cb3ba3cc
paper_authors.name: Ziquan Liu
paper_authors._id: 67d8ca56e94f1237cb3ba3cd
paper_authors.name: Chenyang Si
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-18T09:32:21.924Z
paper_authors._id: 635f8ed47c05eb9f59963d3a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/635f8ed47c05eb9f59963d3a/uQf4p9N9pSaFy87Wg9v4k.jpeg
paper_authors.fullname: ChenyangSi
paper_authors.type: user
paper_authors.user: ChenyangSi
paper_authors._id: 67d8ca56e94f1237cb3ba3ce
paper_authors.name: Wei Li
paper_authors._id: 67d8ca56e94f1237cb3ba3cf
paper_authors.name: Shaogang Gong
paper_title: V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoning
paper_summary: Human processes video reasoning in a sequential spatio-temporal reasoning
  logic, we first identify the relevant frames ("when") and then analyse the
  spatial relationships ("where") between key objects, and finally leverage these
  relationships to draw inferences ("what"). However, can Video Large Language
  Models (Video-LLMs) also "reason through a sequential spatio-temporal logic" in
  videos? Existing Video-LLM benchmarks primarily focus on assessing object
  presence, neglecting relational reasoning. Consequently, it is difficult to
  measure whether a model truly comprehends object interactions (actions/events)
  in videos or merely relies on pre-trained "memory" of co-occurrences as biases
  in generating answers. In this work, we introduce a Video Spatio-Temporal
  Reasoning (V-STaR) benchmark to address these shortcomings. The key idea is to
  decompose video understanding into a Reverse Spatio-Temporal Reasoning (RSTR)
  task that simultaneously evaluates what objects are present, when events occur,
  and where they are located while capturing the underlying Chain-of-thought
  (CoT) logic. To support this evaluation, we construct a dataset to elicit the
  spatial-temporal reasoning process of Video-LLMs. It contains coarse-to-fine
  CoT questions generated by a semi-automated GPT-4-powered pipeline, embedding
  explicit reasoning chains to mimic human cognition. Experiments from 14
  Video-LLMs on our V-STaR reveal significant gaps between current Video-LLMs and
  the needs for robust and consistent spatio-temporal reasoning.
paper_ai_summary: A new Video Spatio-Temporal Reasoning (V-STaR) benchmark assesses Video Large Language Models' ability to understand and reason through spatio-temporal relationships in videos.
paper_ai_keywords: Video Large Language Models (Video-LLMs) Video Spatio-Temporal Reasoning (V-STaR) Reverse Spatio-Temporal Reasoning (RSTR) Chain-of-thought (CoT)