title: ST-VLM: Kinematic Instruction Tuning for Spatio-Temporal Reasoning in
    Vision-Language Models
summary: Spatio-temporal reasoning is essential in understanding real-world
  environments in various fields, eg, autonomous driving and sports analytics.
  Recent advances have improved the spatial reasoning ability of Vision-Language
  Models (VLMs) by introducing large-scale data, but these models still struggle
  to analyze kinematic elements like traveled distance and speed of moving
  objects. To bridge this gap, we construct a spatio-temporal reasoning dataset
  and benchmark involving kinematic instruction tuning, referred to as STKit and
  STKit-Bench. They consist of real-world videos with 3D annotations, detailing
  object motion dynamics: traveled distance, speed, movement direction,
  inter-object distance comparisons, and relative movement direction. To further
  scale such data construction to videos without 3D labels, we propose an
  automatic pipeline to generate pseudo-labels using 4D reconstruction in
  real-world scale. With our kinematic instruction tuning data for
  spatio-temporal reasoning, we present ST-VLM, a VLM enhanced for
  spatio-temporal reasoning, which exhibits outstanding performance on
  STKit-Bench. Furthermore, we show that ST-VLM generalizes robustly across
  diverse domains and tasks, outperforming baselines on other spatio-temporal
  benchmarks (eg, ActivityNet, TVQA+). Finally, by integrating learned
  spatio-temporal reasoning with existing abilities, ST-VLM enables complex
  multi-step reasoning. Project page: https://ikodoh.github.io/ST-VLM.
paper_authors._id: 67e46aeafe1f5acc68e9ee04
paper_authors.name: Dohwan Ko
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-27T09:03:45.162Z
paper_authors._id: 62bab5a69f045fd1fb9f760e
paper_authors.avatarUrl: /avatars/3f12a908faec1135e0221444f0769496.svg
paper_authors.fullname: Dohwan Ko
paper_authors.type: user
paper_authors.user: ikodoh
paper_authors._id: 67e46aeafe1f5acc68e9ee05
paper_authors.name: Sihyeon Kim
paper_authors._id: 67e46aeafe1f5acc68e9ee06
paper_authors.name: Yumin Suh
paper_authors._id: 67e46aeafe1f5acc68e9ee07
paper_authors.name: Vijay Kumar B. G
paper_authors._id: 67e46aeafe1f5acc68e9ee08
paper_authors.name: Minseo Yoon
paper_authors._id: 67e46aeafe1f5acc68e9ee09
paper_authors.name: Manmohan Chandraker
paper_authors._id: 67e46aeafe1f5acc68e9ee0a
paper_authors.name: Hyunwoo J. Kim
paper_title: ST-VLM: Kinematic Instruction Tuning for Spatio-Temporal Reasoning in
    Vision-Language Models
paper_summary: Spatio-temporal reasoning is essential in understanding real-world
  environments in various fields, eg, autonomous driving and sports analytics.
  Recent advances have improved the spatial reasoning ability of Vision-Language
  Models (VLMs) by introducing large-scale data, but these models still struggle
  to analyze kinematic elements like traveled distance and speed of moving
  objects. To bridge this gap, we construct a spatio-temporal reasoning dataset
  and benchmark involving kinematic instruction tuning, referred to as STKit and
  STKit-Bench. They consist of real-world videos with 3D annotations, detailing
  object motion dynamics: traveled distance, speed, movement direction,
  inter-object distance comparisons, and relative movement direction. To further
  scale such data construction to videos without 3D labels, we propose an
  automatic pipeline to generate pseudo-labels using 4D reconstruction in
  real-world scale. With our kinematic instruction tuning data for
  spatio-temporal reasoning, we present ST-VLM, a VLM enhanced for
  spatio-temporal reasoning, which exhibits outstanding performance on
  STKit-Bench. Furthermore, we show that ST-VLM generalizes robustly across
  diverse domains and tasks, outperforming baselines on other spatio-temporal
  benchmarks (eg, ActivityNet, TVQA+). Finally, by integrating learned
  spatio-temporal reasoning with existing abilities, ST-VLM enables complex
  multi-step reasoning. Project page: https://ikodoh.github.io/ST-VLM.
paper_ai_summary: ST-VLM enhances vision-language models with spatio-temporal reasoning using a new dataset and automatic pipeline, achieving superior performance on benchmarks involving kinematic elements.
paper_ai_keywords: spatio-temporal reasoning Vision-Language Models (VLMs) STKit STKit-Bench 3D annotations 4D reconstruction kinematic instruction tuning multi-step reasoning ActivityNet TVQA+