ST-VLM: Kinematic Instruction Tuning for Spatio-Temporal Reasoning in
  Vision-Language Models
Spatio-temporal reasoning is essential in understanding real-world
environments in various fields, eg, autonomous driving and sports analytics.
Recent advances have improved the spatial reasoning ability of Vision-Language
Models (VLMs) by introducing large-scale data, but these models still struggle
to analyze kinematic elements like traveled distance and speed of moving
objects. To bridge this gap, we construct a spatio-temporal reasoning dataset
and benchmark involving kinematic instruction tuning, referred to as STKit and
STKit-Bench. They consist of real-world videos with 3D annotations, detailing
object motion dynamics: traveled distance, speed, movement direction,
inter-object distance comparisons, and relative movement direction. To further
scale such data construction to videos without 3D labels, we propose an
automatic pipeline to generate pseudo-labels using 4D reconstruction in
real-world scale. With our kinematic instruction tuning data for
spatio-temporal reasoning, we present ST-VLM, a VLM enhanced for
spatio-temporal reasoning, which exhibits outstanding performance on
STKit-Bench. Furthermore, we show that ST-VLM generalizes robustly across
diverse domains and tasks, outperforming baselines on other spatio-temporal
benchmarks (eg, ActivityNet, TVQA+). Finally, by integrating learned
spatio-temporal reasoning with existing abilities, ST-VLM enables complex
multi-step reasoning. Project page: https://ikodoh.github.io/ST-VLM.
67e46aeafe1f5acc68e9ee04
Dohwan Ko
claimed_verified
2025-03-27T09:03:45.162Z
62bab5a69f045fd1fb9f760e
/avatars/3f12a908faec1135e0221444f0769496.svg
Dohwan Ko
user
ikodoh
67e46aeafe1f5acc68e9ee05
Sihyeon Kim
67e46aeafe1f5acc68e9ee06
Yumin Suh
67e46aeafe1f5acc68e9ee07
Vijay Kumar B. G
67e46aeafe1f5acc68e9ee08
Minseo Yoon
67e46aeafe1f5acc68e9ee09
Manmohan Chandraker
67e46aeafe1f5acc68e9ee0a
Hyunwoo J. Kim
claimed_verified
2025-06-16T15:51:22.204Z
63183a30a760ff8cff318be5
/avatars/1b314afd8d3319423915ffc2ae56d567.svg
Hyunwoo Kim
user
hkimcvml
ST-VLM: Kinematic Instruction Tuning for Spatio-Temporal Reasoning in
  Vision-Language Models
Spatio-temporal reasoning is essential in understanding real-world
environments in various fields, eg, autonomous driving and sports analytics.
Recent advances have improved the spatial reasoning ability of Vision-Language
Models (VLMs) by introducing large-scale data, but these models still struggle
to analyze kinematic elements like traveled distance and speed of moving
objects. To bridge this gap, we construct a spatio-temporal reasoning dataset
and benchmark involving kinematic instruction tuning, referred to as STKit and
STKit-Bench. They consist of real-world videos with 3D annotations, detailing
object motion dynamics: traveled distance, speed, movement direction,
inter-object distance comparisons, and relative movement direction. To further
scale such data construction to videos without 3D labels, we propose an
automatic pipeline to generate pseudo-labels using 4D reconstruction in
real-world scale. With our kinematic instruction tuning data for
spatio-temporal reasoning, we present ST-VLM, a VLM enhanced for
spatio-temporal reasoning, which exhibits outstanding performance on
STKit-Bench. Furthermore, we show that ST-VLM generalizes robustly across
diverse domains and tasks, outperforming baselines on other spatio-temporal
benchmarks (eg, ActivityNet, TVQA+). Finally, by integrating learned
spatio-temporal reasoning with existing abilities, ST-VLM enables complex
multi-step reasoning. Project page: https://ikodoh.github.io/ST-VLM.
ST-VLM enhances vision-language models with spatio-temporal reasoning using a new dataset and automatic pipeline, achieving superior performance on benchmarks involving kinematic elements.
spatio-temporal reasoning Vision-Language Models (VLMs) STKit STKit-Bench 3D annotations 4D reconstruction kinematic instruction tuning multi-step reasoning ActivityNet TVQA+