SpatialPIN: Enhancing Spatial Reasoning Capabilities of Vision-Language Models through Prompting and Interacting 3D Priors
<|reference_start|>SpatialPIN: Enhancing Spatial Reasoning Capabilities of Vision-Language Models through Prompting and Interacting 3D Priors: Current state-of-the-art spatial reasoning-enhanced VLMs are trained to excel at spatial visual question answering (VQA). However, we believe that higher-level 3D-aware tasks, such as articulating dynamic scene changes and motion planning, require a fundamental and explicit 3D understanding beyond current spatial VQA datasets. In this work, we present SpatialPIN, a framework designed to enhance the spatial reasoning capabilities of VLMs through prompting and interacting with priors from multiple 3D foundation models in a zero-shot, training-free manner. Extensive experiments demonstrate that our spatial reasoning-imbued VLM performs well on various forms of spatial VQA and can extend to help in various downstream robotics tasks such as pick and stack and trajectory planning.<|reference_end|>
@article{ma2024spatialpin:,
    title={SpatialPIN: Enhancing Spatial Reasoning Capabilities of Vision-Language
  Models through Prompting and Interacting 3D Priors},
    author={Chenyang Ma, Kai Lu, Ta-Ying Cheng, Niki Trigoni, Andrew Markham},
    journal={arXiv preprint arXiv:2403.13438},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.13438},
    primaryClass={cs.CV}
}