I Know About "Up"! Enhancing Spatial Reasoning in Visual Language Models Through 3D Reconstruction
<|reference_start|>I Know About "Up"! Enhancing Spatial Reasoning in Visual Language Models Through 3D Reconstruction: Visual Language Models (VLMs) are essential for various tasks, particularly visual reasoning tasks, due to their robust multi-modal information integration, visual reasoning capabilities, and contextual awareness. However, existing \VLMs{}' visual spatial reasoning capabilities are often inadequate, struggling even with basic tasks such as distinguishing left from right. To address this, we propose the \ours{} model, designed to enhance the visual spatial reasoning abilities of VLMS. ZeroVLM employs Zero-1-to-3, a 3D reconstruction model for obtaining different views of the input images and incorporates a prompting mechanism to further improve visual spatial reasoning. Experimental results on four visual spatial reasoning datasets show that our \ours{} achieves up to 19.48% accuracy improvement, which indicates the effectiveness of the 3D reconstruction and prompting mechanisms of our ZeroVLM.<|reference_end|>
@article{meng2024i,
    title={I Know About "Up"! Enhancing Spatial Reasoning in Visual Language Models
  Through 3D Reconstruction},
    author={Zaiqiao Meng and Hao Zhou and Yifang Chen},
    journal={arXiv preprint arXiv:2407.14133},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.14133},
    primaryClass={cs.CL}
}