Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMs
<|reference_start|>Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMs: Integration of Large Language Models (LLMs) into visual domain tasks, resulting in visual-LLMs (V-LLMs), has enabled exceptional performance in vision-language tasks, particularly for visual question answering (VQA). However, existing V-LLMs (e.g. BLIP-2, LLaVA) demonstrate weak spatial reasoning and localization awareness. Despite generating highly descriptive and elaborate textual answers, these models fail at simple tasks like distinguishing a left vs right location. In this work, we explore how image-space coordinate based instruction fine-tuning objectives could inject spatial awareness into V-LLMs. We discover optimal coordinate representations, data-efficient instruction fine-tuning objectives, and pseudo-data generation strategies that lead to improved spatial awareness in V-LLMs. Additionally, our resulting model improves VQA across image and video domains, reduces undesired hallucination, and generates better contextual object descriptions. Experiments across 5 vision-language tasks involving 14 different datasets establish the clear performance improvements achieved by our proposed framework.<|reference_end|>
@article{ranasinghe2024learning,
    title={Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMs},
    author={Kanchana Ranasinghe and Satya Narayan Shukla and Omid Poursaeed and
  Michael S. Ryoo and Tsung-Yu Lin},
    journal={arXiv preprint arXiv:2404.07449},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.07449},
    primaryClass={cs.CV}
}