CoLLaVO: Crayon Large Language and Vision mOdel
<|reference_start|>CoLLaVO: Crayon Large Language and Vision mOdel: The remarkable success of Large Language Models (LLMs) and instruction tuning drives the evolution of Vision Language Models (VLMs) towards a versatile general-purpose model. Yet, it remains unexplored whether current VLMs genuinely possess quality object-level image understanding capabilities determined from 'what objects are in the image?' or 'which object corresponds to a specified bounding box?'. Our findings reveal that the image understanding capabilities of current VLMs are strongly correlated with their zero-shot performance on vision language (VL) tasks. This suggests that prioritizing basic image understanding is crucial for VLMs to excel at VL tasks. To enhance object-level image understanding, we propose Crayon Large Language and Vision mOdel (CoLLaVO), which incorporates instruction tuning with Crayon Prompt as a new visual prompt tuning scheme based on panoptic color maps. Furthermore, we present a learning strategy of Dual QLoRA to preserve object-level image understanding without forgetting it during visual instruction tuning, thereby achieving a significant leap in numerous VL benchmarks in a zero-shot setting.<|reference_end|>
@article{lee2024collavo:,
    title={CoLLaVO: Crayon Large Language and Vision mOdel},
    author={Byung-Kwan Lee, Beomchan Park, Chae Won Kim, Yong Man Ro},
    journal={arXiv preprint arXiv:2402.11248},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.11248},
    primaryClass={cs.CV}
}