title: CoLLaVO: Crayon Large Language and Vision mOdel
summary: The remarkable success of Large Language Models (LLMs) and instruction tuning
  drives the evolution of Vision Language Models (VLMs) towards a versatile
  general-purpose model. Yet, it remains unexplored whether current VLMs
  genuinely possess quality object-level image understanding capabilities
  determined from 'what objects are in the image?' or 'which object corresponds
  to a specified bounding box?'. Our findings reveal that the image understanding
  capabilities of current VLMs are strongly correlated with their zero-shot
  performance on Vision Language (VL) tasks. This suggests that prioritizing
  basic image understanding is crucial for VLMs to excel at VL tasks. To enhance
  object-level image understanding, we propose Crayon Large Language and Vision
  mOdel (CoLLaVO), which incorporates instruction tuning with crayon prompt as a
  new visual prompt tuning scheme based on panoptic color maps. Furthermore, we
  present a learning strategy of Dual QLoRA to preserve object-level image
  understanding without forgetting it during visual instruction tuning, thereby
  achieving a significant leap in zero-shot numerous VL benchmarks.
paper_authors._id: 65d43a60adc348158d095a23
paper_authors.name: Byung-Kwan Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T10:13:56.582Z
paper_authors._id: 657152eb12f162153b50ec9d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/657152eb12f162153b50ec9d/qnldHP35PclV0pDz_05q8.jpeg
paper_authors.fullname: Byung-Kwan Lee
paper_authors.type: user
paper_authors.user: BK-Lee
paper_authors._id: 65d43a60adc348158d095a24
paper_authors.name: Beomchan Park
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-20T10:10:04.916Z
paper_authors._id: 65390e7ce279d2fc80a3f800
paper_authors.avatarUrl: /avatars/a930c1f20fa1eef372ca92139cb32887.svg
paper_authors.fullname: Beomchan Park
paper_authors.type: user
paper_authors.user: bpbpbp0810
paper_authors._id: 65d43a60adc348158d095a25
paper_authors.name: Chae Won Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-20T11:43:57.662Z
paper_authors._id: 65aa2e84e2a2c86356263340
paper_authors.avatarUrl: /avatars/cba9ecceff0f67b2b5dde02e7c0a9991.svg
paper_authors.fullname: Chae Won Kim
paper_authors.type: user
paper_authors.user: chae-won-kim
paper_authors._id: 65d43a60adc348158d095a26
paper_authors.name: Yong Man Ro
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-03T09:05:18.949Z
paper_authors._id: 660603529e3555d648e3baf1
paper_authors.avatarUrl: /avatars/0f54479afcfc19df00b25d5aedb4cf67.svg
paper_authors.fullname: Yong Man Ro
paper_authors.type: user
paper_authors.user: dwightro
paper_title: CoLLaVO: Crayon Large Language and Vision mOdel
paper_summary: The remarkable success of Large Language Models (LLMs) and instruction tuning
  drives the evolution of Vision Language Models (VLMs) towards a versatile
  general-purpose model. Yet, it remains unexplored whether current VLMs
  genuinely possess quality object-level image understanding capabilities
  determined from 'what objects are in the image?' or 'which object corresponds
  to a specified bounding box?'. Our findings reveal that the image understanding
  capabilities of current VLMs are strongly correlated with their zero-shot
  performance on Vision Language (VL) tasks. This suggests that prioritizing
  basic image understanding is crucial for VLMs to excel at VL tasks. To enhance
  object-level image understanding, we propose Crayon Large Language and Vision
  mOdel (CoLLaVO), which incorporates instruction tuning with crayon prompt as a
  new visual prompt tuning scheme based on panoptic color maps. Furthermore, we
  present a learning strategy of Dual QLoRA to preserve object-level image
  understanding without forgetting it during visual instruction tuning, thereby
  achieving a significant leap in zero-shot numerous VL benchmarks.
paper_ai_keywords: Large Language Models (LLMs) Vision Language Models (VLMs) versatile general-purpose model object-level image understanding bounding box zero-shot performance Vision Language (VL) tasks Crayon Large Language and Vision mOdel (CoLLaVO) instruction tuning crayon prompt panoptic color maps Dual QLoRA zero-shot VL benchmarks