title: VCoder: Versatile Vision Encoders for Multimodal Large Language Models
summary: Humans possess the remarkable skill of Visual Perception, the ability to see
  and understand the seen, helping them make sense of the visual world and, in
  turn, reason. Multimodal Large Language Models (MLLM) have recently achieved
  impressive performance on vision-language tasks ranging from visual
  question-answering and image captioning to visual reasoning and image
  generation. However, when prompted to identify or count (perceive) the entities
  in a given image, existing MLLM systems fail. Working towards developing an
  accurate MLLM system for perception and reasoning, we propose using Versatile
  vision enCoders (VCoder) as perception eyes for Multimodal LLMs. We feed the
  VCoder with perception modalities such as segmentation or depth maps, improving
  the MLLM's perception abilities. Secondly, we leverage the images from COCO and
  outputs from off-the-shelf vision perception models to create our COCO
  Segmentation Text (COST) dataset for training and evaluating MLLMs on the
  object perception task. Thirdly, we introduce metrics to assess the object
  perception abilities in MLLMs on our COST dataset. Lastly, we provide extensive
  experimental evidence proving the VCoder's improved object-level perception
  skills over existing Multimodal LLMs, including GPT-4V. We open-source our
  dataset, code, and models to promote research. We open-source our code at
  https://github.com/SHI-Labs/VCoder
paper_authors._id: 65894bc265df457a55d9bec1
paper_authors.name: Jitesh Jain
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-08T08:49:17.508Z
paper_authors._id: 623dfe96dcda6a715304cbca
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/623dfe96dcda6a715304cbca/B7V_IbQNXAcotOKE_mJQ4.jpeg
paper_authors.fullname: Jitesh Jain
paper_authors.type: user
paper_authors.user: praeclarumjj3
paper_authors._id: 65894bc265df457a55d9bec2
paper_authors.name: Jianwei Yang
paper_authors._id: 65894bc265df457a55d9bec3
paper_authors.name: Humphrey Shi
paper_title: VCoder: Versatile Vision Encoders for Multimodal Large Language Models
paper_summary: Humans possess the remarkable skill of Visual Perception, the ability to see
  and understand the seen, helping them make sense of the visual world and, in
  turn, reason. Multimodal Large Language Models (MLLM) have recently achieved
  impressive performance on vision-language tasks ranging from visual
  question-answering and image captioning to visual reasoning and image
  generation. However, when prompted to identify or count (perceive) the entities
  in a given image, existing MLLM systems fail. Working towards developing an
  accurate MLLM system for perception and reasoning, we propose using Versatile
  vision enCoders (VCoder) as perception eyes for Multimodal LLMs. We feed the
  VCoder with perception modalities such as segmentation or depth maps, improving
  the MLLM's perception abilities. Secondly, we leverage the images from COCO and
  outputs from off-the-shelf vision perception models to create our COCO
  Segmentation Text (COST) dataset for training and evaluating MLLMs on the
  object perception task. Thirdly, we introduce metrics to assess the object
  perception abilities in MLLMs on our COST dataset. Lastly, we provide extensive
  experimental evidence proving the VCoder's improved object-level perception
  skills over existing Multimodal LLMs, including GPT-4V. We open-source our
  dataset, code, and models to promote research. We open-source our code at
  https://github.com/SHI-Labs/VCoder
paper_ai_summary: VCoder, a versatile vision encoder, enhances multimodal LLMs' perception and reasoning abilities, particularly in object identification and counting, using specialized perception modalities and a newly created dataset.
paper_ai_keywords: Versatile vision enCoders perception eyes segmentation depth maps COCO Segmentation Text COST dataset object perception Multimodal LLMs GPT-4V