title: VisionLLM: Large Language Model is also an Open-Ended Decoder for
    Vision-Centric Tasks
summary: Large language models (LLMs) have notably accelerated progress towards
  artificial general intelligence (AGI), with their impressive zero-shot capacity
  for user-tailored tasks, endowing them with immense potential across a range of
  applications. However, in the field of computer vision, despite the
  availability of numerous powerful vision foundation models (VFMs), they are
  still restricted to tasks in a pre-defined form, struggling to match the
  open-ended task capabilities of LLMs. In this work, we present an LLM-based
  framework for vision-centric tasks, termed VisionLLM. This framework provides a
  unified perspective for vision and language tasks by treating images as a
  foreign language and aligning vision-centric tasks with language tasks that can
  be flexibly defined and managed using language instructions. An LLM-based
  decoder can then make appropriate predictions based on these instructions for
  open-ended tasks. Extensive experiments show that the proposed VisionLLM can
  achieve different levels of task customization through language instructions,
  from fine-grained object-level to coarse-grained task-level customization, all
  with good results. It's noteworthy that, with a generalist LLM-based framework,
  our model can achieve over 60\% mAP on COCO, on par with detection-specific
  models. We hope this model can set a new baseline for generalist vision and
  language models. The demo shall be released based on
  https://github.com/OpenGVLab/InternGPT. The code shall be released at
  https://github.com/OpenGVLab/VisionLLM.
paper_authors._id: 6466fcce14e059dde8bc6356
paper_authors.name: Wenhai Wang
paper_authors._id: 6466fcce14e059dde8bc6357
paper_authors.name: Zhe Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-19T11:11:56.893Z
paper_authors._id: 64119264f0f81eb569e0d569
paper_authors.avatarUrl: /avatars/00dccedea51ac7d2f398ca9d65a8e78b.svg
paper_authors.fullname: Zhe Chen
paper_authors.type: user
paper_authors.user: czczup
paper_authors._id: 6466fcce14e059dde8bc6358
paper_authors.name: Xiaokang Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-05T15:57:22.272Z
paper_authors._id: 6635e701420baf7bc3f93561
paper_authors.avatarUrl: /avatars/cdbb3085fee73ac520888977e2c575ea.svg
paper_authors.fullname: Xiaokang Chen
paper_authors.type: user
paper_authors.user: CharlesCXK
paper_authors._id: 6466fcce14e059dde8bc6359
paper_authors.name: Jiannan Wu
paper_authors._id: 6466fcce14e059dde8bc635a
paper_authors.name: Xizhou Zhu
paper_authors._id: 6466fcce14e059dde8bc635b
paper_authors.name: Gang Zeng
paper_authors._id: 6466fcce14e059dde8bc635c
paper_authors.name: Ping Luo
paper_authors._id: 6466fcce14e059dde8bc635d
paper_authors.name: Tong Lu
paper_authors._id: 6466fcce14e059dde8bc635e
paper_authors.name: Jie Zhou
paper_authors._id: 6466fcce14e059dde8bc635f
paper_authors.name: Yu Qiao
paper_authors._id: 6466fcce14e059dde8bc6360
paper_authors.name: Jifeng Dai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-22T09:36:54.865Z
paper_authors._id: 64686f7172d9180d4ac8b4e4
paper_authors.avatarUrl: /avatars/db67dd6c4b2b41054ddcce5a18ade6f8.svg
paper_authors.fullname: Jifeng Dai
paper_authors.type: user
paper_authors.user: daijifeng
paper_title: VisionLLM: Large Language Model is also an Open-Ended Decoder for
    Vision-Centric Tasks
paper_summary: Large language models (LLMs) have notably accelerated progress towards
  artificial general intelligence (AGI), with their impressive zero-shot capacity
  for user-tailored tasks, endowing them with immense potential across a range of
  applications. However, in the field of computer vision, despite the
  availability of numerous powerful vision foundation models (VFMs), they are
  still restricted to tasks in a pre-defined form, struggling to match the
  open-ended task capabilities of LLMs. In this work, we present an LLM-based
  framework for vision-centric tasks, termed VisionLLM. This framework provides a
  unified perspective for vision and language tasks by treating images as a
  foreign language and aligning vision-centric tasks with language tasks that can
  be flexibly defined and managed using language instructions. An LLM-based
  decoder can then make appropriate predictions based on these instructions for
  open-ended tasks. Extensive experiments show that the proposed VisionLLM can
  achieve different levels of task customization through language instructions,
  from fine-grained object-level to coarse-grained task-level customization, all
  with good results. It's noteworthy that, with a generalist LLM-based framework,
  our model can achieve over 60\% mAP on COCO, on par with detection-specific
  models. We hope this model can set a new baseline for generalist vision and
  language models. The demo shall be released based on
  https://github.com/OpenGVLab/InternGPT. The code shall be released at
  https://github.com/OpenGVLab/VisionLLM.
paper_ai_keywords: large language models (LLMs) artificial general intelligence (AGI) zero-shot capacity user-tailored tasks vision foundation models (VFMs) VisionLLM unified perspective language instructions open-ended tasks image as a large language language-specific definitions fine-grained object-level coarse-grained task-level mAP COCO detection-specific models generalist vision and language models