title: The All-Seeing Project: Towards Panoptic Visual Recognition and
    Understanding of the Open World
summary: We present the All-Seeing (AS) project: a large-scale data and model for
  recognizing and understanding everything in the open world. Using a scalable
  data engine that incorporates human feedback and efficient models in the loop,
  we create a new dataset (AS-1B) with over 1 billion regions annotated with
  semantic tags, question-answering pairs, and detailed captions. It covers a
  wide range of 3.5 million common and rare concepts in the real world, and has
  132.2 billion tokens that describe the concepts and their attributes.
  Leveraging this new dataset, we develop the All-Seeing model (ASM), a unified
  framework for panoptic visual recognition and understanding. The model is
  trained with open-ended language prompts and locations, which allows it to
  generalize to various vision and language tasks with remarkable zero-shot
  performance, including region-text retrieval, region recognition, captioning,
  and question-answering. We hope that this project can serve as a foundation for
  vision-language artificial general intelligence research. Models and the
  dataset shall be released at https://github.com/OpenGVLab/All-Seeing, and demo
  can be seen at https://huggingface.co/spaces/OpenGVLab/all-seeing.
paper_authors._id: 64cc847d1af278541d922dcd
paper_authors.name: Weiyun Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-04T12:37:39.193Z
paper_authors._id: 619507e7b74b6c591f794340
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/619507e7b74b6c591f794340/JbPDoy6Ko1V1-6oJJwFV8.jpeg
paper_authors.fullname: Weiyun Wang
paper_authors.type: user
paper_authors.user: Weiyun1025
paper_authors._id: 64cc847d1af278541d922dce
paper_authors.name: Min Shi
paper_authors._id: 64cc847d1af278541d922dcf
paper_authors.name: Qingyun Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-04T12:39:01.086Z
paper_authors._id: 637f100c1dbae09191088e4c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1669271529782-noauth.jpeg
paper_authors.fullname: Qingyun Li
paper_authors.type: user
paper_authors.user: Qingyun
paper_authors._id: 64cc847d1af278541d922dd0
paper_authors.name: Wenhai Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-04T12:39:42.218Z
paper_authors._id: 644f58e6f8b353c94914c7e0
paper_authors.avatarUrl: /avatars/16c7e80aafbea9a52cfc309068eedc9a.svg
paper_authors.fullname: wangwenhai
paper_authors.type: user
paper_authors.user: yicixi
paper_authors._id: 64cc847d1af278541d922dd1
paper_authors.name: Zhenhang Huang
paper_authors._id: 64cc847d1af278541d922dd2
paper_authors.name: Linjie Xing
paper_authors._id: 64cc847d1af278541d922dd3
paper_authors.name: Zhe Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-09T14:51:20.313Z
paper_authors._id: 64119264f0f81eb569e0d569
paper_authors.avatarUrl: /avatars/00dccedea51ac7d2f398ca9d65a8e78b.svg
paper_authors.fullname: Zhe Chen
paper_authors.type: user
paper_authors.user: czczup
paper_authors._id: 64cc847d1af278541d922dd4
paper_authors.name: Hao Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-16T09:44:27.755Z
paper_authors._id: 64b9033777ae61bcc80aa4f3
paper_authors.avatarUrl: /avatars/408c335395c79f3df69fd9bf70abc312.svg
paper_authors.fullname: Hao Li
paper_authors.type: user
paper_authors.user: cpsxhao
paper_authors._id: 64cc847d1af278541d922dd5
paper_authors.name: Xizhou Zhu
paper_authors._id: 64cc847d1af278541d922dd6
paper_authors.name: Zhiguo Cao
paper_authors._id: 64cc847d1af278541d922dd7
paper_authors.name: Yushi Chen
paper_authors._id: 64cc847d1af278541d922dd8
paper_authors.name: Tong Lu
paper_authors._id: 64cc847d1af278541d922dd9
paper_authors.name: Jifeng Dai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-04T12:41:59.461Z
paper_authors._id: 64686f7172d9180d4ac8b4e4
paper_authors.avatarUrl: /avatars/db67dd6c4b2b41054ddcce5a18ade6f8.svg
paper_authors.fullname: Jifeng Dai
paper_authors.type: user
paper_authors.user: daijifeng
paper_authors._id: 64cc847d1af278541d922dda
paper_authors.name: Yu Qiao
paper_title: The All-Seeing Project: Towards Panoptic Visual Recognition and
    Understanding of the Open World
paper_summary: We present the All-Seeing (AS) project: a large-scale data and model for
  recognizing and understanding everything in the open world. Using a scalable
  data engine that incorporates human feedback and efficient models in the loop,
  we create a new dataset (AS-1B) with over 1 billion regions annotated with
  semantic tags, question-answering pairs, and detailed captions. It covers a
  wide range of 3.5 million common and rare concepts in the real world, and has
  132.2 billion tokens that describe the concepts and their attributes.
  Leveraging this new dataset, we develop the All-Seeing model (ASM), a unified
  framework for panoptic visual recognition and understanding. The model is
  trained with open-ended language prompts and locations, which allows it to
  generalize to various vision and language tasks with remarkable zero-shot
  performance, including region-text retrieval, region recognition, captioning,
  and question-answering. We hope that this project can serve as a foundation for
  vision-language artificial general intelligence research. Models and the
  dataset shall be released at https://github.com/OpenGVLab/All-Seeing, and demo
  can be seen at https://huggingface.co/spaces/OpenGVLab/all-seeing.
paper_ai_keywords: panoptic visual recognition open-ended language prompts zero-shot performance region-text retrieval region recognition captioning question-answering