title: DINO-X: A Unified Vision Model for Open-World Object Detection and
    Understanding
summary: In this paper, we introduce DINO-X, which is a unified object-centric vision
  model developed by IDEA Research with the best open-world object detection
  performance to date. DINO-X employs the same Transformer-based encoder-decoder
  architecture as Grounding DINO 1.5 to pursue an object-level representation for
  open-world object understanding. To make long-tailed object detection easy,
  DINO-X extends its input options to support text prompt, visual prompt, and
  customized prompt. With such flexible prompt options, we develop a universal
  object prompt to support prompt-free open-world detection, making it possible
  to detect anything in an image without requiring users to provide any prompt.
  To enhance the model's core grounding capability, we have constructed a
  large-scale dataset with over 100 million high-quality grounding samples,
  referred to as Grounding-100M, for advancing the model's open-vocabulary
  detection performance. Pre-training on such a large-scale grounding dataset
  leads to a foundational object-level representation, which enables DINO-X to
  integrate multiple perception heads to simultaneously support multiple object
  perception and understanding tasks, including detection, segmentation, pose
  estimation, object captioning, object-based QA, etc. Experimental results
  demonstrate the superior performance of DINO-X. Specifically, the DINO-X Pro
  model achieves 56.0 AP, 59.8 AP, and 52.4 AP on the COCO, LVIS-minival, and
  LVIS-val zero-shot object detection benchmarks, respectively. Notably, it
  scores 63.3 AP and 56.5 AP on the rare classes of LVIS-minival and LVIS-val
  benchmarks, both improving the previous SOTA performance by 5.8 AP. Such a
  result underscores its significantly improved capacity for recognizing
  long-tailed objects.
paper_authors._id: 6740b5bbb7c8a9b0ff8a9392
paper_authors.name: Tianhe Ren
paper_authors._id: 6740b5bbb7c8a9b0ff8a9393
paper_authors.name: Yihao Chen
paper_authors._id: 6740b5bbb7c8a9b0ff8a9394
paper_authors.name: Qing Jiang
paper_authors._id: 6740b5bbb7c8a9b0ff8a9395
paper_authors.name: Zhaoyang Zeng
paper_authors._id: 6740b5bbb7c8a9b0ff8a9396
paper_authors.name: Yuda Xiong
paper_authors._id: 6740b5bbb7c8a9b0ff8a9397
paper_authors.name: Wenlong Liu
paper_authors._id: 6740b5bbb7c8a9b0ff8a9398
paper_authors.name: Zhengyu Ma
paper_authors._id: 6740b5bbb7c8a9b0ff8a9399
paper_authors.name: Junyi Shen
paper_authors._id: 6740b5bbb7c8a9b0ff8a939a
paper_authors.name: Yuan Gao
paper_authors._id: 6740b5bbb7c8a9b0ff8a939b
paper_authors.name: Xiaoke Jiang
paper_authors._id: 6740b5bbb7c8a9b0ff8a939c
paper_authors.name: Xingyu Chen
paper_authors._id: 6740b5bbb7c8a9b0ff8a939d
paper_authors.name: Zhuheng Song
paper_authors._id: 6740b5bbb7c8a9b0ff8a939e
paper_authors.name: Yuhong Zhang
paper_authors._id: 6740b5bbb7c8a9b0ff8a939f
paper_authors.name: Hongjie Huang
paper_authors._id: 6740b5bbb7c8a9b0ff8a93a0
paper_authors.name: Han Gao
paper_authors._id: 6740b5bbb7c8a9b0ff8a93a1
paper_authors.name: Shilong Liu
paper_authors._id: 6740b5bbb7c8a9b0ff8a93a2
paper_authors.name: Hao Zhang
paper_authors._id: 6740b5bbb7c8a9b0ff8a93a3
paper_authors.name: Feng Li
paper_authors._id: 6740b5bbb7c8a9b0ff8a93a4
paper_authors.name: Kent Yu
paper_authors._id: 6740b5bbb7c8a9b0ff8a93a5
paper_authors.name: Lei Zhang
paper_title: DINO-X: A Unified Vision Model for Open-World Object Detection and
    Understanding
paper_summary: In this paper, we introduce DINO-X, which is a unified object-centric vision
  model developed by IDEA Research with the best open-world object detection
  performance to date. DINO-X employs the same Transformer-based encoder-decoder
  architecture as Grounding DINO 1.5 to pursue an object-level representation for
  open-world object understanding. To make long-tailed object detection easy,
  DINO-X extends its input options to support text prompt, visual prompt, and
  customized prompt. With such flexible prompt options, we develop a universal
  object prompt to support prompt-free open-world detection, making it possible
  to detect anything in an image without requiring users to provide any prompt.
  To enhance the model's core grounding capability, we have constructed a
  large-scale dataset with over 100 million high-quality grounding samples,
  referred to as Grounding-100M, for advancing the model's open-vocabulary
  detection performance. Pre-training on such a large-scale grounding dataset
  leads to a foundational object-level representation, which enables DINO-X to
  integrate multiple perception heads to simultaneously support multiple object
  perception and understanding tasks, including detection, segmentation, pose
  estimation, object captioning, object-based QA, etc. Experimental results
  demonstrate the superior performance of DINO-X. Specifically, the DINO-X Pro
  model achieves 56.0 AP, 59.8 AP, and 52.4 AP on the COCO, LVIS-minival, and
  LVIS-val zero-shot object detection benchmarks, respectively. Notably, it
  scores 63.3 AP and 56.5 AP on the rare classes of LVIS-minival and LVIS-val
  benchmarks, both improving the previous SOTA performance by 5.8 AP. Such a
  result underscores its significantly improved capacity for recognizing
  long-tailed objects.
paper_ai_keywords: Transformer-based encoder-decoder architecture Grounding DINO 1.5 long-tailed object detection text prompt visual prompt customized prompt universal object prompt Grounding-100M open-vocabulary detection perception heads object detection object segmentation pose estimation object captioning object-based QA COCO LVIS-minival LVIS-val zero-shot object detection AP (Average Precision)