title: Visual In-Context Prompting
summary: In-context prompting in large language models (LLMs) has become a prevalent
  approach to improve zero-shot capabilities, but this idea is less explored in
  the vision domain. Existing visual prompting methods focus on referring
  segmentation to segment the most relevant object, falling short of addressing
  many generic vision tasks like open-set segmentation and detection. In this
  paper, we introduce a universal visual in-context prompting framework for both
  tasks. In particular, we build on top of an encoder-decoder architecture, and
  develop a versatile prompt encoder to support a variety of prompts like
  strokes, boxes, and points. We further enhance it to take an arbitrary number
  of reference image segments as the context. Our extensive explorations show
  that the proposed visual in-context prompting elicits extraordinary referring
  and generic segmentation capabilities to refer and detect, yielding competitive
  performance to close-set in-domain datasets and showing promising results on
  many open-set segmentation datasets. By joint training on COCO and SA-1B, our
  model achieves 57.7 PQ on COCO and 23.2 PQ on ADE20K. Code will be
  available at https://github.com/UX-Decoder/DINOv.
paper_authors._id: 655eeba0c073ea9cf858ebd8
paper_authors.name: Feng Li
paper_authors._id: 655eeba0c073ea9cf858ebd9
paper_authors.name: Qing Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T16:22:17.025Z
paper_authors._id: 647f46b6838ac3601fc89852
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647f46b6838ac3601fc89852/N5cr1MFEtgKLJ4sVAhS04.jpeg
paper_authors.fullname: Qing Jiang
paper_authors.type: user
paper_authors.user: Mountchicken
paper_authors._id: 655eeba0c073ea9cf858ebda
paper_authors.name: Hao Zhang
paper_authors._id: 655eeba0c073ea9cf858ebdb
paper_authors.name: Tianhe Ren
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T16:23:46.736Z
paper_authors._id: 6357ba47d3ec8a8aca6ea254
paper_authors.avatarUrl: /avatars/2deb725505aa510a712d91bd0b6b1967.svg
paper_authors.fullname: Ren Tianhe
paper_authors.type: user
paper_authors.user: rentianhe
paper_authors._id: 655eeba0c073ea9cf858ebdc
paper_authors.name: Shilong Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T16:23:54.353Z
paper_authors._id: 638b13c0c1d591879698f4e2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/638b13c0c1d591879698f4e2/X8X4EWMXuzhBpG62wO2xS.jpeg
paper_authors.fullname: Shilong Liu
paper_authors.type: user
paper_authors.user: ShilongLiu
paper_authors._id: 655eeba0c073ea9cf858ebdd
paper_authors.name: Xueyan Zou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T16:24:05.130Z
paper_authors._id: 62520988818a5dc29ab91d6f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1671670066375-62520988818a5dc29ab91d6f.png
paper_authors.fullname: Xueyan Zou
paper_authors.type: user
paper_authors.user: xueyanz
paper_authors._id: 655eeba0c073ea9cf858ebde
paper_authors.name: Huaizhe Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T16:24:15.706Z
paper_authors._id: 62a942ff3d8a300f79e08dfc
paper_authors.avatarUrl: /avatars/eef061293cb1523960a1d48845d6aa88.svg
paper_authors.fullname: Xu
paper_authors.type: user
paper_authors.user: Huaizhe
paper_authors._id: 655eeba0c073ea9cf858ebdf
paper_authors.name: Hongyang Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T16:25:28.249Z
paper_authors._id: 6499b0184936457997180c90
paper_authors.avatarUrl: /avatars/b8be7bfabf746639e30330f5f623f560.svg
paper_authors.fullname: Hongyang Li
paper_authors.type: user
paper_authors.user: compileme
paper_authors._id: 655eeba0c073ea9cf858ebe0
paper_authors.name: Chunyuan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T16:25:34.810Z
paper_authors._id: 62aba526cae4462c0c6caa0f
paper_authors.avatarUrl: /avatars/430560ec2c2547f819225769ab432f30.svg
paper_authors.fullname: Chunyuan Li
paper_authors.type: user
paper_authors.user: Chunyuan24
paper_authors._id: 655eeba0c073ea9cf858ebe1
paper_authors.name: Jianwei Yang
paper_authors._id: 655eeba0c073ea9cf858ebe2
paper_authors.name: Lei Zhang
paper_authors._id: 655eeba0c073ea9cf858ebe3
paper_authors.name: Jianfeng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T16:26:27.433Z
paper_authors._id: 641904caf9d6f1d772ec7af7
paper_authors.avatarUrl: /avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
paper_authors.fullname: Jianfeng Gao
paper_authors.type: user
paper_authors.user: wyngjf
paper_title: Visual In-Context Prompting
paper_summary: In-context prompting in large language models (LLMs) has become a prevalent
  approach to improve zero-shot capabilities, but this idea is less explored in
  the vision domain. Existing visual prompting methods focus on referring
  segmentation to segment the most relevant object, falling short of addressing
  many generic vision tasks like open-set segmentation and detection. In this
  paper, we introduce a universal visual in-context prompting framework for both
  tasks. In particular, we build on top of an encoder-decoder architecture, and
  develop a versatile prompt encoder to support a variety of prompts like
  strokes, boxes, and points. We further enhance it to take an arbitrary number
  of reference image segments as the context. Our extensive explorations show
  that the proposed visual in-context prompting elicits extraordinary referring
  and generic segmentation capabilities to refer and detect, yielding competitive
  performance to close-set in-domain datasets and showing promising results on
  many open-set segmentation datasets. By joint training on COCO and SA-1B, our
  model achieves 57.7 PQ on COCO and 23.2 PQ on ADE20K. Code will be
  available at https://github.com/UX-Decoder/DINOv.
paper_ai_keywords: encoder-decoder architecture prompt encoder strokes boxes points reference image segments referring segmentation generic segmentation open-set segmentation detection close-set in-domain datasets PQ (Panoptic Quality) COCO SA-1B ADE20K