Visual In-Context Prompting
In-context prompting in large language models (LLMs) has become a prevalent
approach to improve zero-shot capabilities, but this idea is less explored in
the vision domain. Existing visual prompting methods focus on referring
segmentation to segment the most relevant object, falling short of addressing
many generic vision tasks like open-set segmentation and detection. In this
paper, we introduce a universal visual in-context prompting framework for both
tasks. In particular, we build on top of an encoder-decoder architecture, and
develop a versatile prompt encoder to support a variety of prompts like
strokes, boxes, and points. We further enhance it to take an arbitrary number
of reference image segments as the context. Our extensive explorations show
that the proposed visual in-context prompting elicits extraordinary referring
and generic segmentation capabilities to refer and detect, yielding competitive
performance to close-set in-domain datasets and showing promising results on
many open-set segmentation datasets. By joint training on COCO and SA-1B, our
model achieves 57.7 PQ on COCO and 23.2 PQ on ADE20K. Code will be
available at https://github.com/UX-Decoder/DINOv.
655eeba0c073ea9cf858ebd8
Feng Li
655eeba0c073ea9cf858ebd9
Qing Jiang
admin_assigned
2023-11-23T16:22:17.025Z
647f46b6838ac3601fc89852
https://cdn-avatars.huggingface.co/v1/production/uploads/647f46b6838ac3601fc89852/N5cr1MFEtgKLJ4sVAhS04.jpeg
Qing Jiang
user
Mountchicken
655eeba0c073ea9cf858ebda
Hao Zhang
655eeba0c073ea9cf858ebdb
Tianhe Ren
admin_assigned
2023-11-23T16:23:46.736Z
6357ba47d3ec8a8aca6ea254
/avatars/2deb725505aa510a712d91bd0b6b1967.svg
Ren Tianhe
user
rentianhe
655eeba0c073ea9cf858ebdc
Shilong Liu
admin_assigned
2023-11-23T16:23:54.353Z
638b13c0c1d591879698f4e2
https://cdn-avatars.huggingface.co/v1/production/uploads/638b13c0c1d591879698f4e2/X8X4EWMXuzhBpG62wO2xS.jpeg
Shilong Liu
user
ShilongLiu
655eeba0c073ea9cf858ebdd
Xueyan Zou
admin_assigned
2023-11-23T16:24:05.130Z
62520988818a5dc29ab91d6f
https://cdn-avatars.huggingface.co/v1/production/uploads/1671670066375-62520988818a5dc29ab91d6f.png
Xueyan Zou
user
xueyanz
655eeba0c073ea9cf858ebde
Huaizhe Xu
admin_assigned
2023-11-23T16:24:15.706Z
62a942ff3d8a300f79e08dfc
/avatars/eef061293cb1523960a1d48845d6aa88.svg
Xu
user
Huaizhe
655eeba0c073ea9cf858ebdf
Hongyang Li
admin_assigned
2023-11-23T16:25:28.249Z
6499b0184936457997180c90
/avatars/b8be7bfabf746639e30330f5f623f560.svg
Hongyang Li
user
compileme
655eeba0c073ea9cf858ebe0
Chunyuan Li
admin_assigned
2023-11-23T16:25:34.810Z
62aba526cae4462c0c6caa0f
/avatars/430560ec2c2547f819225769ab432f30.svg
Chunyuan Li
user
Chunyuan24
655eeba0c073ea9cf858ebe1
Jianwei Yang
655eeba0c073ea9cf858ebe2
Lei Zhang
655eeba0c073ea9cf858ebe3
Jianfeng Gao
admin_assigned
2023-11-23T16:26:27.433Z
641904caf9d6f1d772ec7af7
/avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
Jianfeng Gao
user
wyngjf
Visual In-Context Prompting
In-context prompting in large language models (LLMs) has become a prevalent
approach to improve zero-shot capabilities, but this idea is less explored in
the vision domain. Existing visual prompting methods focus on referring
segmentation to segment the most relevant object, falling short of addressing
many generic vision tasks like open-set segmentation and detection. In this
paper, we introduce a universal visual in-context prompting framework for both
tasks. In particular, we build on top of an encoder-decoder architecture, and
develop a versatile prompt encoder to support a variety of prompts like
strokes, boxes, and points. We further enhance it to take an arbitrary number
of reference image segments as the context. Our extensive explorations show
that the proposed visual in-context prompting elicits extraordinary referring
and generic segmentation capabilities to refer and detect, yielding competitive
performance to close-set in-domain datasets and showing promising results on
many open-set segmentation datasets. By joint training on COCO and SA-1B, our
model achieves 57.7 PQ on COCO and 23.2 PQ on ADE20K. Code will be
available at https://github.com/UX-Decoder/DINOv.
A universal visual in-context prompting framework enhances zero-shot capabilities for both referring and generic vision tasks by using a versatile prompt encoder and reference image segments as context.
in-context prompting large language models (LLMs) encoder-decoder architecture prompt encoder strokes boxes points open-set segmentation detection COCO SA-1B ADE20K PQ