UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended
  Language Interface
Generalist models have achieved remarkable success in both language and
vision-language tasks, showcasing the potential of unified modeling. However,
effectively integrating fine-grained perception tasks like detection and
segmentation into these models remains a significant challenge. This is
primarily because these tasks often rely heavily on task-specific designs and
architectures that can complicate the modeling process. To address this
challenge, we present \ours, a framework that Unifies
Fine-grained visual perception tasks through an Open-ended
language interface. By transforming all perception targets into the language
space, \ours unifies object-level detection, pixel-level segmentation, and
image-level vision-language tasks into a single model. Additionally, we
introduce a novel embedding retrieval approach that relies solely on the
language interface to support segmentation tasks. Our framework bridges the gap
between fine-grained perception and vision-language tasks, significantly
simplifying architectural design and training strategies while achieving
comparable or superior performance to methods with intricate task-specific
designs. After multi-task training on five standard visual perception datasets,
\ours outperforms the previous state-of-the-art generalist models by 12.3 mAP
on COCO instance segmentation and 3.3 mIoU on ADE20K semantic segmentation.
Furthermore, our method seamlessly integrates with existing MLLMs, effectively
combining fine-grained perception capabilities with their advanced language
abilities, thereby enabling more challenging tasks such as reasoning
segmentation. Code and models will be publicly available.
67c6b46e8389d77f5ba87179
Hao Tang
claimed_verified
2025-03-04T08:34:43.034Z
6585493b53c37507639fe3ba
/avatars/b7e71d4fa5ebb89a7ed6b2a8313687b5.svg
Hao Tang
user
kanashi6
67c6b46e8389d77f5ba8717a
Chenwei Xie
admin_assigned
2025-03-05T11:10:05.785Z
66592c72f4124d863fd55574
/avatars/98f0d5e6ba3728e8a1164aa5188a3298.svg
Chenwei Xie
user
chenweix7
67c6b46e8389d77f5ba8717b
Haiyang Wang
admin_assigned
2025-03-05T11:10:27.485Z
65f43c3cc9940817caaf4434
/avatars/ecec2856ba7a7d3421a2071a0a88800b.svg
Haiyang Wang
user
Haiyang-W
67c6b46e8389d77f5ba8717c
Xiaoyi Bao
admin_assigned
2025-03-05T11:10:34.475Z
6615fc38d6af2bbe8bd86a72
/avatars/812277f2f5fd673f2e18988cd499c74d.svg
Xiaoyi Bao
user
Shawnee-bxy
67c6b46e8389d77f5ba8717d
Tingyu Weng
admin_assigned
2025-03-05T11:10:41.931Z
6489713d06a6dc54460725bb
/avatars/11b0e766eb8ccf67511e04a0c75e171e.svg
Tingyu Weng
user
windmillknight
67c6b46e8389d77f5ba8717e
Pandeng Li
67c6b46e8389d77f5ba8717f
Yun Zheng
67c6b46e8389d77f5ba87180
Liwei Wang
admin_assigned
2025-03-05T11:14:43.973Z
6497272b6401266eb6752b89
/avatars/d94a011660f4cad5344756027840dc30.svg
Liwei Wang
user
asdfg80
UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended
  Language Interface
Generalist models have achieved remarkable success in both language and
vision-language tasks, showcasing the potential of unified modeling. However,
effectively integrating fine-grained perception tasks like detection and
segmentation into these models remains a significant challenge. This is
primarily because these tasks often rely heavily on task-specific designs and
architectures that can complicate the modeling process. To address this
challenge, we present \ours, a framework that Unifies
Fine-grained visual perception tasks through an Open-ended
language interface. By transforming all perception targets into the language
space, \ours unifies object-level detection, pixel-level segmentation, and
image-level vision-language tasks into a single model. Additionally, we
introduce a novel embedding retrieval approach that relies solely on the
language interface to support segmentation tasks. Our framework bridges the gap
between fine-grained perception and vision-language tasks, significantly
simplifying architectural design and training strategies while achieving
comparable or superior performance to methods with intricate task-specific
designs. After multi-task training on five standard visual perception datasets,
\ours outperforms the previous state-of-the-art generalist models by 12.3 mAP
on COCO instance segmentation and 3.3 mIoU on ADE20K semantic segmentation.
Furthermore, our method seamlessly integrates with existing MLLMs, effectively
combining fine-grained perception capabilities with their advanced language
abilities, thereby enabling more challenging tasks such as reasoning
segmentation. Code and models will be publicly available.
A framework unifies fine-grained visual perception tasks with vision-language tasks through a language interface, demonstrating superior performance across multiple datasets and tasks.
generalist models unified modeling fine-grained perception tasks object-level detection pixel-level segmentation image-level vision-language tasks embedding retrieval approach language interface multi-task training COCO instance segmentation ADE20K semantic segmentation reasoning segmentation