title: UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended
    Language Interface
summary: Generalist models have achieved remarkable success in both language and
  vision-language tasks, showcasing the potential of unified modeling. However,
  effectively integrating fine-grained perception tasks like detection and
  segmentation into these models remains a significant challenge. This is
  primarily because these tasks often rely heavily on task-specific designs and
  architectures that can complicate the modeling process. To address this
  challenge, we present \ours, a framework that Unifies
  Fine-grained visual perception tasks through an Open-ended
  language interface. By transforming all perception targets into the language
  space, \ours unifies object-level detection, pixel-level segmentation, and
  image-level vision-language tasks into a single model. Additionally, we
  introduce a novel embedding retrieval approach that relies solely on the
  language interface to support segmentation tasks. Our framework bridges the gap
  between fine-grained perception and vision-language tasks, significantly
  simplifying architectural design and training strategies while achieving
  comparable or superior performance to methods with intricate task-specific
  designs. After multi-task training on five standard visual perception datasets,
  \ours outperforms the previous state-of-the-art generalist models by 12.3 mAP
  on COCO instance segmentation and 3.3 mIoU on ADE20K semantic segmentation.
  Furthermore, our method seamlessly integrates with existing MLLMs, effectively
  combining fine-grained perception capabilities with their advanced language
  abilities, thereby enabling more challenging tasks such as reasoning
  segmentation. Code and models will be publicly available.
paper_authors._id: 67c6b46e8389d77f5ba87179
paper_authors.name: Hao Tang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-04T08:34:43.034Z
paper_authors._id: 6585493b53c37507639fe3ba
paper_authors.avatarUrl: /avatars/b7e71d4fa5ebb89a7ed6b2a8313687b5.svg
paper_authors.fullname: Hao Tang
paper_authors.type: user
paper_authors.user: kanashi6
paper_authors._id: 67c6b46e8389d77f5ba8717a
paper_authors.name: Chenwei Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-05T11:10:05.785Z
paper_authors._id: 66592c72f4124d863fd55574
paper_authors.avatarUrl: /avatars/98f0d5e6ba3728e8a1164aa5188a3298.svg
paper_authors.fullname: Chenwei Xie
paper_authors.type: user
paper_authors.user: chenweix7
paper_authors._id: 67c6b46e8389d77f5ba8717b
paper_authors.name: Haiyang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-05T11:10:27.485Z
paper_authors._id: 65f43c3cc9940817caaf4434
paper_authors.avatarUrl: /avatars/ecec2856ba7a7d3421a2071a0a88800b.svg
paper_authors.fullname: Haiyang Wang
paper_authors.type: user
paper_authors.user: Haiyang-W
paper_authors._id: 67c6b46e8389d77f5ba8717c
paper_authors.name: Xiaoyi Bao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-05T11:10:34.475Z
paper_authors._id: 6615fc38d6af2bbe8bd86a72
paper_authors.avatarUrl: /avatars/812277f2f5fd673f2e18988cd499c74d.svg
paper_authors.fullname: Xiaoyi Bao
paper_authors.type: user
paper_authors.user: Shawnee-bxy
paper_authors._id: 67c6b46e8389d77f5ba8717d
paper_authors.name: Tingyu Weng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-05T11:10:41.931Z
paper_authors._id: 6489713d06a6dc54460725bb
paper_authors.avatarUrl: /avatars/11b0e766eb8ccf67511e04a0c75e171e.svg
paper_authors.fullname: Tingyu Weng
paper_authors.type: user
paper_authors.user: windmillknight
paper_authors._id: 67c6b46e8389d77f5ba8717e
paper_authors.name: Pandeng Li
paper_authors._id: 67c6b46e8389d77f5ba8717f
paper_authors.name: Yun Zheng
paper_authors._id: 67c6b46e8389d77f5ba87180
paper_authors.name: Liwei Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-05T11:14:43.973Z
paper_authors._id: 6497272b6401266eb6752b89
paper_authors.avatarUrl: /avatars/d94a011660f4cad5344756027840dc30.svg
paper_authors.fullname: Liwei Wang
paper_authors.type: user
paper_authors.user: asdfg80
paper_title: UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended
    Language Interface
paper_summary: Generalist models have achieved remarkable success in both language and
  vision-language tasks, showcasing the potential of unified modeling. However,
  effectively integrating fine-grained perception tasks like detection and
  segmentation into these models remains a significant challenge. This is
  primarily because these tasks often rely heavily on task-specific designs and
  architectures that can complicate the modeling process. To address this
  challenge, we present \ours, a framework that Unifies
  Fine-grained visual perception tasks through an Open-ended
  language interface. By transforming all perception targets into the language
  space, \ours unifies object-level detection, pixel-level segmentation, and
  image-level vision-language tasks into a single model. Additionally, we
  introduce a novel embedding retrieval approach that relies solely on the
  language interface to support segmentation tasks. Our framework bridges the gap
  between fine-grained perception and vision-language tasks, significantly
  simplifying architectural design and training strategies while achieving
  comparable or superior performance to methods with intricate task-specific
  designs. After multi-task training on five standard visual perception datasets,
  \ours outperforms the previous state-of-the-art generalist models by 12.3 mAP
  on COCO instance segmentation and 3.3 mIoU on ADE20K semantic segmentation.
  Furthermore, our method seamlessly integrates with existing MLLMs, effectively
  combining fine-grained perception capabilities with their advanced language
  abilities, thereby enabling more challenging tasks such as reasoning
  segmentation. Code and models will be publicly available.
paper_ai_keywords: fine-grained perception tasks detection segmentation unified modeling convolutional multimours language interface object-level detection pixel-level questions embedding retrieval COCO instance segmentation ADE20K semantic segmentation MLLMs (Multimodal Large Language Models) reasoning