YOLOE: Real-Time Seeing Anything
Object detection and segmentation are widely employed in computer vision
applications, yet conventional models like YOLO series, while efficient and
accurate, are limited by predefined categories, hindering adaptability in open
scenarios. Recent open-set methods leverage text prompts, visual cues, or
prompt-free paradigm to overcome this, but often compromise between performance
and efficiency due to high computational demands or deployment complexity. In
this work, we introduce YOLOE, which integrates detection and segmentation
across diverse open prompt mechanisms within a single highly efficient model,
achieving real-time seeing anything. For text prompts, we propose
Re-parameterizable Region-Text Alignment (RepRTA) strategy. It refines
pretrained textual embeddings via a re-parameterizable lightweight auxiliary
network and enhances visual-textual alignment with zero inference and
transferring overhead. For visual prompts, we present Semantic-Activated Visual
Prompt Encoder (SAVPE). It employs decoupled semantic and activation branches
to bring improved visual embedding and accuracy with minimal complexity. For
prompt-free scenario, we introduce Lazy Region-Prompt Contrast (LRPC) strategy.
It utilizes a built-in large vocabulary and specialized embedding to identify
all objects, avoiding costly language model dependency. Extensive experiments
show YOLOE's exceptional zero-shot performance and transferability with high
inference efficiency and low training cost. Notably, on LVIS, with 3times
less training cost and 1.4times inference speedup, YOLOE-v8-S surpasses
YOLO-Worldv2-S by 3.5 AP. When transferring to COCO, YOLOE-v8-L achieves 0.6
AP^b and 0.4 AP^m gains over closed-set YOLOv8-L with nearly 4times less
training time. Code and models are available at
https://github.com/THU-MIG/yoloe.
67cfaaed7f229132171f596b
Ao Wang
admin_assigned
2025-04-02T16:10:42.022Z
628ece6054698ce61d1e7be3
/avatars/c6ab33843fd0d8ef003650c1094214c0.svg
Ao Wang
user
jameslahm
67cfaaed7f229132171f596c
Lihao Liu
67cfaaed7f229132171f596d
Hui Chen
67cfaaed7f229132171f596e
Zijia Lin
67cfaaed7f229132171f596f
Jungong Han
67cfaaed7f229132171f5970
Guiguang Ding
YOLOE: Real-Time Seeing Anything
Object detection and segmentation are widely employed in computer vision
applications, yet conventional models like YOLO series, while efficient and
accurate, are limited by predefined categories, hindering adaptability in open
scenarios. Recent open-set methods leverage text prompts, visual cues, or
prompt-free paradigm to overcome this, but often compromise between performance
and efficiency due to high computational demands or deployment complexity. In
this work, we introduce YOLOE, which integrates detection and segmentation
across diverse open prompt mechanisms within a single highly efficient model,
achieving real-time seeing anything. For text prompts, we propose
Re-parameterizable Region-Text Alignment (RepRTA) strategy. It refines
pretrained textual embeddings via a re-parameterizable lightweight auxiliary
network and enhances visual-textual alignment with zero inference and
transferring overhead. For visual prompts, we present Semantic-Activated Visual
Prompt Encoder (SAVPE). It employs decoupled semantic and activation branches
to bring improved visual embedding and accuracy with minimal complexity. For
prompt-free scenario, we introduce Lazy Region-Prompt Contrast (LRPC) strategy.
It utilizes a built-in large vocabulary and specialized embedding to identify
all objects, avoiding costly language model dependency. Extensive experiments
show YOLOE's exceptional zero-shot performance and transferability with high
inference efficiency and low training cost. Notably, on LVIS, with 3times
less training cost and 1.4times inference speedup, YOLOE-v8-S surpasses
YOLO-Worldv2-S by 3.5 AP. When transferring to COCO, YOLOE-v8-L achieves 0.6
AP^b and 0.4 AP^m gains over closed-set YOLOv8-L with nearly 4times less
training time. Code and models are available at
https://github.com/THU-MIG/yoloe.
YOLOE combines detection and segmentation using diverse open prompt mechanisms, achieving real-time performance and high efficiency in open-set scenarios.
Re-parameterizable Region-Text Alignment Semantic-Activated Visual Prompt Encoder Lazy Region-Prompt Contrast YOLOE visual-textual alignment visual embedding LVIS COCO zero-shot performance transferability inference efficiency training cost