title: YOLOE: Real-Time Seeing Anything
summary: Object detection and segmentation are widely employed in computer vision
  applications, yet conventional models like YOLO series, while efficient and
  accurate, are limited by predefined categories, hindering adaptability in open
  scenarios. Recent open-set methods leverage text prompts, visual cues, or
  prompt-free paradigm to overcome this, but often compromise between performance
  and efficiency due to high computational demands or deployment complexity. In
  this work, we introduce YOLOE, which integrates detection and segmentation
  across diverse open prompt mechanisms within a single highly efficient model,
  achieving real-time seeing anything. For text prompts, we propose
  Re-parameterizable Region-Text Alignment (RepRTA) strategy. It refines
  pretrained textual embeddings via a re-parameterizable lightweight auxiliary
  network and enhances visual-textual alignment with zero inference and
  transferring overhead. For visual prompts, we present Semantic-Activated Visual
  Prompt Encoder (SAVPE). It employs decoupled semantic and activation branches
  to bring improved visual embedding and accuracy with minimal complexity. For
  prompt-free scenario, we introduce Lazy Region-Prompt Contrast (LRPC) strategy.
  It utilizes a built-in large vocabulary and specialized embedding to identify
  all objects, avoiding costly language model dependency. Extensive experiments
  show YOLOE's exceptional zero-shot performance and transferability with high
  inference efficiency and low training cost. Notably, on LVIS, with 3times
  less training cost and 1.4times inference speedup, YOLOE-v8-S surpasses
  YOLO-Worldv2-S by 3.5 AP. When transferring to COCO, YOLOE-v8-L achieves 0.6
  AP^b and 0.4 AP^m gains over closed-set YOLOv8-L with nearly 4times less
  training time. Code and models are available at
  https://github.com/THU-MIG/yoloe.
paper_authors._id: 67cfaaed7f229132171f596b
paper_authors.name: Ao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-02T16:10:42.022Z
paper_authors._id: 628ece6054698ce61d1e7be3
paper_authors.avatarUrl: /avatars/c6ab33843fd0d8ef003650c1094214c0.svg
paper_authors.fullname: Ao Wang
paper_authors.type: user
paper_authors.user: jameslahm
paper_authors._id: 67cfaaed7f229132171f596c
paper_authors.name: Lihao Liu
paper_authors._id: 67cfaaed7f229132171f596d
paper_authors.name: Hui Chen
paper_authors._id: 67cfaaed7f229132171f596e
paper_authors.name: Zijia Lin
paper_authors._id: 67cfaaed7f229132171f596f
paper_authors.name: Jungong Han
paper_authors._id: 67cfaaed7f229132171f5970
paper_authors.name: Guiguang Ding
paper_title: YOLOE: Real-Time Seeing Anything
paper_summary: Object detection and segmentation are widely employed in computer vision
  applications, yet conventional models like YOLO series, while efficient and
  accurate, are limited by predefined categories, hindering adaptability in open
  scenarios. Recent open-set methods leverage text prompts, visual cues, or
  prompt-free paradigm to overcome this, but often compromise between performance
  and efficiency due to high computational demands or deployment complexity. In
  this work, we introduce YOLOE, which integrates detection and segmentation
  across diverse open prompt mechanisms within a single highly efficient model,
  achieving real-time seeing anything. For text prompts, we propose
  Re-parameterizable Region-Text Alignment (RepRTA) strategy. It refines
  pretrained textual embeddings via a re-parameterizable lightweight auxiliary
  network and enhances visual-textual alignment with zero inference and
  transferring overhead. For visual prompts, we present Semantic-Activated Visual
  Prompt Encoder (SAVPE). It employs decoupled semantic and activation branches
  to bring improved visual embedding and accuracy with minimal complexity. For
  prompt-free scenario, we introduce Lazy Region-Prompt Contrast (LRPC) strategy.
  It utilizes a built-in large vocabulary and specialized embedding to identify
  all objects, avoiding costly language model dependency. Extensive experiments
  show YOLOE's exceptional zero-shot performance and transferability with high
  inference efficiency and low training cost. Notably, on LVIS, with 3times
  less training cost and 1.4times inference speedup, YOLOE-v8-S surpasses
  YOLO-Worldv2-S by 3.5 AP. When transferring to COCO, YOLOE-v8-L achieves 0.6
  AP^b and 0.4 AP^m gains over closed-set YOLOv8-L with nearly 4times less
  training time. Code and models are available at
  https://github.com/THU-MIG/yoloe.
paper_ai_summary: YOLOE combines detection and segmentation using diverse open prompt mechanisms, achieving real-time performance and high efficiency in open-set scenarios.
paper_ai_keywords: Re-parameterizable Region-Text Alignment Semantic-Activated Visual Prompt Encoder Lazy Region-Prompt Contrast YOLOE visual-textual alignment visual embedding LVIS COCO zero-shot performance transferability inference efficiency training cost