title: Referring to Any Person
summary: Humans are undoubtedly the most important participants in computer vision,
  and the ability to detect any individual given a natural language description,
  a task we define as referring to any person, holds substantial practical value.
  However, we find that existing models generally fail to achieve real-world
  usability, and current benchmarks are limited by their focus on one-to-one
  referring, that hinder progress in this area. In this work, we revisit this
  task from three critical perspectives: task definition, dataset design, and
  model architecture. We first identify five aspects of referable entities and
  three distinctive characteristics of this task. Next, we introduce HumanRef, a
  novel dataset designed to tackle these challenges and better reflect real-world
  applications. From a model design perspective, we integrate a multimodal large
  language model with an object detection framework, constructing a robust
  referring model named RexSeek. Experimental results reveal that
  state-of-the-art models, which perform well on commonly used benchmarks like
  RefCOCO/+/g, struggle with HumanRef due to their inability to detect multiple
  individuals. In contrast, RexSeek not only excels in human referring but also
  generalizes effectively to common object referring, making it broadly
  applicable across various perception tasks. Code is available at
  https://github.com/IDEA-Research/RexSeek
paper_authors._id: 67d15293e3afecf451aceab7
paper_authors.name: Qing Jiang
paper_authors._id: 67d15293e3afecf451aceab8
paper_authors.name: Lin Wu
paper_authors._id: 67d15293e3afecf451aceab9
paper_authors.name: Zhaoyang Zeng
paper_authors._id: 67d15293e3afecf451aceaba
paper_authors.name: Tianhe Ren
paper_authors._id: 67d15293e3afecf451aceabb
paper_authors.name: Yuda Xiong
paper_authors._id: 67d15293e3afecf451aceabc
paper_authors.name: Yihao Chen
paper_authors._id: 67d15293e3afecf451aceabd
paper_authors.name: Qin Liu
paper_authors._id: 67d15293e3afecf451aceabe
paper_authors.name: Lei Zhang
paper_title: Referring to Any Person
paper_summary: Humans are undoubtedly the most important participants in computer vision,
  and the ability to detect any individual given a natural language description,
  a task we define as referring to any person, holds substantial practical value.
  However, we find that existing models generally fail to achieve real-world
  usability, and current benchmarks are limited by their focus on one-to-one
  referring, that hinder progress in this area. In this work, we revisit this
  task from three critical perspectives: task definition, dataset design, and
  model architecture. We first identify five aspects of referable entities and
  three distinctive characteristics of this task. Next, we introduce HumanRef, a
  novel dataset designed to tackle these challenges and better reflect real-world
  applications. From a model design perspective, we integrate a multimodal large
  language model with an object detection framework, constructing a robust
  referring model named RexSeek. Experimental results reveal that
  state-of-the-art models, which perform well on commonly used benchmarks like
  RefCOCO/+/g, struggle with HumanRef due to their inability to detect multiple
  individuals. In contrast, RexSeek not only excels in human referring but also
  generalizes effectively to common object referring, making it broadly
  applicable across various perception tasks. Code is available at
  https://github.com/IDEA-Research/RexSeek
paper_ai_keywords: referable entities multimodal large language model object detection framework HumanRef RexSeek RefCOCO/+/g