title: Rex-Thinker: Grounded Object Referring via Chain-of-Thought Reasoning
summary: Object referring aims to detect all objects in an image that match a given
  natural language description. We argue that a robust object referring model
  should be grounded, meaning its predictions should be both explainable and
  faithful to the visual content. Specifically, it should satisfy two key
  properties: 1) Verifiable, by producing interpretable reasoning that justifies
  its predictions and clearly links them to visual evidence; and 2) Trustworthy,
  by learning to abstain when no object in the image satisfies the given
  expression. However, most methods treat referring as a direct bounding box
  prediction task, offering limited interpretability and struggling to reject
  expressions with no matching object. In this work, we propose Rex-Thinker, a
  model that formulates object referring as an explicit CoT reasoning task. Given
  a referring expression, we first identify all candidate object instances
  corresponding to the referred object category. Rex-Thinker then performs
  step-by-step reasoning over each candidate to assess whether it matches the
  given expression, before making a final prediction. To support this paradigm,
  we construct a large-scale CoT-style referring dataset named HumanRef-CoT by
  prompting GPT-4o on the HumanRef dataset. Each reasoning trace follows a
  structured planning, action, and summarization format, enabling the model to
  learn decomposed, interpretable reasoning over object candidates. We then train
  Rex-Thinker in two stages: a cold-start supervised fine-tuning phase to teach
  the model how to perform structured reasoning, followed by GRPO-based RL
  learning to improve accuracy and generalization. Experiments show that our
  approach outperforms standard baselines in both precision and interpretability
  on in-domain evaluation, while also demonstrating improved ability to reject
  hallucinated outputs and strong generalization in out-of-domain settings.
paper_authors._id: 6840ff0b535bfb4942b31576
paper_authors.name: Qing Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T13:00:10.958Z
paper_authors._id: 647f46b6838ac3601fc89852
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647f46b6838ac3601fc89852/N5cr1MFEtgKLJ4sVAhS04.jpeg
paper_authors.fullname: Qing Jiang
paper_authors.type: user
paper_authors.user: Mountchicken
paper_authors._id: 6840ff0b535bfb4942b31577
paper_authors.name: Xingyu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T13:00:18.113Z
paper_authors._id: 660cefee15392b34d995e542
paper_authors.avatarUrl: /avatars/ab062a569757ad30ab7d347d737e3998.svg
paper_authors.fullname: Xingyu Chen
paper_authors.type: user
paper_authors.user: AsixaChen
paper_authors._id: 6840ff0b535bfb4942b31578
paper_authors.name: Zhaoyang Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T13:00:25.218Z
paper_authors._id: 669f0abdc13da2bbe268c5c3
paper_authors.avatarUrl: /avatars/87961bfd01c53c420ae13133d643f22c.svg
paper_authors.fullname: Zhaoyang Zeng
paper_authors.type: user
paper_authors.user: zengzhaoyang
paper_authors._id: 6840ff0b535bfb4942b31579
paper_authors.name: Junzhi Yu
paper_authors._id: 6840ff0b535bfb4942b3157a
paper_authors.name: Lei Zhang
paper_title: Rex-Thinker: Grounded Object Referring via Chain-of-Thought Reasoning
paper_summary: Object referring aims to detect all objects in an image that match a given
  natural language description. We argue that a robust object referring model
  should be grounded, meaning its predictions should be both explainable and
  faithful to the visual content. Specifically, it should satisfy two key
  properties: 1) Verifiable, by producing interpretable reasoning that justifies
  its predictions and clearly links them to visual evidence; and 2) Trustworthy,
  by learning to abstain when no object in the image satisfies the given
  expression. However, most methods treat referring as a direct bounding box
  prediction task, offering limited interpretability and struggling to reject
  expressions with no matching object. In this work, we propose Rex-Thinker, a
  model that formulates object referring as an explicit CoT reasoning task. Given
  a referring expression, we first identify all candidate object instances
  corresponding to the referred object category. Rex-Thinker then performs
  step-by-step reasoning over each candidate to assess whether it matches the
  given expression, before making a final prediction. To support this paradigm,
  we construct a large-scale CoT-style referring dataset named HumanRef-CoT by
  prompting GPT-4o on the HumanRef dataset. Each reasoning trace follows a
  structured planning, action, and summarization format, enabling the model to
  learn decomposed, interpretable reasoning over object candidates. We then train
  Rex-Thinker in two stages: a cold-start supervised fine-tuning phase to teach
  the model how to perform structured reasoning, followed by GRPO-based RL
  learning to improve accuracy and generalization. Experiments show that our
  approach outperforms standard baselines in both precision and interpretability
  on in-domain evaluation, while also demonstrating improved ability to reject
  hallucinated outputs and strong generalization in out-of-domain settings.
paper_ai_summary: Rex-Thinker is a CoT-based model that enhances object referring by performing step-by-step reasoning over candidate objects, leading to improved interpretability and rejection of mismatched queries.
paper_ai_keywords: CoT reasoning HumanRef-CoT GPT-4o structured reasoning cold-start supervised fine-tuning GRPO-based RL learning