Rex-Thinker: Grounded Object Referring via Chain-of-Thought Reasoning
Object referring aims to detect all objects in an image that match a given
natural language description. We argue that a robust object referring model
should be grounded, meaning its predictions should be both explainable and
faithful to the visual content. Specifically, it should satisfy two key
properties: 1) Verifiable, by producing interpretable reasoning that justifies
its predictions and clearly links them to visual evidence; and 2) Trustworthy,
by learning to abstain when no object in the image satisfies the given
expression. However, most methods treat referring as a direct bounding box
prediction task, offering limited interpretability and struggling to reject
expressions with no matching object. In this work, we propose Rex-Thinker, a
model that formulates object referring as an explicit CoT reasoning task. Given
a referring expression, we first identify all candidate object instances
corresponding to the referred object category. Rex-Thinker then performs
step-by-step reasoning over each candidate to assess whether it matches the
given expression, before making a final prediction. To support this paradigm,
we construct a large-scale CoT-style referring dataset named HumanRef-CoT by
prompting GPT-4o on the HumanRef dataset. Each reasoning trace follows a
structured planning, action, and summarization format, enabling the model to
learn decomposed, interpretable reasoning over object candidates. We then train
Rex-Thinker in two stages: a cold-start supervised fine-tuning phase to teach
the model how to perform structured reasoning, followed by GRPO-based RL
learning to improve accuracy and generalization. Experiments show that our
approach outperforms standard baselines in both precision and interpretability
on in-domain evaluation, while also demonstrating improved ability to reject
hallucinated outputs and strong generalization in out-of-domain settings.
6840ff0b535bfb4942b31576
Qing Jiang
admin_assigned
2025-06-05T13:00:10.958Z
647f46b6838ac3601fc89852
https://cdn-avatars.huggingface.co/v1/production/uploads/647f46b6838ac3601fc89852/N5cr1MFEtgKLJ4sVAhS04.jpeg
Qing Jiang
user
Mountchicken
6840ff0b535bfb4942b31577
Xingyu Chen
admin_assigned
2025-06-05T13:00:18.113Z
660cefee15392b34d995e542
/avatars/ab062a569757ad30ab7d347d737e3998.svg
Xingyu Chen
user
AsixaChen
6840ff0b535bfb4942b31578
Zhaoyang Zeng
admin_assigned
2025-06-05T13:00:25.218Z
669f0abdc13da2bbe268c5c3
/avatars/87961bfd01c53c420ae13133d643f22c.svg
Zhaoyang Zeng
user
zengzhaoyang
6840ff0b535bfb4942b31579
Junzhi Yu
6840ff0b535bfb4942b3157a
Lei Zhang
Rex-Thinker: Grounded Object Referring via Chain-of-Thought Reasoning
Object referring aims to detect all objects in an image that match a given
natural language description. We argue that a robust object referring model
should be grounded, meaning its predictions should be both explainable and
faithful to the visual content. Specifically, it should satisfy two key
properties: 1) Verifiable, by producing interpretable reasoning that justifies
its predictions and clearly links them to visual evidence; and 2) Trustworthy,
by learning to abstain when no object in the image satisfies the given
expression. However, most methods treat referring as a direct bounding box
prediction task, offering limited interpretability and struggling to reject
expressions with no matching object. In this work, we propose Rex-Thinker, a
model that formulates object referring as an explicit CoT reasoning task. Given
a referring expression, we first identify all candidate object instances
corresponding to the referred object category. Rex-Thinker then performs
step-by-step reasoning over each candidate to assess whether it matches the
given expression, before making a final prediction. To support this paradigm,
we construct a large-scale CoT-style referring dataset named HumanRef-CoT by
prompting GPT-4o on the HumanRef dataset. Each reasoning trace follows a
structured planning, action, and summarization format, enabling the model to
learn decomposed, interpretable reasoning over object candidates. We then train
Rex-Thinker in two stages: a cold-start supervised fine-tuning phase to teach
the model how to perform structured reasoning, followed by GRPO-based RL
learning to improve accuracy and generalization. Experiments show that our
approach outperforms standard baselines in both precision and interpretability
on in-domain evaluation, while also demonstrating improved ability to reject
hallucinated outputs and strong generalization in out-of-domain settings.
Rex-Thinker is a CoT-based model that enhances object referring by performing step-by-step reasoning over candidate objects, leading to improved interpretability and rejection of mismatched queries.
CoT reasoning HumanRef-CoT GPT-4o structured reasoning cold-start supervised fine-tuning GRPO-based RL learning