RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language
  Models for Robotics
Spatial referring is a fundamental capability of embodied robots to interact
with the 3D physical world. However, even with the powerful pretrained vision
language models (VLMs), recent approaches are still not qualified to accurately
understand the complex 3D scenes and dynamically reason about the
instruction-indicated locations for interaction. To this end, we propose
RoboRefer, a 3D-aware VLM that can first achieve precise spatial understanding
by integrating a disentangled but dedicated depth encoder via supervised
fine-tuning (SFT). Moreover, RoboRefer advances generalized multi-step spatial
reasoning via reinforcement fine-tuning (RFT), with metric-sensitive process
reward functions tailored for spatial referring tasks. To support SFT and RFT
training, we introduce RefSpatial, a large-scale dataset of 20M QA pairs (2x
prior), covering 31 spatial relations (vs. 15 prior) and supporting complex
reasoning processes (up to 5 steps). In addition, we introduce
RefSpatial-Bench, a challenging benchmark filling the gap in evaluating spatial
referring with multi-step reasoning. Experiments show that SFT-trained
RoboRefer achieves state-of-the-art spatial understanding, with an average
success rate of 89.6%. RFT-trained RoboRefer further outperforms all other
baselines by a large margin, even surpassing Gemini-2.5-Pro by 17.4% in average
accuracy on RefSpatial-Bench. Notably, RoboRefer can be integrated with various
control policies to execute long-horizon, dynamic tasks across diverse robots
(e,g., UR5, G1 humanoid) in cluttered real-world scenes.
68424dc48d0422fce0273e99
Enshen Zhou
claimed_verified
2025-06-06T07:41:21.339Z
63f08dc79cf89c9ed1bb89cd
/avatars/37290358ad00bbd752f519cfdec02f3e.svg
Zhoues
user
Zhoues
68424dc48d0422fce0273e9a
Jingkun An
claimed_verified
2025-06-07T05:48:57.586Z
654d084c761c998a5b69a877
https://cdn-avatars.huggingface.co/v1/production/uploads/654d084c761c998a5b69a877/j4CR4lA5QXII-Aob4oaaj.png
Jingkun
user
JingkunAn
68424dc48d0422fce0273e9b
Cheng Chi
claimed_verified
2025-07-05T07:54:16.024Z
650bf938677f9e45963d672e
/avatars/7d4159067b5005a3a635e36b26b7b239.svg
Cheng Chi
user
ChuckChi
68424dc48d0422fce0273e9c
Yi Han
claimed_verified
2025-07-08T12:40:44.374Z
64660e07326128fd2c696c2e
https://cdn-avatars.huggingface.co/v1/production/uploads/64660e07326128fd2c696c2e/EDOTh9c5i7NUzZ_BfcsKA.png
Han.Yi
user
hany01rye
68424dc48d0422fce0273e9d
Shanyu Rong
68424dc48d0422fce0273e9e
Chi Zhang
68424dc48d0422fce0273e9f
Pengwei Wang
68424dc48d0422fce0273ea0
Zhongyuan Wang
68424dc48d0422fce0273ea1
Tiejun Huang
68424dc48d0422fce0273ea2
Lu Sheng
claimed_verified
2025-06-25T09:19:34.907Z
65b722dbe02a17f0f8d1cc6b
/avatars/65f20601ef9b8ebfdddadd737f9153d6.svg
Lu Sheng
user
lsheng2024
68424dc48d0422fce0273ea3
Shanghang Zhang
RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language
  Models for Robotics
Spatial referring is a fundamental capability of embodied robots to interact
with the 3D physical world. However, even with the powerful pretrained vision
language models (VLMs), recent approaches are still not qualified to accurately
understand the complex 3D scenes and dynamically reason about the
instruction-indicated locations for interaction. To this end, we propose
RoboRefer, a 3D-aware VLM that can first achieve precise spatial understanding
by integrating a disentangled but dedicated depth encoder via supervised
fine-tuning (SFT). Moreover, RoboRefer advances generalized multi-step spatial
reasoning via reinforcement fine-tuning (RFT), with metric-sensitive process
reward functions tailored for spatial referring tasks. To support SFT and RFT
training, we introduce RefSpatial, a large-scale dataset of 20M QA pairs (2x
prior), covering 31 spatial relations (vs. 15 prior) and supporting complex
reasoning processes (up to 5 steps). In addition, we introduce
RefSpatial-Bench, a challenging benchmark filling the gap in evaluating spatial
referring with multi-step reasoning. Experiments show that SFT-trained
RoboRefer achieves state-of-the-art spatial understanding, with an average
success rate of 89.6%. RFT-trained RoboRefer further outperforms all other
baselines by a large margin, even surpassing Gemini-2.5-Pro by 17.4% in average
accuracy on RefSpatial-Bench. Notably, RoboRefer can be integrated with various
control policies to execute long-horizon, dynamic tasks across diverse robots
(e,g., UR5, G1 humanoid) in cluttered real-world scenes.
RoboRefer, a 3D-aware vision language model, enhances spatial understanding and multi-step reasoning in embodied robots through supervised and reinforcement fine-tuning, using the RefSpatial dataset and RefSpatial-Bench benchmark.
3D-aware VLM disentangled depth encoder supervised fine-tuning (SFT) reinforcement fine-tuning (RFT) metric-sensitive reward functions RefSpatial RefSpatial-Bench spatial referring tasks multi-step reasoning state-of-the-art spatial understanding long-horizon dynamic tasks