title: RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language
    Models for Robotics
summary: Spatial referring is a fundamental capability of embodied robots to interact
  with the 3D physical world. However, even with the powerful pretrained vision
  language models (VLMs), recent approaches are still not qualified to accurately
  understand the complex 3D scenes and dynamically reason about the
  instruction-indicated locations for interaction. To this end, we propose
  RoboRefer, a 3D-aware VLM that can first achieve precise spatial understanding
  by integrating a disentangled but dedicated depth encoder via supervised
  fine-tuning (SFT). Moreover, RoboRefer advances generalized multi-step spatial
  reasoning via reinforcement fine-tuning (RFT), with metric-sensitive process
  reward functions tailored for spatial referring tasks. To support SFT and RFT
  training, we introduce RefSpatial, a large-scale dataset of 20M QA pairs (2x
  prior), covering 31 spatial relations (vs. 15 prior) and supporting complex
  reasoning processes (up to 5 steps). In addition, we introduce
  RefSpatial-Bench, a challenging benchmark filling the gap in evaluating spatial
  referring with multi-step reasoning. Experiments show that SFT-trained
  RoboRefer achieves state-of-the-art spatial understanding, with an average
  success rate of 89.6%. RFT-trained RoboRefer further outperforms all other
  baselines by a large margin, even surpassing Gemini-2.5-Pro by 17.4% in average
  accuracy on RefSpatial-Bench. Notably, RoboRefer can be integrated with various
  control policies to execute long-horizon, dynamic tasks across diverse robots
  (e,g., UR5, G1 humanoid) in cluttered real-world scenes.
paper_authors._id: 68424dc48d0422fce0273e99
paper_authors.name: Enshen Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-06T07:41:21.339Z
paper_authors._id: 63f08dc79cf89c9ed1bb89cd
paper_authors.avatarUrl: /avatars/37290358ad00bbd752f519cfdec02f3e.svg
paper_authors.fullname: Zhoues
paper_authors.type: user
paper_authors.user: Zhoues
paper_authors._id: 68424dc48d0422fce0273e9a
paper_authors.name: Jingkun An
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-07T05:48:57.586Z
paper_authors._id: 654d084c761c998a5b69a877
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/654d084c761c998a5b69a877/j4CR4lA5QXII-Aob4oaaj.png
paper_authors.fullname: Jingkun
paper_authors.type: user
paper_authors.user: JingkunAn
paper_authors._id: 68424dc48d0422fce0273e9b
paper_authors.name: Cheng Chi
paper_authors._id: 68424dc48d0422fce0273e9c
paper_authors.name: Yi Han
paper_authors._id: 68424dc48d0422fce0273e9d
paper_authors.name: Shanyu Rong
paper_authors._id: 68424dc48d0422fce0273e9e
paper_authors.name: Chi Zhang
paper_authors._id: 68424dc48d0422fce0273e9f
paper_authors.name: Pengwei Wang
paper_authors._id: 68424dc48d0422fce0273ea0
paper_authors.name: Zhongyuan Wang
paper_authors._id: 68424dc48d0422fce0273ea1
paper_authors.name: Tiejun Huang
paper_authors._id: 68424dc48d0422fce0273ea2
paper_authors.name: Lu Sheng
paper_authors._id: 68424dc48d0422fce0273ea3
paper_authors.name: Shanghang Zhang
paper_title: RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language
    Models for Robotics
paper_summary: Spatial referring is a fundamental capability of embodied robots to interact
  with the 3D physical world. However, even with the powerful pretrained vision
  language models (VLMs), recent approaches are still not qualified to accurately
  understand the complex 3D scenes and dynamically reason about the
  instruction-indicated locations for interaction. To this end, we propose
  RoboRefer, a 3D-aware VLM that can first achieve precise spatial understanding
  by integrating a disentangled but dedicated depth encoder via supervised
  fine-tuning (SFT). Moreover, RoboRefer advances generalized multi-step spatial
  reasoning via reinforcement fine-tuning (RFT), with metric-sensitive process
  reward functions tailored for spatial referring tasks. To support SFT and RFT
  training, we introduce RefSpatial, a large-scale dataset of 20M QA pairs (2x
  prior), covering 31 spatial relations (vs. 15 prior) and supporting complex
  reasoning processes (up to 5 steps). In addition, we introduce
  RefSpatial-Bench, a challenging benchmark filling the gap in evaluating spatial
  referring with multi-step reasoning. Experiments show that SFT-trained
  RoboRefer achieves state-of-the-art spatial understanding, with an average
  success rate of 89.6%. RFT-trained RoboRefer further outperforms all other
  baselines by a large margin, even surpassing Gemini-2.5-Pro by 17.4% in average
  accuracy on RefSpatial-Bench. Notably, RoboRefer can be integrated with various
  control policies to execute long-horizon, dynamic tasks across diverse robots
  (e,g., UR5, G1 humanoid) in cluttered real-world scenes.
paper_ai_summary: RoboRefer, a 3D-aware vision language model, enhances spatial understanding and multi-step reasoning in embodied robots through supervised and reinforcement fine-tuning, using the RefSpatial dataset and RefSpatial-Bench benchmark.
paper_ai_keywords: 3D-aware VLM disentangled depth encoder supervised fine-tuning (SFT) reinforcement fine-tuning (RFT) metric-sensitive reward functions RefSpatial RefSpatial-Bench spatial referring tasks multi-step reasoning state-of-the-art spatial understanding long-horizon dynamic tasks