title: SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and
    Object Manipulation
summary: Spatial intelligence is a critical component of embodied AI, promoting robots
  to understand and interact with their environments. While recent advances have
  enhanced the ability of VLMs to perceive object locations and positional
  relationships, they still lack the capability to precisely understand object
  orientations-a key requirement for tasks involving fine-grained manipulations.
  Addressing this limitation not only requires geometric reasoning but also an
  expressive and intuitive way to represent orientation. In this context, we
  propose that natural language offers a more flexible representation space than
  canonical frames, making it particularly suitable for instruction-following
  robotic systems. In this paper, we introduce the concept of semantic
  orientation, which defines object orientations using natural language in a
  reference-frame-free manner (e.g., the ''plug-in'' direction of a USB or the
  ''handle'' direction of a knife). To support this, we construct OrienText300K,
  a large-scale dataset of 3D models annotated with semantic orientations that
  link geometric understanding to functional semantics. By integrating semantic
  orientation into a VLM system, we enable robots to generate manipulation
  actions with both positional and orientational constraints. Extensive
  experiments in simulation and real world demonstrate that our approach
  significantly enhances robotic manipulation capabilities, e.g., 48.7% accuracy
  on Open6DOR and 74.9% accuracy on SIMPLER.
paper_authors._id: 67b546c0d8a1eac02c605f6a
paper_authors.name: Zekun Qi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-19T09:01:21.001Z
paper_authors._id: 63c3e8abc7d7f4c63a515a02
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63c3e8abc7d7f4c63a515a02/npMHnVP2hHLbvoUGe7C4O.jpeg
paper_authors.fullname: Zekun Qi
paper_authors.type: user
paper_authors.user: qizekun
paper_authors._id: 67b546c0d8a1eac02c605f6b
paper_authors.name: Wenyao Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T10:08:31.789Z
paper_authors._id: 65f9533b136fb8ddbd14e1fa
paper_authors.avatarUrl: /avatars/d88f75da0448093ccd1babba2a37d73f.svg
paper_authors.fullname: Zhang
paper_authors.type: user
paper_authors.user: WenyaoZhang
paper_authors._id: 67b546c0d8a1eac02c605f6c
paper_authors.name: Yufei Ding
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T10:08:57.294Z
paper_authors._id: 66bde456198f9d79f2be2d17
paper_authors.avatarUrl: /avatars/8c349aecb8a3a7cd7ef9d69e94eca8bd.svg
paper_authors.fullname: Yufei Ding
paper_authors.type: user
paper_authors.user: YufeiD
paper_authors._id: 67b546c0d8a1eac02c605f6d
paper_authors.name: Runpei Dong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-19T09:01:18.622Z
paper_authors._id: 6201fc5d91d53938a6432fbf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6201fc5d91d53938a6432fbf/VLs8ZYaZrop4KBpZn53fH.jpeg
paper_authors.fullname: Runpei Dong
paper_authors.type: user
paper_authors.user: RunpeiDong
paper_authors._id: 67b546c0d8a1eac02c605f6e
paper_authors.name: Xinqiang Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-20T09:37:27.869Z
paper_authors._id: 675296fdea02ef7b84609893
paper_authors.avatarUrl: /avatars/27b3886ca048f108ac26a942f151410b.svg
paper_authors.fullname: Yu
paper_authors.type: user
paper_authors.user: XinXinQiang
paper_authors._id: 67b546c0d8a1eac02c605f6f
paper_authors.name: Jingwen Li
paper_authors._id: 67b546c0d8a1eac02c605f70
paper_authors.name: Lingyun Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T10:09:41.570Z
paper_authors._id: 6745c4a80739b09408862ac9
paper_authors.avatarUrl: /avatars/8bc082fcbbf933b150a252c78d1bb3be.svg
paper_authors.fullname: lingyun xu
paper_authors.type: user
paper_authors.user: codered010
paper_authors._id: 67b546c0d8a1eac02c605f71
paper_authors.name: Baoyu Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T10:09:48.223Z
paper_authors._id: 67302fa362930cbc461511a8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/L9x68oxL8Ot88pyeU6w7Z.png
paper_authors.fullname: Baoyu Li
paper_authors.type: user
paper_authors.user: boeyyyy
paper_authors._id: 67b546c0d8a1eac02c605f72
paper_authors.name: Xialin He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T10:10:04.207Z
paper_authors._id: 67b42039cca7aff798979e80
paper_authors.avatarUrl: /avatars/d410b3617395cd7e2a9c0c89ff12f23d.svg
paper_authors.fullname: Xialin He
paper_authors.type: user
paper_authors.user: XialinHe
paper_authors._id: 67b546c0d8a1eac02c605f73
paper_authors.name: Guofan Fan
paper_authors._id: 67b546c0d8a1eac02c605f74
paper_authors.name: Jiazhao Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T10:11:41.498Z
paper_authors._id: 65658233d35fc55406e8b00d
paper_authors.avatarUrl: /avatars/660eaa1923cd3e3478cec8197936a75c.svg
paper_authors.fullname: Jiazhao Zhang
paper_authors.type: user
paper_authors.user: Jzzhang
paper_authors._id: 67b546c0d8a1eac02c605f75
paper_authors.name: Jiawei He
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-19T14:37:41.045Z
paper_authors._id: 649cd8deccfa6c1a3c4d05ec
paper_authors.avatarUrl: /avatars/2b8f72a0643dfd74bc08fba5ed98ce95.svg
paper_authors.fullname: Jiawei
paper_authors.type: user
paper_authors.user: jiaweihe
paper_authors._id: 67b546c0d8a1eac02c605f76
paper_authors.name: Jiayuan Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T10:11:49.058Z
paper_authors._id: 638aa283cebef0d13aa2ec2e
paper_authors.avatarUrl: /avatars/3b67e6a6073033864a817230e97c27ca.svg
paper_authors.fullname: Jiayuan Gu
paper_authors.type: user
paper_authors.user: jigu
paper_authors._id: 67b546c0d8a1eac02c605f77
paper_authors.name: Xin Jin
paper_authors._id: 67b546c0d8a1eac02c605f78
paper_authors.name: Kaisheng Ma
paper_authors._id: 67b546c0d8a1eac02c605f79
paper_authors.name: Zhizheng Zhang
paper_authors._id: 67b546c0d8a1eac02c605f7a
paper_authors.name: He Wang
paper_authors._id: 67b546c0d8a1eac02c605f7b
paper_authors.name: Li Yi
paper_title: SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and
    Object Manipulation
paper_summary: Spatial intelligence is a critical component of embodied AI, promoting robots
  to understand and interact with their environments. While recent advances have
  enhanced the ability of VLMs to perceive object locations and positional
  relationships, they still lack the capability to precisely understand object
  orientations-a key requirement for tasks involving fine-grained manipulations.
  Addressing this limitation not only requires geometric reasoning but also an
  expressive and intuitive way to represent orientation. In this context, we
  propose that natural language offers a more flexible representation space than
  canonical frames, making it particularly suitable for instruction-following
  robotic systems. In this paper, we introduce the concept of semantic
  orientation, which defines object orientations using natural language in a
  reference-frame-free manner (e.g., the ''plug-in'' direction of a USB or the
  ''handle'' direction of a knife). To support this, we construct OrienText300K,
  a large-scale dataset of 3D models annotated with semantic orientations that
  link geometric understanding to functional semantics. By integrating semantic
  orientation into a VLM system, we enable robots to generate manipulation
  actions with both positional and orientational constraints. Extensive
  experiments in simulation and real world demonstrate that our approach
  significantly enhances robotic manipulation capabilities, e.g., 48.7% accuracy
  on Open6DOR and 74.9% accuracy on SIMPLER.
paper_ai_keywords: Spatial intelligence Embodied AI VLMs (Vision-Language Models) Percept object locations Positional relationships Object orientations Geometric reasoning Natural language Reference-frame-free representation Semantic orientations OrienText300K 3D models Functional semantics Manipulation actions Positional constraints Open6DOR SIMPLER