SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and
  Object Manipulation
Spatial intelligence is a critical component of embodied AI, promoting robots
to understand and interact with their environments. While recent advances have
enhanced the ability of VLMs to perceive object locations and positional
relationships, they still lack the capability to precisely understand object
orientations-a key requirement for tasks involving fine-grained manipulations.
Addressing this limitation not only requires geometric reasoning but also an
expressive and intuitive way to represent orientation. In this context, we
propose that natural language offers a more flexible representation space than
canonical frames, making it particularly suitable for instruction-following
robotic systems. In this paper, we introduce the concept of semantic
orientation, which defines object orientations using natural language in a
reference-frame-free manner (e.g., the ''plug-in'' direction of a USB or the
''handle'' direction of a knife). To support this, we construct OrienText300K,
a large-scale dataset of 3D models annotated with semantic orientations that
link geometric understanding to functional semantics. By integrating semantic
orientation into a VLM system, we enable robots to generate manipulation
actions with both positional and orientational constraints. Extensive
experiments in simulation and real world demonstrate that our approach
significantly enhances robotic manipulation capabilities, e.g., 48.7% accuracy
on Open6DOR and 74.9% accuracy on SIMPLER.
67b546c0d8a1eac02c605f6a
Zekun Qi
claimed_verified
2025-02-19T09:01:21.001Z
63c3e8abc7d7f4c63a515a02
https://cdn-avatars.huggingface.co/v1/production/uploads/63c3e8abc7d7f4c63a515a02/npMHnVP2hHLbvoUGe7C4O.jpeg
Zekun Qi
user
qizekun
67b546c0d8a1eac02c605f6b
Wenyao Zhang
admin_assigned
2025-02-19T10:08:31.789Z
65f9533b136fb8ddbd14e1fa
/avatars/d88f75da0448093ccd1babba2a37d73f.svg
Zhang
user
WenyaoZhang
67b546c0d8a1eac02c605f6c
Yufei Ding
admin_assigned
2025-02-19T10:08:57.294Z
66bde456198f9d79f2be2d17
/avatars/8c349aecb8a3a7cd7ef9d69e94eca8bd.svg
Yufei Ding
user
YufeiD
67b546c0d8a1eac02c605f6d
Runpei Dong
claimed_verified
2025-02-19T09:01:18.622Z
6201fc5d91d53938a6432fbf
https://cdn-avatars.huggingface.co/v1/production/uploads/6201fc5d91d53938a6432fbf/VLs8ZYaZrop4KBpZn53fH.jpeg
Runpei Dong
user
RunpeiDong
67b546c0d8a1eac02c605f6e
Xinqiang Yu
claimed_verified
2025-02-20T09:37:27.869Z
675296fdea02ef7b84609893
/avatars/27b3886ca048f108ac26a942f151410b.svg
Yu
user
XinXinQiang
67b546c0d8a1eac02c605f6f
Jingwen Li
67b546c0d8a1eac02c605f70
Lingyun Xu
admin_assigned
2025-02-19T10:09:41.570Z
6745c4a80739b09408862ac9
/avatars/8bc082fcbbf933b150a252c78d1bb3be.svg
lingyun xu
user
codered010
67b546c0d8a1eac02c605f71
Baoyu Li
admin_assigned
2025-02-19T10:09:48.223Z
67302fa362930cbc461511a8
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/L9x68oxL8Ot88pyeU6w7Z.png
Baoyu Li
user
boeyyyy
67b546c0d8a1eac02c605f72
Xialin He
admin_assigned
2025-02-19T10:10:04.207Z
67b42039cca7aff798979e80
/avatars/d410b3617395cd7e2a9c0c89ff12f23d.svg
Xialin He
user
XialinHe
67b546c0d8a1eac02c605f73
Guofan Fan
67b546c0d8a1eac02c605f74
Jiazhao Zhang
admin_assigned
2025-02-19T10:11:41.498Z
65658233d35fc55406e8b00d
/avatars/660eaa1923cd3e3478cec8197936a75c.svg
Jiazhao Zhang
user
Jzzhang
67b546c0d8a1eac02c605f75
Jiawei He
claimed_verified
2025-02-19T14:37:41.045Z
649cd8deccfa6c1a3c4d05ec
/avatars/2b8f72a0643dfd74bc08fba5ed98ce95.svg
Jiawei
user
jiaweihe
67b546c0d8a1eac02c605f76
Jiayuan Gu
admin_assigned
2025-02-19T10:11:49.058Z
638aa283cebef0d13aa2ec2e
/avatars/3b67e6a6073033864a817230e97c27ca.svg
Jiayuan Gu
user
jigu
67b546c0d8a1eac02c605f77
Xin Jin
67b546c0d8a1eac02c605f78
Kaisheng Ma
67b546c0d8a1eac02c605f79
Zhizheng Zhang
67b546c0d8a1eac02c605f7a
He Wang
67b546c0d8a1eac02c605f7b
Li Yi
SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and
  Object Manipulation
Spatial intelligence is a critical component of embodied AI, promoting robots
to understand and interact with their environments. While recent advances have
enhanced the ability of VLMs to perceive object locations and positional
relationships, they still lack the capability to precisely understand object
orientations-a key requirement for tasks involving fine-grained manipulations.
Addressing this limitation not only requires geometric reasoning but also an
expressive and intuitive way to represent orientation. In this context, we
propose that natural language offers a more flexible representation space than
canonical frames, making it particularly suitable for instruction-following
robotic systems. In this paper, we introduce the concept of semantic
orientation, which defines object orientations using natural language in a
reference-frame-free manner (e.g., the ''plug-in'' direction of a USB or the
''handle'' direction of a knife). To support this, we construct OrienText300K,
a large-scale dataset of 3D models annotated with semantic orientations that
link geometric understanding to functional semantics. By integrating semantic
orientation into a VLM system, we enable robots to generate manipulation
actions with both positional and orientational constraints. Extensive
experiments in simulation and real world demonstrate that our approach
significantly enhances robotic manipulation capabilities, e.g., 48.7% accuracy
on Open6DOR and 74.9% accuracy on SIMPLER.
Incorporating semantic orientation described in natural language enhances robotic manipulation capabilities, improving accuracy in tasks involving object orientations.
spatial intelligence embodied AI VLMs object orientations geometric reasoning semantic orientation OrienText300K 3D models functional semantics robotic manipulation Open6DOR SIMPLER