OmniManip: Towards General Robotic Manipulation via Object-Centric
  Interaction Primitives as Spatial Constraints
The development of general robotic systems capable of manipulating in
unstructured environments is a significant challenge. While Vision-Language
Models(VLM) excel in high-level commonsense reasoning, they lack the
fine-grained 3D spatial understanding required for precise manipulation tasks.
Fine-tuning VLM on robotic datasets to create Vision-Language-Action
Models(VLA) is a potential solution, but it is hindered by high data collection
costs and generalization issues. To address these challenges, we propose a
novel object-centric representation that bridges the gap between VLM's
high-level reasoning and the low-level precision required for manipulation. Our
key insight is that an object's canonical space, defined by its functional
affordances, provides a structured and semantically meaningful way to describe
interaction primitives, such as points and directions. These primitives act as
a bridge, translating VLM's commonsense reasoning into actionable 3D spatial
constraints. In this context, we introduce a dual closed-loop, open-vocabulary
robotic manipulation system: one loop for high-level planning through primitive
resampling, interaction rendering and VLM checking, and another for low-level
execution via 6D pose tracking. This design ensures robust, real-time control
without requiring VLM fine-tuning. Extensive experiments demonstrate strong
zero-shot generalization across diverse robotic manipulation tasks,
highlighting the potential of this approach for automating large-scale
simulation data generation.
677f66a83e2b5456e79f60bd
Mingjie Pan
claimed_verified
2025-01-09T10:06:23.044Z
655ecd7c56e5ceaf05344b24
/avatars/faad88525197bb6c63be0068f19de418.svg
MingjieP
user
pmj110119
677f66a83e2b5456e79f60be
Jiyao Zhang
admin_assigned
2025-01-13T09:16:41.553Z
6784a033eb390961206f542e
/avatars/d2b5bd89535618a8ca383a3dd2537055.svg
Jiyao Zhang
user
JiyaoZhang
677f66a83e2b5456e79f60bf
Tianshu Wu
677f66a83e2b5456e79f60c0
Yinghao Zhao
admin_assigned
2025-01-13T09:17:05.656Z
675ed5f808a1109506e04a62
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/xCgJ6I7-rATf5J1A5tCJM.png
zhaoyinghao
user
Ergogogogo
677f66a83e2b5456e79f60c1
Wenlong Gao
677f66a83e2b5456e79f60c2
Hao Dong
OmniManip: Towards General Robotic Manipulation via Object-Centric
  Interaction Primitives as Spatial Constraints
The development of general robotic systems capable of manipulating in
unstructured environments is a significant challenge. While Vision-Language
Models(VLM) excel in high-level commonsense reasoning, they lack the
fine-grained 3D spatial understanding required for precise manipulation tasks.
Fine-tuning VLM on robotic datasets to create Vision-Language-Action
Models(VLA) is a potential solution, but it is hindered by high data collection
costs and generalization issues. To address these challenges, we propose a
novel object-centric representation that bridges the gap between VLM's
high-level reasoning and the low-level precision required for manipulation. Our
key insight is that an object's canonical space, defined by its functional
affordances, provides a structured and semantically meaningful way to describe
interaction primitives, such as points and directions. These primitives act as
a bridge, translating VLM's commonsense reasoning into actionable 3D spatial
constraints. In this context, we introduce a dual closed-loop, open-vocabulary
robotic manipulation system: one loop for high-level planning through primitive
resampling, interaction rendering and VLM checking, and another for low-level
execution via 6D pose tracking. This design ensures robust, real-time control
without requiring VLM fine-tuning. Extensive experiments demonstrate strong
zero-shot generalization across diverse robotic manipulation tasks,
highlighting the potential of this approach for automating large-scale
simulation data generation.
A novel object-centric representation bridges the gap between Vision-Language Models' high-level reasoning and the low-level precision needed for robotic manipulation tasks, enabling zero-shot generalization across diverse tasks.
Vision-Language Models Vision-Language-Action Models object-centric representation canonical space functional affordances interaction primitives dual closed-loop open-vocabulary 6D pose tracking zero-shot generalization large-scale simulation data generation