title: OmniManip: Towards General Robotic Manipulation via Object-Centric
    Interaction Primitives as Spatial Constraints
summary: The development of general robotic systems capable of manipulating in
  unstructured environments is a significant challenge. While Vision-Language
  Models(VLM) excel in high-level commonsense reasoning, they lack the
  fine-grained 3D spatial understanding required for precise manipulation tasks.
  Fine-tuning VLM on robotic datasets to create Vision-Language-Action
  Models(VLA) is a potential solution, but it is hindered by high data collection
  costs and generalization issues. To address these challenges, we propose a
  novel object-centric representation that bridges the gap between VLM's
  high-level reasoning and the low-level precision required for manipulation. Our
  key insight is that an object's canonical space, defined by its functional
  affordances, provides a structured and semantically meaningful way to describe
  interaction primitives, such as points and directions. These primitives act as
  a bridge, translating VLM's commonsense reasoning into actionable 3D spatial
  constraints. In this context, we introduce a dual closed-loop, open-vocabulary
  robotic manipulation system: one loop for high-level planning through primitive
  resampling, interaction rendering and VLM checking, and another for low-level
  execution via 6D pose tracking. This design ensures robust, real-time control
  without requiring VLM fine-tuning. Extensive experiments demonstrate strong
  zero-shot generalization across diverse robotic manipulation tasks,
  highlighting the potential of this approach for automating large-scale
  simulation data generation.
paper_authors._id: 677f66a83e2b5456e79f60bd
paper_authors.name: Mingjie Pan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-09T10:06:23.044Z
paper_authors._id: 655ecd7c56e5ceaf05344b24
paper_authors.avatarUrl: /avatars/faad88525197bb6c63be0068f19de418.svg
paper_authors.fullname: MingjieP
paper_authors.type: user
paper_authors.user: pmj110119
paper_authors._id: 677f66a83e2b5456e79f60be
paper_authors.name: Jiyao Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-13T09:16:41.553Z
paper_authors._id: 6784a033eb390961206f542e
paper_authors.avatarUrl: /avatars/d2b5bd89535618a8ca383a3dd2537055.svg
paper_authors.fullname: Jiyao Zhang
paper_authors.type: user
paper_authors.user: JiyaoZhang
paper_authors._id: 677f66a83e2b5456e79f60bf
paper_authors.name: Tianshu Wu
paper_authors._id: 677f66a83e2b5456e79f60c0
paper_authors.name: Yinghao Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-13T09:17:05.656Z
paper_authors._id: 675ed5f808a1109506e04a62
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/xCgJ6I7-rATf5J1A5tCJM.png
paper_authors.fullname: zhaoyinghao
paper_authors.type: user
paper_authors.user: Ergogogogo
paper_authors._id: 677f66a83e2b5456e79f60c1
paper_authors.name: Wenlong Gao
paper_authors._id: 677f66a83e2b5456e79f60c2
paper_authors.name: Hao Dong
paper_title: OmniManip: Towards General Robotic Manipulation via Object-Centric
    Interaction Primitives as Spatial Constraints
paper_summary: The development of general robotic systems capable of manipulating in
  unstructured environments is a significant challenge. While Vision-Language
  Models(VLM) excel in high-level commonsense reasoning, they lack the
  fine-grained 3D spatial understanding required for precise manipulation tasks.
  Fine-tuning VLM on robotic datasets to create Vision-Language-Action
  Models(VLA) is a potential solution, but it is hindered by high data collection
  costs and generalization issues. To address these challenges, we propose a
  novel object-centric representation that bridges the gap between VLM's
  high-level reasoning and the low-level precision required for manipulation. Our
  key insight is that an object's canonical space, defined by its functional
  affordances, provides a structured and semantically meaningful way to describe
  interaction primitives, such as points and directions. These primitives act as
  a bridge, translating VLM's commonsense reasoning into actionable 3D spatial
  constraints. In this context, we introduce a dual closed-loop, open-vocabulary
  robotic manipulation system: one loop for high-level planning through primitive
  resampling, interaction rendering and VLM checking, and another for low-level
  execution via 6D pose tracking. This design ensures robust, real-time control
  without requiring VLM fine-tuning. Extensive experiments demonstrate strong
  zero-shot generalization across diverse robotic manipulation tasks,
  highlighting the potential of this approach for automating large-scale
  simulation data generation.
paper_ai_summary: A novel object-centric representation bridges the gap between Vision-Language Models' high-level reasoning and the low-level precision needed for robotic manipulation tasks, enabling zero-shot generalization across diverse tasks.
paper_ai_keywords: Vision-Language Models Vision-Language-Action Models object-centric representation canonical space functional affordances interaction primitives dual closed-loop open-vocabulary 6D pose tracking zero-shot generalization large-scale simulation data generation