title: KUDA: Keypoints to Unify Dynamics Learning and Visual Prompting for
    Open-Vocabulary Robotic Manipulation
summary: With the rapid advancement of large language models (LLMs) and
  vision-language models (VLMs), significant progress has been made in developing
  open-vocabulary robotic manipulation systems. However, many existing approaches
  overlook the importance of object dynamics, limiting their applicability to
  more complex, dynamic tasks. In this work, we introduce KUDA, an
  open-vocabulary manipulation system that integrates dynamics learning and
  visual prompting through keypoints, leveraging both VLMs and learning-based
  neural dynamics models. Our key insight is that a keypoint-based target
  specification is simultaneously interpretable by VLMs and can be efficiently
  translated into cost functions for model-based planning. Given language
  instructions and visual observations, KUDA first assigns keypoints to the RGB
  image and queries the VLM to generate target specifications. These abstract
  keypoint-based representations are then converted into cost functions, which
  are optimized using a learned dynamics model to produce robotic trajectories.
  We evaluate KUDA on a range of manipulation tasks, including free-form language
  instructions across diverse object categories, multi-object interactions, and
  deformable or granular objects, demonstrating the effectiveness of our
  framework. The project page is available at http://kuda-dynamics.github.io.
paper_authors._id: 67da141d6b2857e3ec1412a7
paper_authors.name: Zixian Liu
paper_authors._id: 67da141d6b2857e3ec1412a8
paper_authors.name: Mingtong Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:46:28.495Z
paper_authors._id: 671c6a3e255aa50ebb504fc5
paper_authors.avatarUrl: /avatars/b549bb19990ae21690472799817f951e.svg
paper_authors.fullname: Mingtong Zhang
paper_authors.type: user
paper_authors.user: Mingtongz
paper_authors._id: 67da141d6b2857e3ec1412a9
paper_authors.name: Yunzhu Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:46:19.486Z
paper_authors._id: 64a46562c641afb468a3dc04
paper_authors.avatarUrl: /avatars/bc54e5c13f657251b05bf2725086e56d.svg
paper_authors.fullname: Yunzhu Li
paper_authors.type: user
paper_authors.user: yunzhuli
paper_title: KUDA: Keypoints to Unify Dynamics Learning and Visual Prompting for
    Open-Vocabulary Robotic Manipulation
paper_summary: With the rapid advancement of large language models (LLMs) and
  vision-language models (VLMs), significant progress has been made in developing
  open-vocabulary robotic manipulation systems. However, many existing approaches
  overlook the importance of object dynamics, limiting their applicability to
  more complex, dynamic tasks. In this work, we introduce KUDA, an
  open-vocabulary manipulation system that integrates dynamics learning and
  visual prompting through keypoints, leveraging both VLMs and learning-based
  neural dynamics models. Our key insight is that a keypoint-based target
  specification is simultaneously interpretable by VLMs and can be efficiently
  translated into cost functions for model-based planning. Given language
  instructions and visual observations, KUDA first assigns keypoints to the RGB
  image and queries the VLM to generate target specifications. These abstract
  keypoint-based representations are then converted into cost functions, which
  are optimized using a learned dynamics model to produce robotic trajectories.
  We evaluate KUDA on a range of manipulation tasks, including free-form language
  instructions across diverse object categories, multi-object interactions, and
  deformable or granular objects, demonstrating the effectiveness of our
  framework. The project page is available at http://kuda-dynamics.github.io.
paper_ai_summary: KUDA integrates dynamics learning and visual prompting through keypoints to enhance robotic manipulation systems, enabling interpretation of language instructions and planning of robotic trajectories with cost functions derived from VLMs and neural dynamics models.
paper_ai_keywords: open-vocabulary robotic manipulation systems large language models vision-language models dynamics learning visual prompting keypoints target specifications model-based planning cost functions learned dynamics model robotic trajectories free-form language instructions multi-object interactions deformable objects granular objects