title: Creative Robot Tool Use with Large Language Models
summary: Tool use is a hallmark of advanced intelligence, exemplified in both animal
  behavior and robotic capabilities. This paper investigates the feasibility of
  imbuing robots with the ability to creatively use tools in tasks that involve
  implicit physical constraints and long-term planning. Leveraging Large Language
  Models (LLMs), we develop RoboTool, a system that accepts natural language
  instructions and outputs executable code for controlling robots in both
  simulated and real-world environments. RoboTool incorporates four pivotal
  components: (i) an "Analyzer" that interprets natural language to discern key
  task-related concepts, (ii) a "Planner" that generates comprehensive strategies
  based on the language input and key concepts, (iii) a "Calculator" that
  computes parameters for each skill, and (iv) a "Coder" that translates these
  plans into executable Python code. Our results show that RoboTool can not only
  comprehend explicit or implicit physical constraints and environmental factors
  but also demonstrate creative tool use. Unlike traditional Task and Motion
  Planning (TAMP) methods that rely on explicit optimization, our LLM-based
  system offers a more flexible, efficient, and user-friendly solution for
  complex robotics tasks. Through extensive experiments, we validate that
  RoboTool is proficient in handling tasks that would otherwise be infeasible
  without the creative use of tools, thereby expanding the capabilities of
  robotic systems. Demos are available on our project page:
  https://creative-robotool.github.io/.
paper_authors._id: 6535da7924c30636968cc691
paper_authors.name: Mengdi Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-23T09:46:07.342Z
paper_authors._id: 639187d92e1b430e96d45bc6
paper_authors.avatarUrl: /avatars/1e1ee95a75adc4145166c5b50cbd524f.svg
paper_authors.fullname: Mengdi Xu
paper_authors.type: user
paper_authors.user: mengdi
paper_authors._id: 6535da7924c30636968cc692
paper_authors.name: Peide Huang
paper_authors._id: 6535da7924c30636968cc693
paper_authors.name: Wenhao Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-23T09:46:24.098Z
paper_authors._id: 5feab3a28a3201f8e554c969
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1660795228685-5feab3a28a3201f8e554c969.png
paper_authors.fullname: Wenhao Yu
paper_authors.type: user
paper_authors.user: wyu1
paper_authors._id: 6535da7924c30636968cc694
paper_authors.name: Shiqi Liu
paper_authors._id: 6535da7924c30636968cc695
paper_authors.name: Xilun Zhang
paper_authors._id: 6535da7924c30636968cc696
paper_authors.name: Yaru Niu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-23T14:46:32.276Z
paper_authors._id: 65366f44dea545ecdacf5cc9
paper_authors.avatarUrl: /avatars/da6918251907872f446f70ba3bea1d4a.svg
paper_authors.fullname: Yaru Niu
paper_authors.type: user
paper_authors.user: chrisyrniu
paper_authors._id: 6535da7924c30636968cc697
paper_authors.name: Tingnan Zhang
paper_authors._id: 6535da7924c30636968cc698
paper_authors.name: Fei Xia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-23T08:09:57.218Z
paper_authors._id: 62cfa9ee9b96f22525c36d47
paper_authors.avatarUrl: /avatars/8bfe3bf63f0609b3b6b7d7c5d7f6b3e9.svg
paper_authors.fullname: Fei Xia
paper_authors.type: user
paper_authors.user: fxia22
paper_authors._id: 6535da7924c30636968cc699
paper_authors.name: Jie Tan
paper_authors._id: 6535da7924c30636968cc69a
paper_authors.name: Ding Zhao
paper_title: Creative Robot Tool Use with Large Language Models
paper_summary: Tool use is a hallmark of advanced intelligence, exemplified in both animal
  behavior and robotic capabilities. This paper investigates the feasibility of
  imbuing robots with the ability to creatively use tools in tasks that involve
  implicit physical constraints and long-term planning. Leveraging Large Language
  Models (LLMs), we develop RoboTool, a system that accepts natural language
  instructions and outputs executable code for controlling robots in both
  simulated and real-world environments. RoboTool incorporates four pivotal
  components: (i) an "Analyzer" that interprets natural language to discern key
  task-related concepts, (ii) a "Planner" that generates comprehensive strategies
  based on the language input and key concepts, (iii) a "Calculator" that
  computes parameters for each skill, and (iv) a "Coder" that translates these
  plans into executable Python code. Our results show that RoboTool can not only
  comprehend explicit or implicit physical constraints and environmental factors
  but also demonstrate creative tool use. Unlike traditional Task and Motion
  Planning (TAMP) methods that rely on explicit optimization, our LLM-based
  system offers a more flexible, efficient, and user-friendly solution for
  complex robotics tasks. Through extensive experiments, we validate that
  RoboTool is proficient in handling tasks that would otherwise be infeasible
  without the creative use of tools, thereby expanding the capabilities of
  robotic systems. Demos are available on our project page:
  https://creative-robotool.github.io/.
paper_ai_keywords: Large Language Models (LLMs) Task and Motion Planning (TAMP)