title: ROS-LLM: A ROS framework for embodied AI with task feedback and
    structured reasoning
summary: We present a framework for intuitive robot programming by non-experts,
  leveraging natural language prompts and contextual information from the Robot
  Operating System (ROS). Our system integrates large language models (LLMs),
  enabling non-experts to articulate task requirements to the system through a
  chat interface. Key features of the framework include: integration of ROS with
  an AI agent connected to a plethora of open-source and commercial LLMs,
  automatic extraction of a behavior from the LLM output and execution of ROS
  actions/services, support for three behavior modes (sequence, behavior tree,
  state machine), imitation learning for adding new robot actions to the library
  of possible actions, and LLM reflection via human and environment feedback.
  Extensive experiments validate the framework, showcasing robustness,
  scalability, and versatility in diverse scenarios, including long-horizon
  tasks, tabletop rearrangements, and remote supervisory control. To facilitate
  the adoption of our framework and support the reproduction of our results, we
  have made our code open-source. You can access it at:
  https://github.com/huawei-noah/HEBO/tree/master/ROSLLM.
paper_authors._id: 668402130e8e112087145bff
paper_authors.name: Christopher E. Mower
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T09:03:10.159Z
paper_authors._id: 645a5179e7e8a4b57bd8efd4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/K5QaVWvQdmfRs42cZhzZw.jpeg
paper_authors.fullname: Christopher E. Mower
paper_authors.type: user
paper_authors.user: cmower
paper_authors._id: 668402130e8e112087145c00
paper_authors.name: Yuhui Wan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-02T21:08:56.149Z
paper_authors._id: 66840ec0e3415c7a88360e4f
paper_authors.avatarUrl: /avatars/71cae63ff0fc9e891fd78fcc6d39b0ba.svg
paper_authors.fullname: Yuhui Wan
paper_authors.type: user
paper_authors.user: W9YBZ
paper_authors._id: 668402130e8e112087145c01
paper_authors.name: Hongzhan Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T09:03:17.953Z
paper_authors._id: 64def66a288ed5cc73ff6c11
paper_authors.avatarUrl: /avatars/8bbc05070d687246992eca897acbe492.svg
paper_authors.fullname: Hongzhan Yu
paper_authors.type: user
paper_authors.user: hoy021
paper_authors._id: 668402130e8e112087145c02
paper_authors.name: Antoine Grosnit
paper_authors._id: 668402130e8e112087145c03
paper_authors.name: Jonas Gonzalez-Billandon
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-04T13:29:15.227Z
paper_authors._id: 64ad708b7ac35b7de066d860
paper_authors.avatarUrl: /avatars/a87d76e40ab85af668de9e26aabbf34a.svg
paper_authors.fullname: Jonas Gonzalez
paper_authors.type: user
paper_authors.user: Johnas
paper_authors._id: 668402130e8e112087145c04
paper_authors.name: Matthieu Zimmer
paper_authors._id: 668402130e8e112087145c05
paper_authors.name: Jinlong Wang
paper_authors._id: 668402130e8e112087145c06
paper_authors.name: Xinyu Zhang
paper_authors._id: 668402130e8e112087145c07
paper_authors.name: Yao Zhao
paper_authors._id: 668402130e8e112087145c08
paper_authors.name: Anbang Zhai
paper_authors._id: 668402130e8e112087145c09
paper_authors.name: Puze Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T09:07:13.292Z
paper_authors._id: 63fcc17e93b993a4ebd6be54
paper_authors.avatarUrl: /avatars/5a841e1d5abbc9f2f0dbe512f49d69f6.svg
paper_authors.fullname: Puze Liu
paper_authors.type: user
paper_authors.user: Belongsx
paper_authors._id: 668402130e8e112087145c0a
paper_authors.name: Davide Tateo
paper_authors._id: 668402130e8e112087145c0b
paper_authors.name: Cesar Cadena
paper_authors._id: 668402130e8e112087145c0c
paper_authors.name: Marco Hutter
paper_authors._id: 668402130e8e112087145c0d
paper_authors.name: Jan Peters
paper_authors._id: 668402130e8e112087145c0e
paper_authors.name: Guangjian Tian
paper_authors._id: 668402130e8e112087145c0f
paper_authors.name: Yuzheng Zhuang
paper_authors._id: 668402130e8e112087145c10
paper_authors.name: Kun Shao
paper_authors._id: 668402130e8e112087145c11
paper_authors.name: Xingyue Quan
paper_authors._id: 668402130e8e112087145c12
paper_authors.name: Jianye Hao
paper_authors._id: 668402130e8e112087145c13
paper_authors.name: Jun Wang
paper_authors._id: 668402130e8e112087145c14
paper_authors.name: Haitham Bou-Ammar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T09:04:26.528Z
paper_authors._id: 631c375768f7da9ad2496bf6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/631c375768f7da9ad2496bf6/1sDOoecA6e1v_hn_VAgUq.jpeg
paper_authors.fullname: Haitham Bou Ammar
paper_authors.type: user
paper_authors.user: hba123
paper_title: ROS-LLM: A ROS framework for embodied AI with task feedback and
    structured reasoning
paper_summary: We present a framework for intuitive robot programming by non-experts,
  leveraging natural language prompts and contextual information from the Robot
  Operating System (ROS). Our system integrates large language models (LLMs),
  enabling non-experts to articulate task requirements to the system through a
  chat interface. Key features of the framework include: integration of ROS with
  an AI agent connected to a plethora of open-source and commercial LLMs,
  automatic extraction of a behavior from the LLM output and execution of ROS
  actions/services, support for three behavior modes (sequence, behavior tree,
  state machine), imitation learning for adding new robot actions to the library
  of possible actions, and LLM reflection via human and environment feedback.
  Extensive experiments validate the framework, showcasing robustness,
  scalability, and versatility in diverse scenarios, including long-horizon
  tasks, tabletop rearrangements, and remote supervisory control. To facilitate
  the adoption of our framework and support the reproduction of our results, we
  have made our code open-source. You can access it at:
  https://github.com/huawei-noah/HEBO/tree/master/ROSLLM.
paper_ai_keywords: Robot Operating System (ROS) large language models (LLMs) AI agent behavior comprehension ROS actions/services behavior tree state machine imitation learning LLM reflection