ROS-LLM: A ROS framework for embodied AI with task feedback and
  structured reasoning
We present a framework for intuitive robot programming by non-experts,
leveraging natural language prompts and contextual information from the Robot
Operating System (ROS). Our system integrates large language models (LLMs),
enabling non-experts to articulate task requirements to the system through a
chat interface. Key features of the framework include: integration of ROS with
an AI agent connected to a plethora of open-source and commercial LLMs,
automatic extraction of a behavior from the LLM output and execution of ROS
actions/services, support for three behavior modes (sequence, behavior tree,
state machine), imitation learning for adding new robot actions to the library
of possible actions, and LLM reflection via human and environment feedback.
Extensive experiments validate the framework, showcasing robustness,
scalability, and versatility in diverse scenarios, including long-horizon
tasks, tabletop rearrangements, and remote supervisory control. To facilitate
the adoption of our framework and support the reproduction of our results, we
have made our code open-source. You can access it at:
https://github.com/huawei-noah/HEBO/tree/master/ROSLLM.
668402130e8e112087145bff
Christopher E. Mower
admin_assigned
2024-07-03T09:03:10.159Z
645a5179e7e8a4b57bd8efd4
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/K5QaVWvQdmfRs42cZhzZw.jpeg
Christopher E. Mower
user
cmower
668402130e8e112087145c00
Yuhui Wan
claimed_verified
2024-07-02T21:08:56.149Z
66840ec0e3415c7a88360e4f
/avatars/71cae63ff0fc9e891fd78fcc6d39b0ba.svg
Yuhui Wan
user
W9YBZ
668402130e8e112087145c01
Hongzhan Yu
admin_assigned
2024-07-03T09:03:17.953Z
64def66a288ed5cc73ff6c11
/avatars/8bbc05070d687246992eca897acbe492.svg
Hongzhan Yu
user
hoy021
668402130e8e112087145c02
Antoine Grosnit
668402130e8e112087145c03
Jonas Gonzalez-Billandon
claimed_verified
2024-07-04T13:29:15.227Z
64ad708b7ac35b7de066d860
/avatars/a87d76e40ab85af668de9e26aabbf34a.svg
Jonas Gonzalez
user
Johnas
668402130e8e112087145c04
Matthieu Zimmer
668402130e8e112087145c05
Jinlong Wang
668402130e8e112087145c06
Xinyu Zhang
668402130e8e112087145c07
Yao Zhao
668402130e8e112087145c08
Anbang Zhai
668402130e8e112087145c09
Puze Liu
admin_assigned
2024-07-03T09:07:13.292Z
63fcc17e93b993a4ebd6be54
/avatars/5a841e1d5abbc9f2f0dbe512f49d69f6.svg
Puze Liu
user
Belongsx
668402130e8e112087145c0a
Davide Tateo
668402130e8e112087145c0b
Cesar Cadena
668402130e8e112087145c0c
Marco Hutter
668402130e8e112087145c0d
Jan Peters
668402130e8e112087145c0e
Guangjian Tian
668402130e8e112087145c0f
Yuzheng Zhuang
668402130e8e112087145c10
Kun Shao
668402130e8e112087145c11
Xingyue Quan
668402130e8e112087145c12
Jianye Hao
668402130e8e112087145c13
Jun Wang
668402130e8e112087145c14
Haitham Bou-Ammar
admin_assigned
2024-07-03T09:04:26.528Z
631c375768f7da9ad2496bf6
https://cdn-avatars.huggingface.co/v1/production/uploads/631c375768f7da9ad2496bf6/1sDOoecA6e1v_hn_VAgUq.jpeg
Haitham Bou Ammar
user
hba123
ROS-LLM: A ROS framework for embodied AI with task feedback and
  structured reasoning
We present a framework for intuitive robot programming by non-experts,
leveraging natural language prompts and contextual information from the Robot
Operating System (ROS). Our system integrates large language models (LLMs),
enabling non-experts to articulate task requirements to the system through a
chat interface. Key features of the framework include: integration of ROS with
an AI agent connected to a plethora of open-source and commercial LLMs,
automatic extraction of a behavior from the LLM output and execution of ROS
actions/services, support for three behavior modes (sequence, behavior tree,
state machine), imitation learning for adding new robot actions to the library
of possible actions, and LLM reflection via human and environment feedback.
Extensive experiments validate the framework, showcasing robustness,
scalability, and versatility in diverse scenarios, including long-horizon
tasks, tabletop rearrangements, and remote supervisory control. To facilitate
the adoption of our framework and support the reproduction of our results, we
have made our code open-source. You can access it at:
https://github.com/huawei-noah/HEBO/tree/master/ROSLLM.
A framework enables non-experts to program robots using natural language prompts and contextual information from the Robot Operating System (ROS), supported by large language models and offering various behavior modes, imitation learning, and feedback mechanisms.
large language models Robot Operating System AI agent behavior modes sequence behavior tree state machine imitation learning LLM reflection open-source chat interface