One to rule them all: natural language to bind communication, perception
  and action
In recent years, research in the area of human-robot interaction has focused
on developing robots capable of understanding complex human instructions and
performing tasks in dynamic and diverse environments. These systems have a wide
range of applications, from personal assistance to industrial robotics,
emphasizing the importance of robots interacting flexibly, naturally and safely
with humans. This paper presents an advanced architecture for robotic action
planning that integrates communication, perception, and planning with Large
Language Models (LLMs). Our system is designed to translate commands expressed
in natural language into executable robot actions, incorporating environmental
information and dynamically updating plans based on real-time feedback. The
Planner Module is the core of the system where LLMs embedded in a modified
ReAct framework are employed to interpret and carry out user commands. By
leveraging their extensive pre-trained knowledge, LLMs can effectively process
user requests without the need to introduce new knowledge on the changing
environment. The modified ReAct framework further enhances the execution space
by providing real-time environmental perception and the outcomes of physical
actions. By combining robust and dynamic semantic map representations as graphs
with control components and failure explanations, this architecture enhances a
robot adaptability, task execution, and seamless collaboration with human users
in shared and dynamic environments. Through the integration of continuous
feedback loops with the environment the system can dynamically adjusts the plan
to accommodate unexpected changes, optimizing the robot ability to perform
tasks. Using a dataset of previous experience is possible to provide detailed
feedback about the failure. Updating the LLMs context of the next iteration
with suggestion on how to overcame the issue.
67442e8d28cc74d48002625a
Simone Colombani
claimed_verified
2024-11-25T08:33:54.023Z
65ddf03dd53ad584a7df9dac
https://cdn-avatars.huggingface.co/v1/production/uploads/65ddf03dd53ad584a7df9dac/oi0pvfpCJmnODErAfF_9o.jpeg
Simone Colombani
user
colo286
67442e8d28cc74d48002625b
Dimitri Ognibene
claimed_verified
2024-11-25T21:08:24.635Z
6744e5f279d6f3a9da53e7e3
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/2Iz-MSPungfwCgPSW85_u.png
dimitri ognibene
user
ddo77
67442e8d28cc74d48002625c
Giuseppe Boccignone
One to rule them all: natural language to bind communication, perception
  and action
In recent years, research in the area of human-robot interaction has focused
on developing robots capable of understanding complex human instructions and
performing tasks in dynamic and diverse environments. These systems have a wide
range of applications, from personal assistance to industrial robotics,
emphasizing the importance of robots interacting flexibly, naturally and safely
with humans. This paper presents an advanced architecture for robotic action
planning that integrates communication, perception, and planning with Large
Language Models (LLMs). Our system is designed to translate commands expressed
in natural language into executable robot actions, incorporating environmental
information and dynamically updating plans based on real-time feedback. The
Planner Module is the core of the system where LLMs embedded in a modified
ReAct framework are employed to interpret and carry out user commands. By
leveraging their extensive pre-trained knowledge, LLMs can effectively process
user requests without the need to introduce new knowledge on the changing
environment. The modified ReAct framework further enhances the execution space
by providing real-time environmental perception and the outcomes of physical
actions. By combining robust and dynamic semantic map representations as graphs
with control components and failure explanations, this architecture enhances a
robot adaptability, task execution, and seamless collaboration with human users
in shared and dynamic environments. Through the integration of continuous
feedback loops with the environment the system can dynamically adjusts the plan
to accommodate unexpected changes, optimizing the robot ability to perform
tasks. Using a dataset of previous experience is possible to provide detailed
feedback about the failure. Updating the LLMs context of the next iteration
with suggestion on how to overcame the issue.
Advanced robotic action planning architecture integrates Large Language Models and ReAct framework for natural language command interpretation and execution, enhancing flexibility and adaptability in dynamic environments.
Large Language Models ReAct framework interface Planner Module real-time feedback semantic map representations graphs control components failure explanations dynamic adaptation task execution human collaboration continuous feedback loops context updating