title: One to rule them all: natural language to bind communication, perception
    and action
summary: In recent years, research in the area of human-robot interaction has focused
  on developing robots capable of understanding complex human instructions and
  performing tasks in dynamic and diverse environments. These systems have a wide
  range of applications, from personal assistance to industrial robotics,
  emphasizing the importance of robots interacting flexibly, naturally and safely
  with humans. This paper presents an advanced architecture for robotic action
  planning that integrates communication, perception, and planning with Large
  Language Models (LLMs). Our system is designed to translate commands expressed
  in natural language into executable robot actions, incorporating environmental
  information and dynamically updating plans based on real-time feedback. The
  Planner Module is the core of the system where LLMs embedded in a modified
  ReAct framework are employed to interpret and carry out user commands. By
  leveraging their extensive pre-trained knowledge, LLMs can effectively process
  user requests without the need to introduce new knowledge on the changing
  environment. The modified ReAct framework further enhances the execution space
  by providing real-time environmental perception and the outcomes of physical
  actions. By combining robust and dynamic semantic map representations as graphs
  with control components and failure explanations, this architecture enhances a
  robot adaptability, task execution, and seamless collaboration with human users
  in shared and dynamic environments. Through the integration of continuous
  feedback loops with the environment the system can dynamically adjusts the plan
  to accommodate unexpected changes, optimizing the robot ability to perform
  tasks. Using a dataset of previous experience is possible to provide detailed
  feedback about the failure. Updating the LLMs context of the next iteration
  with suggestion on how to overcame the issue.
paper_authors._id: 67442e8d28cc74d48002625a
paper_authors.name: Simone Colombani
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-25T08:33:54.023Z
paper_authors._id: 65ddf03dd53ad584a7df9dac
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65ddf03dd53ad584a7df9dac/oi0pvfpCJmnODErAfF_9o.jpeg
paper_authors.fullname: Simone Colombani
paper_authors.type: user
paper_authors.user: colo286
paper_authors._id: 67442e8d28cc74d48002625b
paper_authors.name: Dimitri Ognibene
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-25T21:08:24.635Z
paper_authors._id: 6744e5f279d6f3a9da53e7e3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/2Iz-MSPungfwCgPSW85_u.png
paper_authors.fullname: dimitri ognibene
paper_authors.type: user
paper_authors.user: ddo77
paper_authors._id: 67442e8d28cc74d48002625c
paper_authors.name: Giuseppe Boccignone
paper_title: One to rule them all: natural language to bind communication, perception
    and action
paper_summary: In recent years, research in the area of human-robot interaction has focused
  on developing robots capable of understanding complex human instructions and
  performing tasks in dynamic and diverse environments. These systems have a wide
  range of applications, from personal assistance to industrial robotics,
  emphasizing the importance of robots interacting flexibly, naturally and safely
  with humans. This paper presents an advanced architecture for robotic action
  planning that integrates communication, perception, and planning with Large
  Language Models (LLMs). Our system is designed to translate commands expressed
  in natural language into executable robot actions, incorporating environmental
  information and dynamically updating plans based on real-time feedback. The
  Planner Module is the core of the system where LLMs embedded in a modified
  ReAct framework are employed to interpret and carry out user commands. By
  leveraging their extensive pre-trained knowledge, LLMs can effectively process
  user requests without the need to introduce new knowledge on the changing
  environment. The modified ReAct framework further enhances the execution space
  by providing real-time environmental perception and the outcomes of physical
  actions. By combining robust and dynamic semantic map representations as graphs
  with control components and failure explanations, this architecture enhances a
  robot adaptability, task execution, and seamless collaboration with human users
  in shared and dynamic environments. Through the integration of continuous
  feedback loops with the environment the system can dynamically adjusts the plan
  to accommodate unexpected changes, optimizing the robot ability to perform
  tasks. Using a dataset of previous experience is possible to provide detailed
  feedback about the failure. Updating the LLMs context of the next iteration
  with suggestion on how to overcame the issue.
paper_ai_keywords: Large Language Models (LLMs) modified ReAct framework semantic map representations graphs control components failure explanations